Evaluation Method Deep Dive, Tips for Practice, Steps, Reflections on Practice
“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” - Maya Angelou
The intersection of what we say - or don’t say - and how that evokes emotions is powerful. But, in evaluation, we rarely slow down to explore this relationship. Instead, we often adopt empty jargon and placeholders for numbers (I’m looking at you, Likert Scale). We say a lot without really saying much. And, we neglect the vibrant wisdom steeped in our language…and silences.
Poetry - the every word matters technique - helps us pause, reflect, and tap into multisensory knowledge. While poetry can be awkward, evaluators must embrace more authentic ways of engaging people in decision-making and data collection. Let’s embrace the messy, bizarre - and yes, sometimes cringy - ways we show up. Poetry helps us do that…and so much more.
What does it mean to use poetry as a data collection tool?
Regardless of the specific poetic technique, we follow this pathway for data collection. Always starting with the evaluation question, we:
Generate a prompt for participants to engage with during their poetry creation
Unpack the poetic process and output (e.g., what came up while making poetry and what the resulting poems mean) in an individual or group conversation (e.g., an interview or focus group)
Analyze the data that resulted from the conversation to unpack the poetry (not the poem itself)
Use the findings to inform the next steps (e.g., decision-making, further reflection, recommendations)
Before trying any activities with clients or participants, we ALWAYS practice the prompt, activity, and debrief (interview or focus group). If it doesn’t work with our moms, it won’t work with our clients!
When Would I Use Poetry for Data Collection in Evaluation?
Poetic data collection activities work best when the evaluation question concerns emotions, experiences, perceptions, beliefs, hindsight, and/or values. This is particularly useful when a project or initiative is littered with jargon or has a lot of documentation and we need something to cut through that noise. For example, poetry is great when:
The evaluation aims to render the invisible visible or articulate something previously unspoken (e.g., social or group norms, power dynamics, gut wisdom or intuition, underlying values or assumptions)
The evaluation needs silence or breathing room (e.g., a project is sprinting through implementation and it is unclear who is on which page of the plan)
The evaluation can only move forward after we can let go of or replenish something (e.g., if relationships need mending, we need to build a bridge between where we’ve been and where we’re going, we need space to mourn the loss of Plan A before pursuing Plan B)
The wisdom of folks involved in the evaluation cannot authentically or meaningfully be captured in numbers or qualitative soundbites (e.g., how someone speaks is as - if not more - important as what they say)
How Do I Do It?
Many poetic exercises work well for data collection. Here are just two examples of the techniques we often use to illustrate how poetry can work in evaluation (stay tuned for more). What we love about Erasure Poetry and Found Poetry is that the creative process starts with existing words. Phew! This takes the edge off. For folks terrified of the blank page or who immediately think, "But, I'm not a poet," these two techniques are an awesome starting point.
Example 1: Erasure or "Black-Out" Poetry (also known as newspaper poetry)
Supplies:
An existing text-based document (we recommend no more than 3 pages) that is relevant to your evaluation Variation 1: The evaluator can select the text if it is important to understand how different folks interpret or engage with the same words Variation 2: The participants can select their own text if it is important to understand what people choose and how they engage with the words included in unique documents
A dark marker if you’re facilitating on a hard-copy paper
A digital editing platform with a redact tool (e.g., PDF reader) if you’re facilitating digitally
Steps:
Craft a prompt for participants to make poetry towards that (a) helps you answer your evaluation question - but is not your evaluation question, and (b) connects to the text you or your participants selected
Set a time limit to do the activity that conveys the expectation that these poems are not meant to be perfect or polished, but still creates enough spaciousness for folks to overcome any nervousness, ask questions, and jump in (practicing the activity with a friend will help you figure out timing)
Invite participants to black out either (a) any text that does not speak to them when engaging with the prompt or (b) any text that does speak to them, but is sacred or not to be part of the conversation
Debrief the poems with each poet either individually through an interview (if you want to dive deep into one person’s insights or account for vulnerability/possible power dynamics in a group) or as a group in a focus group (if you want to compare and contrast interpretations, build consensus, or explore diversity in experience). In the debrief, explore the process (e.g., what it felt like to do the activity) and the product (e.g., what the blacked-out portions or remaining words mean relative to the prompt).
Analyze your data summary or transcript from the debrief as you would with qualitative data. Looking across poems may spark questions about perceived similarities and differences in the poems (e.g., I noticed that Jennica and Maya both blacked out the word ____). Resist the urge to make assumptions about what the finished pieces mean. Only the poet knows why they left in or took out words and what the resulting prose and spaces mean.
Example 2: Found Poetry
Supplies:
An existing text-heavy document (we recommend no more than 3 pages) that is relevant to your evaluation or access to an environment where you can listen to and transcribe oral text (e.g., a hospital waiting room, conference presentation, board meeting) Variation 1: The evaluator can select the text or environment if it is important to understand how different folks interpret or engage with the same words Variation 2: The participants can select their own text or environment if it is important to understand what people choose and how they engage with the words included
Scissors and glue/tape if you’re facilitating on a hard-copy
A digital editing platform with crop and rearrange tools if you’re facilitating digitally
Steps:
Craft a prompt for participants (same as the instructions for Erasure Poetry)
Set a time limit and account for more time needed for this exercise than the previous activity because of the added steps and complexity
Invite participants to:
Cut out or write down any text that speaks to them when engaging with the prompt
Rearrange the selected text into a composition that answers the prompt
Add embellishments to illustrate their main point (e.g., this could include distorting or playing with cropped text digitally, adding drawing to a hard copy). In some cases, you may wish to invite poets to add in any text that they feel is missing by handwriting or typing those words in while doing something to signal that these words were added, not found.
Debrief the poems with each poet (same as the instructions for Erasure Poetry)
Analyze your data summary or transcript from the debrief (same as the instructions for Erasure Poetry)
Erasure and Found Poetry Compared
| Erasure Poetry | Found Poetry |
|---|---|---|
Pros |
|
|
Cons |
|
|
Remember that this is how we practice. It may not be how you practice or what is right for your evaluation. Poetry - like evaluation - is not a one-size-fits-all. Many variations could work. This is an example, not a rule.
Poetry and Ethics
Ethics are essential to the success of good arts-based methods. Here are some helpful questions when planning a poetry exercise in evaluation.
How will consent work for participation? (e.g., What process will you follow to get consent from participants? What happens if people change their minds about participation?)
How will you be thoughtful about potential risks? (e.g., What happens if the process of making or unpacking poetry evokes uncomfortable feelings, memories of a difficult experience, or past pain?)
How will consent work for sharing poetry with others either in a group debrief or in any final deliverables? (e.g., Because someone can only consent to sharing their poem after making it, how will you plan for this? If you provided existing texts for the exercise, is there a copyright or license issue to be aware of?)
Who owns the poems and the resulting qualitative data? (e.g., Will there be limits on how poems can be used after/by whom? Are there existing data ownership rules in place for this group or community to be mindful of? )
What existing texts are appropriate for you to engage with in this way? (e.g., Are any texts you selected sacred or inappropriate to mark up, cut, or distort?)
If you are overhearing texts in a location, do you need to engage in a consent process for folks talking those spaces before you overhear? Are there any risks for them?
Limitations
There is no one-size-fits-all method for any context, population, or question (at least, we haven't found it yet). And, poetry is far from a perfect technique. Here are some limitations or potential stumbling points that we have found in our practice at AND:
Poetry isn't always appropriate This exercise does not work well with folks who struggle with language/literacy. Especially because we often do this together in live time (e.g., carving out time to make poems ahead of the qualitative interview or focus group), it can be tricky to put someone on the spot to engage with words if words are not a comfortable place to play.
Poetry evokes BIG FEELS While many people are open-minded, open-hearted, and willing to try creative things, some aren’t. When we ask people to play, tap into imagination, or let things get weird with poetry, we often see peoples' gaze dart to the nearest exit. Unfortunately, the folks who could most benefit from the magic of poetry (like those in decision-making roles who overly rely on jargon to say a lot without saying much) are often the first to excuse themselves from the art-making step, tell us that the activity didn’t work, and recommend a survey instead.
Getting the text and instructions right is key Finding the right starting text can be a challenge. Too much material to work with (e.g., an entire report) can be overwhelming and impractical. But, too little text (e.g., a framework or logic model) can make playing with a good variety of words difficult. So, we follow the lead of three little bears and find the text that is just right.
Learn More
There are WAY more poetry resources available for researchers rather than evaluators. In fact, the field of poetic inquiry is growing at an exciting - and also daunting - speed. That's why we developed our beginner arts-based course and organizational workshops for evaluators to translate techniques across and ensure that context, theoretical foundations, and practical applications make sense.
Poetry is one of the 4 techniques covered in our beginner arts-based methods course for evaluators. No poetic or artistic experience is required. Our next cohort starts in January 2024 and runs for 6 weeks. Don't miss out on our early bird special price, ending November 17, 2023. Learn more.
We offer organizational workshops on arts-based techniques. Ranging from 90 minutes to a full-day deep dive, we teach hands-on skills for folks who have any amount of experience and some or no artistic skill. Book now!
Subscribe to our blog to have new content delivered right to you!
#poetry #poeticinquiry #artsbasedmethods #artsbased #literary #qualitative #qualitativemethods #evaluation #skillbuilding #professionaldevelopment #trysomethingnew #evaluation #programevaluation #research #methodology #learning #creativepractice #creativeconsulting #tipsandtricks #tipsforpractice