Reflections on practice, methodology deep dive, tips and tricks
Boring surveys, long-winded interviews, and exhausting key performance indicators push evaluators and organizations to put on their best late-night infomercial presenter voice and say, "There's gotta be a better way!"
While we're not selling a magic bullet, we are thrilled to see a growing thirst for, and application of, arts in evaluation. And, we're also a little concerned about who is doing which things for what reasons. Classic human problem, right? This blog invites us all to pause and reflect on the growing momentum toward arts-based practices in evaluation to understand:
Whether you're a novice looking to expand your creative evaluation toolbox, an artist looking for pathways into evaluation, a well-versed creative evaluator, a skeptic ready to critically interrogate arts-based methods, a quantitative expert looking for solace in a spreadsheet, or somewhere in-between, we've got you covered.
Ready? Let's jump in!
What Arts-Based Methods Are
In evaluation, arts-based methods are techniques or activities that use an art-making activity paired with a debrief process to elicit context-specific wisdom (data) that help to answer the evaluation question(s) and address the overall project goals. Typically, an art-making process or an art product is used as a conversation aid to tap into rich reflections. Arts activities are diverse - that's why it is arts-based methods, not art-based methods. Common arts-based methods in evaluation include:
Literary methods, like poetry or storytelling
Visual methods, like drawing or photography
Performance methods, like dance or theatre
Audio methods, like music or soundscapes
Multimedia methods, like collage or digital storytelling
It is helpful to be familiar with more than one art method to understand when to use which one in an evaluation project.
What Arts-Based Methods Are Not
Interchangeable or all the same Like cousins at a family reunion, arts methods may share similar traits and quirks but also come from different traditions, carry different baggage, and elicit different kinds of wisdom. While the umbrella term, "arts-based methods" can be helpful shorthand, it is also detrimental to honouring the distinct art forms we engage with.
Therapy Arts therapy aims to support folks along their healing and growth journeys. In evaluation, arts-based methods are used to elicit data. Healing and generating data are not comparable goals. While an art technique may be selected to support feelings of catharsis and mitigate risks of harm while collecting data, evaluators cannot claim that data collection is doing double duty as therapy (unless that evaluator is also a trained professional in a healing practice...in which case, who am I to say otherwise).
Risk-free K, let's be real for a quick sec. Evaluation, like research, is riddled with colonial, capitalist, racist, ableist, and sexist baggage. We cannot claim that doing something fun, novel, creative, or imaginative is inherently lower risk than any other data collection technique. Evaluators can perpetuate harmful, extractive, and oppressive data collection practices just as easily with numbers as with art. Who the evaluator is, what they value, and how they practice is as important as their chosen technique(s).
Fluffy or bogus Please don't be that person who applies Eurocentric and positivist metrics of success to assess the rigour or reliability of a method regardless of where it comes from. Arts-based methods don't come from positivism - or post-positivism. They don't need p values, generalizability, or a big n to be valid. Like the long-standing artistic traditions they originate from, arts-based methods come from diverse cultures, spaces, and people. It is important that they are used/evaluated by folks who know whether a technique fits the intended purpose. Now, let's take that big n energy somewhere else.
Right for all projects and people Art is not always the answer. Like any other evaluation technique, arts-based methods are not a one-size-fits-all. Some methods work in some contexts. Check in to understand if the technique is wanted, and work with the organization or community to design the activity.
A shortcut for - or equivalent to - equity work Arts methods may help to address some issues of power inequities in evaluation by making space for more diverse forms of self-expression. Ideally, this changes whose perspectives are included and privileged - resulting in more agency or decision-making power in the evaluation for folks who have historically or continue to be excluded. But, this isn't a given. Evaluators must intentionally pay attention to - and address - power to promote equity. And, then what? Does this inherently redistribute power in that organization, community, or society? Equity work isn't done when the evaluation report is finished. That's just the starting point.
A free-for-all for anything creative Arts methods come from art techniques. Certain rules of the craft - which are place-based and context-specific - inform the underlying logic of the methodology, like what kind of wisdom can be elicited from each art form. While we are big advocates of being more creative in evaluation projects, not all creative, imaginative, or whimsical activities are a methodology. Suggesting that anything whimsical or creative is an arts-based method undermines the thoughtfulness of designing an activity that uses that art form's aesthetic and multi-sensory properties to elicit relevant data.
How Arts-Based Methods Work & Unique Benefits
How they Work
To collect data in evaluations, we use arts-based methods as follows:
Create a prompt (tied to the evaluation question) and facilitate an art-making exercise to:
Surface insights from the process of making the art (e.g., how it feels to do the activity, what surprises bubble up, how people's perceptions/perspectives change during the process)
Elicit wisdom anchored in the art product (e.g., what certain elements mean or represent relative to key themes or topics in the evaluation)
Facilitate a qualitative debrief to make sense of the art-making process and/or art product
If we want to know about one person's insights in-depth or attend to power dynamics, we may facilitate an individual qualitative interview or informal chat to unpack the art process/product
If we want to know about diverse interpretations of the prompt, compare/contrast perspectives, build consensus, or explore the vastness of experiences, we might facilitate a focus group or informal group conversation to unpack the group wisdom tied to the art process/product We find that more imaginative interviewing techniques are helpful here, something we teach in our Beginner Arts-Based Methods for Evaluators course.
Analyze qualitative data from the debrief with an appropriate technique (e.g., thematic analysis, discourse analysis). We NEVER analyze the art product as a unit of data. Unless we ask, we cannot know what someone means by their art. A hundred people can take a photo of a tree, and that tree will mean or represent something different to everyone. Someone's drawing can have no trees, but still be about a tree because they didn't know how to draw it. Someone could write a sonnet and never once mention a tree, but the whole thing was a metaphor for a tree. Someone else could craft a Haiku about a tree that has nothing to do with trees and everything to do with finding a convenient 1-syllable word. Inviting someone to express themselves creatively is not an invitation to make assumptions about what they think, feel, mean, can represent using that artistic form, or want us to know.
Unique Benefits
Wisdom doesn't just sit in the head. It lives in our bodies, is tied to multiple senses, flirts with imagination, and is connected to memories. So, it is challenging to represent the complexity of who we are and how we experience the world within tidy Likert Scales, drop-down menus of preselected options, or open-ended questions about barriers and facilitators. Art-making creates a conversation with different entry points into an evaluation topic with room to show up as a messy human with complications and contradictions.
Each art form elicits wisdom based on its own aesthetic properties. Of course, there are similarities between multiple forms. For example, drawing and photography elicit deep insights tied to metaphor because visually representing an experience, assumption, perspective, or value invites people to think more abstractly. However, photography invites people to capture an image of something that exists and drawing invites people to invent something from their heads. So, there can be differences in what insights surface and how participating feels during different exercises.
For instance, Jennica and I asked, "What does evaluation mean to you?" Then, we took a photo and created a 5-minute drawing to represent the meaning. Afterward, we unpacked both the photos and the drawings in a conversation to elicit reflections about what each composition meant to us. We created a short caption to summarize the key take-home message of each piece. As you'll see, different things surfaced from each activity.
| Jennica | Maya |
Photo |  |  |
Photo Caption | Evaluation is a reflective and dynamic process to unpack what happened (capturing the beauty, surprises, and oh nos) and to help us better get where we want to go. | Evaluation is way to slow down, think about where you’ve been, where you’re going, and what you’re carrying with you. |
Drawing |  |  |
Drawing Caption | My drawing reflects the complex methods and processes that revolve around people, values, and context. It is never a straight line to the answers. | My drawing reflects the importance of relationships and working in harmony with what resources are available without disrupting the environment. |
Common Pitfalls & Mistakes
Like any method, arts-based techniques can fail. In the interest of helping - rather than shaming - fellow artsy evaluators learn, grow, and hone their arts-based practice, here are some mistakes we've seen to serve as a "what not to do" and spare y'all the disappointment.
Neglecting the evaluation question We've seen numerous evaluators use arts-based methods because it seemed fun, the technique worked with another group, someone wrote it into the grant, they saw something on YouTube and wanted to try it, etc. Getting caught up in the novelty of a technique rather than prioritizing what works best for the context and project may (a) waste people's time with an arts-based activity that does not serve a clear purpose and (b) yield out-of-scope data that leads to data waste.
Treating methods as interchangeable Not all techniques do the same things, feel the same to participate in, pose the same risks to participants, or require the same ethical considerations. Nevertheless, we've seen evaluators swap one method for another within a template-based evaluation project. For example, we've seen evaluators invite participants to send in any art piece (without considering what is being asked of participants and why) or recycling interview guides between projects without crafting questions that fit the form. Throwing spaghetti at the wall to see what sticks is not a methodology.
Trying a new technique for the first time in an evaluation At AND, Jennica and I practice all of our techniques first with each other to understand how something works, what it feels like to participate in/facilitate an exercise, and where surprises and hiccups will inevitably come. We further refine the technique with our colleagues, friends, and families. While surprises are inevitable, we work out the rough patches before ever doing something for data collection. Trying something for the first time with participants and when the stakes are high creates a situation where harm can be (unintentionally) done to both participants and the evaluator. Practice, practice, practice!
Assuming the art will speak for itself Nope! It doesn't. We keep seeing evaluators skip the debriefing step - which is the most important step (after consent) and suggest that the art piece is the unit of data. Again, how can you know what someone meant or intended you - the evaluator - to understand about their piece unless you ask? The art-making process and product is a conversational aid to help elicit data, it is not the data.
Ethical Considerations and Conundrums
Oh my word we have seen some ethically precarious shenanigans! Arts-based methods - like any other method - are subject to ethical considerations. Of course, there are too many issues to explore than is possible to include in a single blog post, so here are a few points to ponder when planning an arts-based evaluation project:
How will people make informed choices about their voluntary participation in art-making? When/how can they change their mind about making art or letting you use it for the evaluation?
Who stands to benefit from your work? What do different groups - yourself included - stand to gain or lose if the art activity (or broader evaluation) succeeds or fails?
Who owns the art and resulting qualitative data?
Are there limits to how you can use the art pieces in your work (e.g., are there cultural or community considerations to be mindful of?)
Art-making activities and resulting art products are not inherently wanted or beneficial to the group(s) involved. How will you - the evaluator - ensure that the activities selected are meaningful, appropriate, and desired?
Like any other data collection technique in evaluation, arts-based methods can be extractive. How will you ensure that you aren't taking art and resulting data away from people and communities?
How will you account for experiences of vulnerability during art-making and debriefing? When we tap into multi-sensory wisdom, stuff can come up. How will you anticipate and mitigate risks of harm (e.g., re-traumatizing, embarrassing, or excluding someone)?
No art is neutral. No person is either. How people express themselves can be riddled with contradictions and problematic assumptions or beliefs. Sometimes, folks are surprised when they see their art and are confronted with a ghost from their past or a sticking point in their unlearning journey. This can be difficult for them to deal with. How will you avoid trapping someone in one unflattering moment? Sometimes, there are no surprises - that person proudly holds a view that is harmful to others. How will you ensure that art products are not used to cause or perpetuate harm to other groups (e.g., stereotypical images, deficit-framing narratives)? How will you navigate multiple accountabilities to ensure everyone is cared for (i.e., to your participants, your client/organization, and the broader community)?
Would you participate in the data collection activity you're planning? If not, why ask someone else to do something you wouldn't do yourself?
Barriers & Mitigation Strategies
Hopefully, it is clear that arts-based methods - though potentially rewarding and awesome - are not without their fair share of issues. Even when common pitfalls are avoided and ethical protocols are on point, there are some barriers we often face in our arts-based work that are beyond our control. Here's what we face and how we deal.
Barrier | Strategy |
Arts-based methods are misunderstood (at least in North America where we practice). |
|
Arts products (resulting from data collection) are used as advertising content |
|
Positivism (at the intersection of other problematic "isms") |
|
Fear of imagination & silliness |
|
Pigeonholing |
|
Last Thoughts
Are arts-based methods right for your next evaluation project, your community, your client/organization, or you and your practice? Well, I suppose that depends on you. I hope that more people see themselves reflected in the possibility of what arts-based methods can be and take this field further. The more evaluators learn about, practice, and advocate for arts-based methods, the easier it will be for us to collectively do high-quality, respectful, and useful work. And, with more diverse input and insights into the methodologies - what works, what doesn't, with whom, when, and how - our collective wisdom and tools become stronger.
At the same time, I also hope that some people take a step back, pause and reflect on what they are or are not doing, or stop. Each time someone brings carelessness to an arts-based project and facilitates a lacklustre activity, it becomes more different for the next evaluator to advocate, get buy-in for, or build trust around arts techniques. We aren't just accountable to our participants, clients, organizations, and communities. We are also accountable to each other. Our collective ability to do good arts-based evaluation work depends on who comes before us and carves out a path and how we support the success of whoever else is coming up behind or working alongside us.
Learn More
There are WAY more arts-based methodology resources for researchers than evaluators. That's why we developed our beginner arts-based course and organizational workshops for evaluators to translate techniques across and ensure that context, theoretical foundations, and practical applications make sense.
Our beginner arts-based methods course covers hands-on 4 techniques for evaluators. Evaluators at any career stage are welcome; no artistic experience is required! Learn more.
We offer organizational workshops teaching many arts-based techniques. From 90 minutes to a full-day deep dive, we teach hands-on skills to teams with any amount of experience. Book now!
Connect, say hi, and share your thoughts with us at publications@andimplementation.ca
Subscribe to our blog to have new content delivered right to you!