Arts-Based Methods in Evaluation 101

Reflections on practice, methodology deep dive, tips and tricks

Boring surveys, long-winded interviews, and exhausting key performance indicators push evaluators and organizations to put on their best late-night infomercial presenter voice and say, "There's gotta be a better way!"

While we're not selling a magic bullet, we are thrilled to see a growing thirst for, and application of, arts in evaluation. And, we're also a little concerned about who is doing which things for what reasons. Classic human problem, right? This blog invites us all to pause and reflect on the growing momentum toward arts-based practices in evaluation to understand:

What arts-based methods are
What arts-based methods are not
How arts-based methods work & their unique benefits
Pitfalls & mistakes when using arts-based methods in evaluation
Ethical considerations & conundrums
Barriers & mitigation strategies

Whether you're a novice looking to expand your creative evaluation toolbox, an artist looking for pathways into evaluation, a well-versed creative evaluator, a skeptic ready to critically interrogate arts-based methods, a quantitative expert looking for solace in a spreadsheet, or somewhere in-between, we've got you covered.

Ready? Let's jump in!

What Arts-Based Methods Are

In evaluation, arts-based methods are techniques or activities that use an art-making activity paired with a debrief process to elicit context-specific wisdom (data) that help to answer the evaluation question(s) and address the overall project goals. Typically, an art-making process or an art product is used as a conversation aid to tap into rich reflections. Arts activities are diverse - that's why it is arts-based methods, not art-based methods. Common arts-based methods in evaluation include:

a feather quill, camera, drama masks, headphones, and a collage

Literary methods, like poetry or storytelling
Visual methods, like drawing or photography
Performance methods, like dance or theatre
Audio methods, like music or soundscapes
Multimedia methods, like collage or digital storytelling

It is helpful to be familiar with more than one art method to understand when to use which one in an evaluation project.

What Arts-Based Methods Are Not

Interchangeable or all the same Like cousins at a family reunion, arts methods may share similar traits and quirks but also come from different traditions, carry different baggage, and elicit different kinds of wisdom. While the umbrella term, "arts-based methods" can be helpful shorthand, it is also detrimental to honouring the distinct art forms we engage with.
Therapy Arts therapy aims to support folks along their healing and growth journeys. In evaluation, arts-based methods are used to elicit data. Healing and generating data are not comparable goals. While an art technique may be selected to support feelings of catharsis and mitigate risks of harm while collecting data, evaluators cannot claim that data collection is doing double duty as therapy (unless that evaluator is also a trained professional in a healing practice...in which case, who am I to say otherwise).
Risk-free K, let's be real for a quick sec. Evaluation, like research, is riddled with colonial, capitalist, racist, ableist, and sexist baggage. We cannot claim that doing something fun, novel, creative, or imaginative is inherently lower risk than any other data collection technique. Evaluators can perpetuate harmful, extractive, and oppressive data collection practices just as easily with numbers as with art. Who the evaluator is, what they value, and how they practice is as important as their chosen technique(s).
Fluffy or bogus Please don't be that person who applies Eurocentric and positivist metrics of success to assess the rigour or reliability of a method regardless of where it comes from. Arts-based methods don't come from positivism - or post-positivism. They don't need p values, generalizability, or a big n to be valid. Like the long-standing artistic traditions they originate from, arts-based methods come from diverse cultures, spaces, and people. It is important that they are used/evaluated by folks who know whether a technique fits the intended purpose. Now, let's take that big n energy somewhere else.
Right for all projects and people Art is not always the answer. Like any other evaluation technique, arts-based methods are not a one-size-fits-all. Some methods work in some contexts. Check in to understand if the technique is wanted, and work with the organization or community to design the activity.
A shortcut for - or equivalent to - equity work Arts methods may help to address some issues of power inequities in evaluation by making space for more diverse forms of self-expression. Ideally, this changes whose perspectives are included and privileged - resulting in more agency or decision-making power in the evaluation for folks who have historically or continue to be excluded. But, this isn't a given. Evaluators must intentionally pay attention to - and address - power to promote equity. And, then what? Does this inherently redistribute power in that organization, community, or society? Equity work isn't done when the evaluation report is finished. That's just the starting point.
A free-for-all for anything creative Arts methods come from art techniques. Certain rules of the craft - which are place-based and context-specific - inform the underlying logic of the methodology, like what kind of wisdom can be elicited from each art form. While we are big advocates of being more creative in evaluation projects, not all creative, imaginative, or whimsical activities are a methodology. Suggesting that anything whimsical or creative is an arts-based method undermines the thoughtfulness of designing an activity that uses that art form's aesthetic and multi-sensory properties to elicit relevant data.

How Arts-Based Methods Work & Unique Benefits

an evaluation journey wheel starting with the evaluation question, leading to data collection (including art-making and a qualitative debrief), data collection, data analysis, and data use

Arts-Based Evaluation Journey, Nichols & Lefkowich 2023

How they Work

To collect data in evaluations, we use arts-based methods as follows:

We create a prompt (tied to the evaluation question) and facilitate an art-making exercise to:

a hand drawing and the word, "art making"

Surface insights from the process of making the art (e.g., how it feels to do the activity, what surprises bubble up, how people's perceptions/perspectives change during the process)
Elicit wisdom anchored in the art product (e.g., what certain elements mean or represent relative to key themes or topics in the evaluation)

Facilitate a qualitative debrief to make sense of the art-making process and/or art product

If we want to know about one person's insights in-depth or attend to power dynamics, we may facilitate an individual qualitative interview or informal chat to unpack the art process/product
If we want to know about diverse interpretations of the prompt, compare/contrast perspectives, build consensus, or explore the vastness of experiences, we might facilitate a focus group or informal group conversation to unpack the group wisdom tied to the art process/product We find that more imaginative interviewing techniques are helpful here, something we teach in our Beginner Arts-Based Methods for Evaluators course.

Analyze qualitative data from the debrief with an appropriate technique (e.g., thematic analysis,

written transcripts and the word, "analysis"

discourse analysis). We NEVER analyze the art product as a unit of data. Unless we ask, we cannot know what someone means by their art. A hundred people can take a photo of a tree, and that tree will mean or represent something different to everyone. Someone's drawing can have no trees, but still be about a tree because they didn't know how to draw it. Someone could write a sonnet and never once mention a tree, but the whole thing was a metaphor for a tree. Someone else could craft a Haiku about a tree that has nothing to do with trees and everything to do with finding a convenient 1-syllable word. Inviting someone to express themselves creatively is not an invitation to make assumptions about what they think, feel, mean, can represent using that artistic form, or want us to know.

Unique Benefits

Wisdom doesn't just sit in the head. It lives in our bodies, is tied to multiple senses, flirts with imagination, and is connected to memories. So, it is challenging to represent the complexity of who we are and how we experience the world within tidy Likert Scales, drop-down menus of preselected options, or open-ended questions about barriers and facilitators. Art-making creates a conversation with different entry points into an evaluation topic with room to show up as a messy human with complications and contradictions.

Each art form elicits wisdom based on its own aesthetic properties. Of course, there are similarities between multiple forms. For example, drawing and photography elicit deep insights tied to metaphor because visually representing an experience, assumption, perspective, or value invites people to think more abstractly. However, photography invites people to capture an image of something that exists and drawing invites people to invent something from their heads. So, there can be differences in what insights surface and how participating feels during different exercises.

For instance, Jennica and I asked, "What does evaluation mean to you?" Then, we took a photo and created a 5-minute drawing to represent the meaning. Afterward, we unpacked both the photos and the drawings in a conversation to elicit reflections about what each composition meant to us. We created a short caption to summarize the key take-home message of each piece. As you'll see, different things surfaced from each activity.

	Jennica	Maya
Photo
Photo Caption	Evaluation is a reflective and dynamic process to unpack what happened (capturing the beauty, surprises, and oh nos) and to help us better get where we want to go.	Evaluation is way to slow down, think about where you’ve been, where you’re going, and what you’re carrying with you.
Drawing
Drawing Caption	My drawing reflects the complex methods and processes that revolve around people, values, and context. It is never a straight line to the answers.	My drawing reflects the importance of relationships and working in harmony with what resources are available without disrupting the environment.

Common Pitfalls & Mistakes

Like any method, arts-based techniques can fail. In the interest of helping - rather than shaming - fellow artsy evaluators learn, grow, and hone their arts-based practice, here are some mistakes we've seen to serve as a "what not to do" and spare y'all the disappointment.

Neglecting the evaluation question We've seen numerous evaluators use arts-based methods because it seemed fun, they had experience using it before, the technique worked with another group, someone wrote it into the grant, they saw something on YouTube and wanted to try it, etc. Getting caught up in the novelty of a technique rather than prioritizing what works best for the context and project may (a) waste people's time with an arts-based activity that does not serve a clear purpose and (b) yield out-of-scope data that leads to data waste.
Treating methods as interchangeable Not all techniques do the same things, feel the same to participate in, pose the same risks to participants, or require the same ethical considerations. Nevertheless, we've seen evaluators swap one method for another within a template-based evaluation project. For example, we've seen evaluators invite participants to send in any art piece (without considering what is being asked of participants and why) or recycling interview guides between projects without crafting questions that fit the form. Throwing spaghetti at the wall to see what sticks is not a methodology.
Trying a new technique for the first time in an evaluation At AND, Jennica and I practice all of our techniques first with each other to understand how something works, what it feels like to participate in/facilitate an exercise, and where surprises and hiccups will inevitably come. We further refine the technique with our colleagues, friends, and families. While surprises are inevitable, we work out the rough patches before ever doing something for data collection. Trying something for the first time with participants, in a project, when the stakes are high creates a situation where harm can be (unintentionally) done to both participants and the evaluator. Practice, practice, practice!
Assuming the art will speak for itself Nope! It doesn't. We keep seeing evaluators skip the debriefing step - which is the most important step (after consent) and suggest that the art piece is the unit of data. Again, how can you know what someone meant or intended you - the evaluator - to understand about their piece unless you ask? The art-making process and product is a conversational aid to help elicit data, it is not the data.

Ethical Considerations and Conundrums

Oh my word we have seen some ethically precarious shenanigans! Arts-based methods - like any other method - are subject to ethical considerations. Of course, there are too many issues to explore than is possible to include in a single blog post, so here are a few points to ponder when planning an arts-based evaluation project:

How will people make informed choices about their voluntary participation in art-making? When/how can they change their mind about making art or letting you use it for the evaluation?
Who stands to benefit from your work? What do different groups - yourself included - stand to gain or lose if the art activity (or broader evaluation) succeeds or fails?
Who owns the art and resulting qualitative data?
Are there limits to how you can use the art pieces in your work (e.g., are there cultural or community considerations to be mindful of?)
Art-making activities and resulting art products are not inherently wanted or beneficial to the group(s) involved. How will you - the evaluator - ensure that the activities selected are meaningful, appropriate, and desired?
Like any other data collection technique in evaluation, arts-based methods can be extractive. How will you ensure that you aren't taking art and resulting data away from people and communities?
How will you account for experiences of vulnerability during art-making and debriefing? When we tap into multi-sensory wisdom, stuff can come up. How will you anticipate and mitigate risks of harm (e.g., re-traumatizing, embarrassing, or excluding someone)?
No art is neutral. No person is either. How people express themselves can be riddled with contradictions and problematic assumptions or beliefs. Sometimes, folks are surprised when they see their art and are confronted with a ghost from their past or a sticking point in their unlearning journey. This can be difficult for them to deal with. How will you avoid trapping someone in one unflattering moment? Sometimes, there are no surprises - that person proudly holds a view that is harmful to others. How will you ensure that art products are not used to cause or perpetuate harm to other groups (e.g., stereotypical images, deficit-framing narratives)? How will you navigate multiple accountabilities to ensure everyone is cared for (i.e., to your participants, your client/organization, and the broader community)?
Would you participate in the data collection activity you're planning? If not, why ask someone else to do something you wouldn't do yourself?

Barriers & Mitigation Strategies

Hopefully, it is clear that arts-based methods - though potentially rewarding and awesome - are not without their fair share of issues. Even when common pitfalls are avoided and ethical protocols are on point, there are some barriers we often face in our arts-based work that are beyond our control. Here's what we face and how we deal.

Barrier	Strategy
Arts-based methods are misunderstood (at least in North America where we practice).	Arts-based methods are not only suitable for art projects and children! We are working hard to educate our community and market. This blog, countless conference presentations, our arts-based course, free webinars and organizational talks are all things we do to help raise awareness. Trying to work differently, unfortunately, comes with an educational and advocacy burden. We can't just work differently, we have to educate others about why (and how they can too). If you're struggling to educate your professional community about arts methods, don't do it alone! We'd love to chat. Send us an email and schedule an e-coffee date to connect. You can also use our resources - send skeptics our blogs and webinar recordings to break the ice.
Arts products (resulting from data collection) are used as advertising content	Evaluation is evaluation. Advertising is advertising. If clients or organizations put conditions on a data collection activity (i.e., art products and resulting qualitative data must serve as findings and marketing), we don't do it. The pressure to create positive testimonials with pretty art products that will shine on social media undermines the consent process and data integrity. So, we help our clients/organizations find a different strategy or a different evaluator for the job.
Positivism (at the intersection of other problematic "isms")	Unfortunately, we have not yet arrived at a place where the average evaluator or organization commissioning an evaluation understands the value of diverse forms of wisdom. There is a strong preference for numbers and tools that result in numbers. Sometimes it is important to count things; sometimes it is important to stop counting things. People get stuck in the way things have always been done and can't imagine different ways of checking, measuring, and monitoring. We often pitch arts-based methods as part of a mixed methods approach. Sure, we'll do a survey (if it is necessary). Absolutely, we can look at key performance indicators (if there is a good reason). And, also we can try something else. Slowly, we introduce people to other methods, build trust and interest in other activities, and try more stuff. Patience and perseverance!
Fear of imagination & silliness	Imagination and play are often seen as juvenile - something to grow out of as we pursue more "serious" or "rigorous" work. So, when we invite our clients or collaborators to do something silly, weird, or imaginative as part of the evaluation, people often feel awkward about it. To help appease the big feels, we live by the motto, "We never ask anyone to do anything we wouldn't do, and we'll go first." With this in mind, we practice everything first to understand just how vulnerable we are asking people to be. Then, we set the expectation that the resulting art can be terrible, show our own examples of terrible art, make art alongside folks participating, and keep things light and voluntary. We also name the elephant in the room. We know things can get weird, and we invite people to trust in the process and go there with us - especially if they are members of groups that hold a lot of power and could use a healthy dose of awkward (like doctors, members of the C-Suite, and funders/decision-makers). If someone still doesn't want to do the thing, they don't have to. Consent!
Pigeonholing	Often, when people learn that we do arts-based methods, they assume we either don't - or can't - do anything else. Because of barrier 1 (misunderstandings about arts-based methods) at the intersection of credentialism and sexism, we face rude questions about our competence as evaluators/ researchers and our ability to engage in any "real" methods. For instance, many people are surprised to learn that Jennica is an epidemiologist whose first love is numbers. While we are working to advocate for arts and are proud of our reputation as, "Those arts-based consultants from Canada" we also spend extra time at networking events, in interviews with potential clients, and in proposals/bids explaining what can do. If you're also struggling to be taken seriously or seen as a multifaceted evaluator, email us. You're not alone. We'll cheer you on!

Last Thoughts

Are arts-based methods right for your next evaluation project, your community, your client/organization, or you and your practice? Well, I suppose that depends on you. I hope that more people see themselves reflected in the possibility of what arts-based methods can be and take this field further. The more evaluators learn about, practice, and advocate for arts-based methods, the easier it will be for us to collectively do high-quality, respectful, and useful work. And, with more diverse input and insights into the methodologies - what works, what doesn't, with whom, when, and how - our collective wisdom and tools become stronger.

At the same time, I also hope that some people take a step back, pause and reflect on what they are or are not doing, or stop. Each time someone brings carelessness to an arts-based project and facilitates a lacklustre activity, it becomes more different for the next evaluator to advocate, get buy-in for, or build trust around arts techniques. We aren't just accountable to our participants, clients, organizations, and communities. We are also accountable to each other. Our collective ability to do good arts-based evaluation work depends on who comes before us and carves out a path and how we support the success of whoever else is coming up behind or working alongside us.

Learn More

There are WAY more arts-based methodology resources for researchers than evaluators. That's why we developed our beginner arts-based course and organizational workshops for evaluators to translate techniques across and ensure that context, theoretical foundations, and practical applications make sense.

Our beginner arts-based methods course covers hands-on 4 techniques for evaluators. Evaluators at any career stage are welcome; no artistic experience is required! The early bird rate is available until November 17, 2023 Learn more.
We offer organizational workshops teaching many arts-based techniques. From 90 minutes to a full-day deep dive, we teach hands-on skills to teams with any amount of experience. Book now!

Connect, say hi, and share your thoughts with us at publications@andimplementation.ca

Subscribe to our blog to have new content delivered right to you!

#arts #artsbasedmethods #methodology #programevaluation #creativepractice #evaluation #creativeevaluation #reflectionsonpractice #strategy #tipsandtricks #advice #professionalpractice #howto #steps #learning #trysomethingnew