How to Choose the Right Evaluation Tool for the Job: Tips & Lessons Learned
Updated: Oct 16
Identifying and using the right tool for the project is difficult. Whether you are a consultant/team pitching an approach to a client, conducting an in-house evaluation, or deciding between responses to your request for proposal (RFP), how do you know which technique will work best?
The Story
The towering buckeye tree next door shed its final leaves, covering the lawn in a thick yellow blanket. My neighbour proceeded to clear the debris with a snow shovel. Dragging the heavy metal blade across the plot, he squashed the leaves into the ground and scraped up clumps of grass and earth instead. I figured it was an inventive – albeit inelegant – solution to the problem of not having a rake.
Weeks later, Vancouver endured a sleet storm that buried the sidewalks in ice and slush. My neighbour cleaned the frigid slop from his driveway. Like nails on a chalkboard, I felt the metal scraping against concrete in my molars. He did have a rake after all!
The Point
Jennica and I often experience a similar predicament to my neighbour. When designing a project, we have ample techniques to choose from. Like seasons, when contexts change between projects, our tools should too. However, clients often ask evaluation consultants to do what is most familiar, rather than most appropriate. We get saddled with (or settle for) the same wrong tool, even when a better one is available.
Because client relationships can be precarious, it isn't obvious how to navigate inappropriate or ineffective ideas and requests. From our work using and teaching arts-based methods, we know it can be risky to remind clients that it is snowing, offer them a shovel, and convince them to use it. Organizations can be skeptical of new tools that require imagination and a willingness to part with longstanding vanity metrics.
Tips
Here’s how we help our clients choose the right tools for the job.
We invest in our imagination competency. We dedicate time to anything that sparks or replenishes creativity. For example, I do weekly creative writing exercises because they’re fun, make me giggle, and sharpen my narrative thinking. You don’t have to be an artist to build imagination muscles.
We ask clients to explain the “why” behind their requested “how.” First, we ask clients (a) what they hope the evaluation ultimately achieves, (b) what they want to learn from the data, and (c) why they want to use specific tools (if they asked for them). Then, we play with different scenarios to illustrate what could happen using many different techniques. The goal is to land the following key point.
While several methods could work in the same project, some help us better reach the goal. Therefore, we must choose the method based on the client’s goal (not retrofit evaluation goals around shiny methods).
We use our imagination competency to build trust with clients. Trust-building is about showing not telling. In the evaluation planning stages, we often use low-risk arts-based exercises and icebreakers to capture relevant information and make decisions. This gives clients a glimpse into the efficacy of unfamiliar tools and helps them trust our expertise when we say, “Something else could work better.” For example, we have successfully integrated arts-based techniques into evaluation activities, like logic modelling and KPI planning. This helps us overcome myths about unfamiliar techniques by proving we won’t bloat a budget, delay a timeline, or do anything pernicious. When clients see the benefits, we offer to do something similar in data collection to extend the same fun, meaningful, and authentic engagement to evaluation partners and participants. By being imaginative early in the evaluation process, we can earn the client’s trust, get buy-in for different techniques slowly, and improve the parts of the evaluation we control.
Last Thoughts
As consultants, we can’t always get our clients to notice the ice and put down that rake. But, we keep trying. Each time we get them to try a shovel, we are one step closer to our dream of using a Zamboni.
Looking for the right tool for your next project? Community-serving orgs can get unstuck in an hour with Data Strategy Office Hours, gain acess to an Organizational Learning Partner to build an effective data and learning system, or take a look under to hood to see what they're already using (and how its going) with a judgement-free Data Health Assessment.
Evaluators can pressure-test their approach with Evaluation & Research Advising, or partner with us on a larger project.
Not sure where to get started? Take our 3-question anonymous quiz to find a good fit.
Like what you read? Subscribe to our blog to have new content delivered right to you!
This blog post is adapted from the original article written for the December Newsletter of the American Evaluation Association's Independent Consulting TIG. Special thanks to Sarah Williams Leng!
Comments