Evaluation Interviewing... and the art, awkward, and absurdity of asking questions
Updated: 7 days ago
When interviews are the right evaluation tool, what they really cost, and how we approach them
Part 1 of our interviewing series: Evaluating Interviewing (this) · An Interview Is Not a Sandwich · Classic Interview Blunders · Don’t Interview in Sandals
Last updated: October 2026
How on earth do you ask another person a question? Seriously. Have you met people? Blurting out words with an upward inflection rarely results in valuable answers. Questions that aim to solve a problem or elicit specific information require expertise to frame and deliver.
Plus, people don’t just answer questions. We sidestep, ramble through, forget, and impose new questions. That’s if we engage at all.
After facilitating over 500 interviews and focus groups with more than 1,000 people, interviewing is still one of our favourite data collection tools. But, it is also one of the most misunderstood and misused. The structured of asking a question and getting an answer is deceptively simple. That's what makes it so easy to mess up.
So, you know how and when to use an interview in evaluation? This post explores answers (from our practice) to the following questions:
When should you use interviews in an evaluation?
Interviews shine when you need to explore the human factor: how people experienced something, why they made the choices they did, and what was going on around them at the time. They’re especially useful when:
The answers are likely to be “it depends,” and you need to know what it depends on
You want stories, illustrative examples, and context
You’re working with a small number of people whose perspectives matter a lot (e.g., key partners, program staff, or leadership)
You started drafting a survey that has mostly open-ended question
Here's how they compare against other methods commonly used in evaluation.
Method | Best when | Ideal Sample Size | Resources |
Interviews | Exploring individual experiences and perceptions in depth | From a handful to whatever your budget can handle | Posts in this series exploring: |
Focus Groups | Generating group wisdom - mapping shared or divergent experiences, building consensus, identifying differences | 5-10 people per session, with groups organized by shared role, identity, or experience to manage power dynamics | |
Surveys | Identifying patterns or trends across a large group or at multiple points in time | The bigger the better - especially if you want to do subgroup analysis or promise anonymity | Should You Create a Survey? |
Arts-Based Methods | Articulating underlying values, beliefs, emotions, and experiences that are hard to put into words. Arts-based methods need an interview or focus group, making art is not a stand-alone method. | From a handful to whatever your budget can handle | Arts-Based Methods 101 |
When are interviews not the right fit?
Interviews are great, but they aren't a perfect fit for every evaluation goal or context. They definitely don't work well when:
You need numbers. If the question need to answer is “how many” or “how much,” a survey will serve you better, and asking people to rate things out loud turns your interview an awkward mess (more on that in Classic Interview Blunders).
People are already over-interviewed, stretched thin, or literally saving babies. If your community has been asked the same questions again and again, are on call in an active job where their time is precious and unpredictable, or the just don't have capacity, it is a good idea to pause and re-think. Exhaustive data collection is unethical.
There’s no time to do it right. If you have to blitz through the interview or take shortcuts in the analysis, you will waste everyone's time and resources by producing unreliable data.
You plan to use AI. Please don't. It sends an incredibly off-putting and out-of-touch message when asking people for their insights, and not bothering to invest in human time or connection to facilitate that interaction. If you want people to feel heard, real people have to listen.
You're not confident or experienced. A lot of people think that interviews don't require skill or training to do well because it can't possibly be as hard as collecting numbers. So, interviews are often assigned to staff who don't have the experience or supervision to do them well. But, strong emotions can bubble up, questions can go sideways, and the conversation can derail. It takes real skill to do interviews well and collect trustworthy data. If you're new to interviewing and need help building your skills and confidence, we can help with Evaluation & Research Advising for fellow practitioners, and Capacity-Building for staff at community-serving organizations.
How much time, energy, and resources do interviews really take?
More than most people budget for. We recommend planning at least four hours of work for every hour of interviewing. So ten one-hour interviews add up to roughly 50 hours, not ten.
What does that time pay for?
That time is what makes interviews respectful and the data usable. We always have to factor in time to:
Develop and test the interview guide to ensure questions make sense
Answer participant questions, set-up interviews, and coordinate honoraria
Write field memos or "evaluator hot takes" about the tone and context of each interview
Send people a copy of their transcript or notes and invite edits (if they want them)
Clean and de-identify transcripts or notes
Analyze the data
The ratio varies by interview type, data analysis strategy, and by what you need at the end (summary notes vs. fully cleaned transcripts). But if you’re planning interviews, 4:1 is a good starting point for your budget.
What happens when you cut corners with interviews?
We were once hired to do a secondary analysis of interviews an organization had already completed. To save money, they recycled an interview guide from another project, asked a student intern to conduct the interviews without training or supervision, and relied on an AI transcription feature that no one checked or cleaned.
The shortcut | The outcome | The cost |
A recycled interview guide | Questions that weren’t built for this evaluation didn't land for participants, and led to irrelevant data that didn't answer the evaluation questions. | Wasted participant time and effort. |
An unsupported student interviewer | The interviewer accidentally asked leading questions, improvised unwarranted prompts, and surfaced sensitive and identifiable data. | Wasted data and opportunities for learning and professional development. |
No supervision or quality control | Had the student been supervised, these mistakes likely would have been caught and corrected early, leading to useable data in later interviews. But, the same mistakes were in all interviews. | |
Unchecked AI transcription | None of transcripts were accuracte. We had to listen to each interview and manually correct all transcripts before analyzing and writing up. | $7,500 of extra labour. Plus, had we not cleaned the data, identifiable data would have been sitting in an unsecure drive. |
More than half the data went in the trash, and cleaning up the rest cost more than doing it right the first time. Every one of these mistakes was preventable. Good data is an investment. Nothing is more expensive than wasting people’s time and trust.
How do we approach interviewing?
Our core belief: the interviewer is the tool, not the interview guide. Why does this matter? Because we invest in the person conducting interviews. Human time, human effort, human skill. And, we don't rely on or recycle interview templates. This series explores some of our strategies in practice with lots of examples and cheeky commentary. Here's what you'll find:
An Interview Is Not a Sandwich
An overview of structured, semi-structured, and unstructured interview styles and how to choose between them for your evaluation context and goal
An exploration of common mistakes people make when interviewing (and how to avoid them) without finger-pointing or shame
An in-depth look at how we get ready to interview (preparation, getting into interviewer character, and staying present during the interview)
The bottom line
Interviews are one of the most powerful tools in evaluation, and one of the easiest to get wrong. They’re not a shortcut to quick answers, and they’re not right for every evaluation question. But when you need to understand how and why, nothing beats a well-designed conversation with a skilled interviewer.
Want to go further with interviewing?
Learn it yourself: our Arts-Based Methods for Evaluators course includes imaginative interviewing, a technique for drawing out richer, more creative answers that we only teach in the course.
Get it built: community-serving organizations can have us design an interview guide through Light Touch Consulting, or build staff interviewing skills through Capacity-Building.
Talk it through: bring your questions about interviews to Data Strategy Office Hours for organizations, or Evaluation & Research Advising for fellow evaluators.
Like what you read? Subscribe to recieve occasional content right to your inbox.
Comments