Should You Create a Survey? A 5-Minute Reality Check for Busy Program Managers

Evaluation Methods, Tips for Effective Surveys, Reflections on Practice

So, you need feedback on your program and a survey seems like the obvious solution. But, before you dive into designing, pause for 5 minutes and consider if a survey is the right choice.

Taking a deep breath now could save money and headaches later.

The Survey Trap - Why Good Intentions Go Wrong

As evaluation consultants, we often see organizations creating surveys because they think they should, not because they have a clear plan for using the information collected. The result? Overwhelmed staff scramble to report something meaningful. Unused data collect digital dust in the cloud. Frustrated and fatigued participants feel distrustful of organizations that wasted their time and didn't act on their suggestions. While surveys are often seen as the cheap and easy solution, we want to remind you of one thing. The most expensive survey is the one that doesn't work.

Here is how to avoid the survey trap in 4 steps:

Step 1: Clarify Your "So What?"

Before you write a single question, finish this sentence: "When I get this information, I will use it to..."

If you can't complete that sentence with something specific and actionable, stop. You're not ready for a survey. Without a clear plan for use, how will you know what information to ask for or if you need this survey at all? Maybe what you need, you already have. Time saved! 

Some good intended uses that complete the above sentence may include: 

• "...decide whether to offer our flagship program again"

• "...show our funder that we're reaching our target demographic"

• "...identify which program components participants find most valuable"

Watch out for intended uses that aren’t actionable, feasible, or helpful. Here are some examples:

• "...understand our impact" (too vague and will likely need more than one data source)

• "...see how we're doing" (does not correspond to a decision or action demonstrating use)

• "...have data for our annual report" (data for data's sake)

  
  

Step 2: Know What Surveys Actually Do Well

Surveys are excellent for answering narrow questions about large groups when you need to know:

• Amounts or proportions (how many program participants belong to which demographic groups?)

• Frequency or quantity (how many participants completed your program?)

• Degree or scope (to what extent did the program elements meet participant expectations?)

Think of surveys as creating a rich fabric where individual threads don't stand out, but patterns across many threads do.

Example: If you want to know whether your program is reaching a key demographic as intended and how satisfied participants are overall, a survey could be a great choice. 

Step 3: Recognize When Surveys Fail

Surveys often fail when you need in-depth information better suited for conversations:

• Unique experiences and stories (why did participants drop out of your program?)

• Nuanced or context-specific experiences (how did cultural factors affect program participation?)

• Complex or sensitive topics (how did participants feel while completing your program?)

• Motivations and processes (what led participants to register for your program?)

If your goal is to tease apart the unique threads to know how individuals experienced or perceived your program, a survey could backfire. Too many open-ended questions are exhausting to complete, making survey respondents more likely to skip the questions or abandon the survey altogether. Plus, you cannot ask clarifying or follow-up questions - often essential for understanding individual experiences in-depth. 

Example: If you want to gather specific stories from participants about how they felt moving through the program or what has happened for them since completing it, consider conducting an interview. Even a 15-minute chat will gather more useful information. 

Step 4: Face the Resource Reality

Many teams turn to surveys, thinking they're easy, fast, and cheap. This is where survey regret begins. Even with AI and templates, creating good surveys requires skill and time. Before committing, be honest about what resources you have available at each of the following three stages:

Design Phase:

• Who has experience crafting questions that actually work?

• Who can create a logical flow that keeps people engaged?

• Who has the time to test the survey and incorporate feedback to ensure it works?

• What budget is available for survey software?

Implementation Phase:

• Who will manage invitations and reminders across your communication channels?

• Who will track response rates and troubleshoot technical issues?

Analysis Phase:

• Who has the time and skill to clean and analyze the data?

• Who will communicate findings to the right people at the right times?

If you plan to use AI, someone on the team needs to have the expertise to review its work. AI can create questions that sound right, but don't work. And, later on it can analyze data incorrectly. We’ve seen sketchy surveys go live because teams did not catch mistakes in time. This can be embarrassing for organizations and get teams into trouble. 

---

Gut Check

So, now that you've considered a few steps to getting survey-ready, is this still the tool for you?

✓ Survey Makes Sense If:

• You can clearly state what you'll do with the results

• You need patterns or averages across many people

• You're asking about quantities, frequencies, or degrees

• You have the resources to do it right (not just quick or cheap)

✗ Skip the Survey If:

• You can't articulate how you'll use the data

• You need deep, nuanced insights (like stories or explanations)

• You're planning to engage a small group

• You don't have time or resources to do it properly

What to Do Instead

If you need deep insights: Schedule interviews or focus groups 

If you need quick feedback: Try a brief pulse check or informal conversations 

If you need ongoing monitoring: Set up simple tracking systems 

If you're not sure: Start with a few conversations with your team to clarify what you really need to know

You're Not Alone!

Sometimes hiring help is more cost-effective than doing it yourself. Consider a consultant when:

• You have high staff turnover

• Your team relies heavily on volunteers

• You want to focus energy on what you do best

• You need to build survey capacity for the future

But consultants aren't always the answer. The wrong consultant can be more frustrating and expensive than muddling through yourself. Look for someone who asks about your target audience, plan for using information, and learning goals before diving into question design. Though not a guarantee, this will help you understand if they can build you what you need or just recycle something they've made for someone else.

The Bottom Line

Surveys can be powerful tools when used strategically. But they're not the default solution for everything. The best survey is often the one you don't create - because you already have the insights you need or have a more appropriate tool available. 

---

Still weighing your options? We'd love to chat. AND offers free 30-minute consultations with our senior evaluation consultants to help you think through what you need to make your feedback collection a success. As people who deeply love data, we're always happy to nerd out about the best approach for your situation. To book, send us an email at admin@andimplementation.ca

---

Like what you read? Subscribe to our blog to have new content delivered right to you!

#collage #artsbasedmethods #artsbased #creative #qualitative #qualitativemethods #evaluation #skillbuilding #professionaldevelopment #trysomethingnew #evaluation #programevaluation #research #methodology #learning #creativepractice #creativeconsulting #tipsandtricks #tipsforpractice #methods