Designing a Survey Part 1: Asking Good Questions
Updated: Oct 19
Five red flags that make people skip, quit, or guess, and how to fix them
Part 2 of our survey series: Should You Create a Survey? · Designing a Survey Part 1 · Part 2 · Delivering a Survey
Last updated: October 2026
So, you decided to make a survey (and if you’re still questioning whether it’s the right choice, you can start with our first post in this series, Should You Create a Survey?).
Crafting good questions isn’t as easy as writing down what you want to know. You are likely managing competing interests from your board, funders, team, and community. Perhaps you feel pressured to measure things that will result in glowing reviews. Or, you might find yourself handcuffed to survey software that only allows for a few types of questions to begin with. And, maybe you are muddling through this survey off the side of your desk, because it isn’t actually what you do!
Limited resources, time crunches, and pressure often result in questions that are clunky, confusing, and burdensome for participants. The result? Participants skip through questions or abandon the survey. The survey falls apart when questions aren’t answerable.
So what makes a survey question a good question?
The short answer: good survey questions are short, specific, respectful, and easy to answer. Miss any of those, and people skip questions, quit early, or make things up.
Don’t waste time and energy on questions that don’t work. Here are the 5 most common mistakes we see in survey questions and easy fixes to make before the survey goes live.
Alienating questions: prioritize need-to-know respectful questions so people feel seen
Forced choices: offer a “not sure” option so answers reflect reality
Chaotic lists: order options logically so people find their answer fast
Making people work too hard: keep it intuitive so you get real answers, not guesses
Marathon Surveys: stick to 10 minutes max so people finish
How long should a survey be?
Most people generally won’t give you more than 10 minutes of their attention. Use that time well by prioritizing the minimum viable questions you need to satisfy the purpose of your survey.
Pro tips:
Remove repetitive questions (e.g., asking what country and what continent people live in)
Use branching or display logic instead of multi-step or dependent questions (e.g., if you answered "Yes" above…)
Add page breaks after similar questions to show progress
Our next post, Designing a Survey Part 2, explores strategies for structure and flow to further reduce survey chaos.
Alienating questions: prioritize need-to-know respectful questions so people feel seen
Stereotypical, invasive, or harmful questions (especially when it comes to demographics) can cause harm to, break trust with, and result in unreliable data from participants. Data isn’t valid if it's harmful.
How should you ask demographic questions in a survey?
Consider how it might feel for someone to misrepresent themselves, reduce their identity to a tick-box, or disclose information that isn’t safe or appropriate for the context. Prioritize respect in your survey by taking the following measures.
Prioritize need-to-know questions
Include only what you need to know to make your survey useful (e.g., make improvements to a program/service).
Ensure that questions, when answered altogether, do not make it possible to guess who responded.
For example, if you were to conduct a survey of consulting firms in Canada, it would only take 3 questions for us to become identifiable: field of practice, location, sexual orientation. As a registered queer-owned evaluation firm in Vancouver, BC someone could very easily Google or guess who we are. Many people are surprised to learn that just a few questions can single someone or an organization out from a large sampling pool. And, that is before getting to open-ended questions where someone might over-disclose or share a unique circumstance.
Tell people how you'll use the information
Write a brief blurb about why you are asking and how you plan to use what they share. This generates goodwill and helps mitigate sketchy vibes. Here's what we use: We have a few questions about your identity that will help us understand if we are meeting the needs of our community and ensure our programs have equitable benefits.
Ensure you can analyze what you ask about
We have lots of messy, fluid, and intersecting identities. So, it is important to understand what you plan to do about that at the analysis stage. Can you engage with how people choose to identify or do you need to collapse into more manageable boxes for ease or just to tick a box for a funder? Take sexual orientation for example. A literal rainbow of possibilities for people to identify with. Do you have the time, resources, capacity, and skill to keep the nuance in your analysis? Or are you planning to collapse into smaller umbrella labels or just a binary (straight/other). If you are going to collapse categories anyway, this is a signal that you might need to pause and revisit if this is a "need to know" question, how you plan to use this information, and what you tell people about intended use. Two ideas for how to move forward:
Skip the question. Tokenizing or misrepresenting people behind the scenes is not a great practice.
Present the minimum viable categories and tell people why. This way, people know what's happening with their answer and can choose to engage. For example, "For X funder, we need to report the number of program participants who are Y. Do you identify as Y (yes, no, prefer not to say)?"
Avoid "other" boxes
It is not a good idea to use an “other” box when there are limited or binary options to begin with (e.g., How do you identify, “Male”, “Female”, or “Other”). This can be disrespectful and damaging to groups that are not meaningfully represented in the limited categories. Instead, ensure you have enough options for your participants to choose from (as per your analysis plan). Then, give the option "I identify as..." instead of a box that literally does the work of othering someone. Again, if you ask for more information (especially if it is vulnerable or personal) you are accountable for that data.
Pro tip:
We often see organizations ask for exhaustive or outdated demographic categories because the funder requires it. Push back gently. Most funders care more about ethics, program effectiveness, and sustained trust within the community than exhaustive demographics. If you have made your case and they won’t budge, consider making the mandatory question(s) optional and ensure you leave an open-field option for people to self-identify instead.
Not sure how to push back on a funder’s requirements or move forward with reasonable demographic questions? Nonprofits can talk it through in Data Strategy Office Hours and get unstuck in an hour. Evaluators navigating client or funder expectations can bring it to Evaluation & Research Advising, so you aren’t making big decisions in an echo chamber.
Forced choices: offer a “not sure” option so answers reflect reality
Often, people don’t know, don’t care, or don’t remember. While it might be important for you to know if they have a preference on program timing, think your mandate is relevant, or feel represented by your leadership, they might not have enough information to give an honest answer. Or, they might not care. So, forcing a choice could result in unhelpful data.
For example, I (Maya) recently did an online survey asking if there is an airline I would never use. The survey forced a choice between a short list of a few airlines. The truth? I'm not brand-loyal, so I don't care. The action? I picked one at random because it was a mandatory question. That result? Invalid data that belongs in the trash.
How can you give people a non-committal answer in a survey?
Here are a few ideas to help people answer honestly by giving them the option to feel "Meh."
Example | Type of Question | Option to Add |
 | Yes or No | Maybe |
 | Matrix or Scale | Unsure or Not Applicable |
 | Preference Drop-Downs | Don't know or Don't care |
Chaotic lists: order options logically so people find their answer fast
If you are asking people to choose from many available options, make it easy for them to find their choice in a list or drop-down. Example: if you are asking people which province or state they are from, you could order alphabetically, or geographically (e.g., east to west or north to south). Or, if you are asking which event(s) they attended, share the options chronologically. This shows that you thought about their experience.
Making people work too hard: keep it intuitive so you get real answers, not guesses
People are volunteering their time and insights. So it should be easy and intuitive for them to complete your survey. If they have to reach for a calculator, check their tax returns, flip through their calendar, write you an essay, or answer skill-testing questions to participate (all things we have seen requested in real surveys), they won’t. That’s why data from labour-intensive questions are notoriously unreliable.
People will make some s*** up and move on.
Example: we recently came across a survey asking people to:
Assess their familiarity with an organization’s mandate on a scale of 1-5
Describe the organization’s mandate in their own words in an open-text box
On the next page
Review the organization’s actual mandate (this included ⅓ of a page of text to read)
Rate the relevance of the organization’s mandate on a scale of 1-5
Summarize the mandate (now having seen it) in their own words
Why? If you want people to review and comment on information, make it easy. Give them a short blurb (like your mission or vision statement) and ask one or two direct questions about it. Simple, direct, and no trick questions!
Or, if you want to see what people gained from your program, ask a simple question about it like, “What was one thing you learned from X program?” or “Since completing X program, which of the following techniques have you tried?”
Reality Check
Good survey questions feel effortless to answer, but take real skill to craft. Even with these guidelines, expect to spend 4 to 5 hours crafting questions and 2 to 3 hours testing and refining. If that feels overwhelming alongside your other responsibilities, you're not alone.
Before your survey goes live, ask yourself (and your colleagues), would you honestly complete this survey if it showed up in your inbox? If the answer is no, your participants won't either.
Consider reviewing each question in your next survey against these red flags. And, we always recommend the buddy system. Ask someone who matches your target participant profile to beta-test the survey. For example, if your program serves working parents, don't test with colleagues who don't have kiddos (their experience won't match your participants' reality).
Don't have the time to do it right? If you're a community-serving org, we can take this off your plate and hand you back a ready-to-use survey with our Light Touch Consulting. Or, we can stress test your existing survey with Data Strategy Office Hours for community-serving orgs and Evaluation & Research Advising for fellow evaluators.
Pro tip:
If you can’t find anyone who matches your participant profile to beta test the survey, ask someone you trust who is a tough critic and won’t shy away from telling you, “This question doesn’t make sense” or “I don't know how to answer this.” We often ask our parents to review our surveys because they answer questions differently than we intended, highlighting the flaws in our logic.
When to call for back-up?
If your team or partners are debating what to measure, you're getting pushback on question wording without clear solutions, or if the survey is taking longer than expected to finalize, those are signs that an outside perspective could save you time, money, and headaches.
Survey giving you grief? We can build it for you, train your team to build it themselves, or partner with you on something bigger. Explore our services, or tell us about your survey woes.
Ready for the next step? Explore our next post: Designing a Survey Part 2
Subscribe to our blog to have new content delivered right to you!
Comments