How to write good survey questions
Effective survey design is crucial for gathering reliable and actionable data. Well-designed questions significantly impact response rates and the quality of data collected, while poorly crafted surveys can lead to inaccurate results, rendering the entire research process redundant.
This article will guide you through the key elements of writing good survey questions, covering:
Types of survey questions
The way you frame a question determines the kind of answers you receive and helps you achieve your research objectives. The primary types of survey questions are closed-ended and open-ended. Both types have their strengths and weaknesses and perform in different ways.
Closed-ended questions
Closed-ended questions provide respondents with a set of predefined answer options. These questions are the foundation for quantitative surveys, making data analysis easier.
One key advantage of closed-ended questions for researchers and clients is that they are easier to administer to large samples and when asked to a robust number of respondents, they provide measurable, statistically significant data. This data is then easier to analyze, group, compare, and illustrate trends and percentages over time. For respondents, these questions should be simple to understand and answer quickly, which can increase the overall response rate.
However, closed-ended questions also have their drawbacks. They limit respondent input, which restricts the depth of insights that can be gathered. Despite this, they are valuable for obtaining clear, quantifiable data.
Examples of close-ended questions
Dichotomous questions: Offer two possible answers and are ideal for straightforward, yes-or-no responses.
Q: Have you heard of this brand before today?
- Yes
- No
Multiple choice questions: Allow respondents to choose multiple options from a list, providing a broader understanding of preferences and opinions.
Q: Which of the following brands have you heard of before? (Select all that apply)
- Brand A
- Brand B
- Brand C
- None of these
Scale questions: Offer options that are incrementally related to each other at either end of the scale, offering insights into respondent attitudes.
Q: To what extent do you agree with the following statement? "This brand is trustworthy."
- Strongly agree
- Tend to agree
- Neither agree nor disagree
- Tend to disagree
- Strongly disagree
Open-ended questions
Open-ended questions allow respondents to answer in their own words without predefined options. They are used to gain detailed and descriptive insights. These questions are particularly useful when seeking in-depth answers that reveal the nuances of respondents' opinions and experiences.
The advantages of open-ended questions include the ability to capture unrestricted opinions and thoughts, more creative and expressive responses, and greater freedom of expression. These responses can provide valuable qualitative insights that enrich the overall findings of a survey.
However, open-ended questions also have their challenges. They are more time-consuming for respondents to answer and can be difficult to analyze and quantify. Despite these challenges, tools like word cloud visualizations can help make sense of the qualitative data collected. Additionally, there is the option to include 'other-specify' options, allowing respondents to add their own answers.
Examples of effective open-ended questions include, "Why do you prefer this product over others?" and "What improvements would you like to see in our service?" These questions encourage respondents to share detailed insights that might not emerge from closed-ended questions alone.
Combining closed-ended questions with open-ended follow-ups can provide richer insights. For instance, you could ask, "How likely are you to recommend this product on a scale from 0-10?" followed by "Why did you give this rating?" This approach allows respondents to elaborate on their quantitative responses, adding valuable context and detail.
Demographic questions
Demographic questions gather information about respondents' characteristics, such as age, gender, and income. These questions help categorize respondents and ensure survey results are representative.
Common types of demographic questions include those asking about age, gender, income, education level, and marital status. For example, you might ask, "What is your age?" with answer options spanning various age ranges, or "What is your highest level of education?" with options ranging from "High school" to "Doctorate."
At YouGov, we hold extensive information about the demographics and profiles of our 27+ million registered panel members, so there is usually no need to ask additional demographic questions. However, if we do, respondents are reminded that their answers will never be individually analyzed and all responses are kept confidential.
Demographic information is crucial for understanding the profile of your respondents and ensuring your survey results are inclusive and representative. This data helps break down a large group of people into smaller segments based on their characteristics, aiding in crafting targeted marketing strategies, for example, and ensuring the fairness and accuracy of the survey findings
Survey design best practices
Crafting effective survey questions requires a deep understanding of the survey's goals; without clear objectives, it's challenging to create questions that will yield meaningful and actionable insights. By adhering to these best practices in survey design, we can ensure that questions not only avoid bias and ambiguity but also directly address key areas of interest, leading to more reliable and valuable data.
Clarity and simplicity
When designing a questionnaire, clarity and simplicity are paramount. Questions should be clear, simple, and unambiguous, avoiding jargon and emotive language. Providing suitable instructions helps guide respondents on how to answer questions correctly.
Neutral language
Using neutral language is also important to avoid leading or loaded questions. This ensures unbiased responses and that all respondents understand the question in the same way. Randomizing response options can help minimize order bias, further enhancing the neutrality of the survey.
Relevance
Relevance is key to maintaining the respondent's engagement and ensuring the questions align with the survey's objectives. Avoid giving respondents information they don't need and use realistic timeframes e.g. if asking respondents to recall past behavior. Applying routing where necessary ensures questions are only presented to relevant respondents, balancing the need for comprehensive data with respondent experience.
Balanced answer choices
Providing balanced and exhaustive answer choices for closed-ended questions ensures that all possible responses are covered. For balanced scales, there should be an equal number of positive and negative answer options with a midpoint. Likewise, there should be a mixture of positive and negative statements. If using bands (e.g., frequency scales), ensure they are continuous and not overlapping. Including opt-out options like "Don't know," "None of these," and "Not applicable" can be important for capturing accurate data without forcing respondents to choose an inappropriate answer. "Prefer not to say" should be added for sensitive questions.
Questionnaire structure
Structuring your questionnaire logically is also crucial. Start with general questions and gradually move to more specific ones. Include an introduction to set respondent expectations, screener questions to ensure respondents meet the criteria, a main body with a mix of question types, and a thank-you page to conclude the survey. The order in which questions are asked can also (unintentionally) affect the way in which respondents answer subsequent questions. Providing a time estimate for respondents can help set their expectations and increase completion rates.
Cosmetic style
In addition to the content, it's crucial to consider the visual style of online surveys, especially for respondents using mobile devices. Grid questions (tabular questions where respondents evaluate row options using the same set of column choices) can be highly efficient for collecting extensive information. However, excessive use of grid questions can lead to respondent fatigue. Balancing efficiency with engagement is key to maintaining the quality and reliability of survey responses.
Common mistakes to avoid
When designing survey questions, it's important to avoid common pitfalls that can confuse respondents or influence their answers.
Ambiguity
Avoid vague or ambiguous questions. If anyone testing the survey finds a question unclear or interprets it differently, it's likely ambiguous. Use factual and simple preambles to clarify the context. Vague questions lead to vague answers, resulting in unactionable findings and insights.
Double-barrelled questions
Double-barrelled questions ask respondents two different things simultaneously, leading to unreliable data. This can push respondents to skip the question or worse answer inaccurately. Always ask one question at a time to ensure clarity and accuracy.
Complexity
Avoid overly complex or technical language. If respondents don't understand what is being asked, they can't provide accurate answers. Keep the language simple and clear to ensure everyone can respond appropriately.
Assumptions
Avoid assuming respondents have certain knowledge or opinions. Only ask questions that respondents can realistically answer. Where necessary, include explanations to ensure everyone has the same base level of understanding when answering a question but keep text factual and simple.
How YouGov can help
Designing effective surveys can be challenging, but YouGov offers tools and services to make the process easier and more efficient. Whether you prefer a hands-on approach or want expert assistance, YouGov has solutions to meet your needs.
Self-serve surveys
YouGov: Self-serve allows you to create and launch surveys easily and receive answers within 24 hours. The intuitive interface guides you through the survey creation process, ensuring you follow best practices for question design. With access to our proprietary panel, you can gather responses from your target audience in no time.
Serviced surveys
For those who prefer a more hands-off approach, YouGov: Serviced provides expert assistance throughout the survey process. Our team of experienced researchers will help you design, deploy, and analyze your survey, ensuring you get the high-quality data you need. This service is ideal for complex projects or when you need additional support to meet your research objectives.
By leveraging YouGov's expertise, you can ensure your surveys are well-designed, engaging, and capable of delivering actionable insights. Whether you choose the self-serve option or prefer to work more closely with our team of experts, we are here to help you achieve your research goals.
Lisa Ball is Research Quality Manager at YouGov.