By Patty Yanes
As a follow-up to our webinar, “What you really need to know about complex survey design” Applied Marketing Science (AMS) Senior Manager Patty Yanes answers a few questions submitted by viewers during the session. The webinar touched upon key tips and tricks for complex survey design including: accessing hard-to-reach respondents, writing clear and concise questions, and ensuring proper organization and structure.
Question: When you have a lot of content in your surveys, how do you make sure you don’t unintentionally add any biases to the data?
Answer: When you have many different questions in the survey and are looking to get accurate and actionable answers to all, you should do what you can to avoid any implicit biases. One method is to use a block design and group the questions into items that go together. This allows you to rotate the blocks of questions to avoid any order bias (respondents being influenced by something they have seen previously in the survey). You can also use filter questions where applicable to ensure that only people who have experience with a certain topic see the corresponding question. In addition, there are a few tools you can use to eliminate any scale biases that may occur, particularly with international audiences, including normalizing, rescaling, and/or weighting the data on the backend.
The hardest part is just recognizing that there may be bias introduced into your survey in the first place. Once you realize that, you can be more cognizant of areas to reduce or eliminate the bias.
Question: Detailed, complex surveys can get to be pretty long. What is the ideal length of a survey and how do you best measure survey length?
Answer: There’s a lot that goes into determining length in complex survey design. You typically have a number of goals to meet including a) solving the problem at hand, b) trying to please stakeholders, and c) working to maintain a positive survey-taking environment for the respondent. These goals can frequently compete with each other. The ideal length of a survey can be measured in multiple ways, but we typically gauge length by the number of questions in the survey, and survey completion time.
Ideally, a web-based survey should be no more than 15 minutes long, but it can be difficult to quantify how many questions that means. We typically suggest 3-5 minutes for screening and ensuring you have the right respondents, 6-8 minutes for the bulk of the important, “money” questions, and a few minutes for follow-up questions and demographics. This, of course, is all dependent on the complexity of the topic and how long each question is. Try to include progress bars and indications of survey time within the survey to make respondents feel their time is being respected.
Question: If the survey is getting too complex, why don’t you just split it into multiple surveys?
Answer: Sometimes looking at a complex survey can become exhausting and it can seem like two or even three surveys combined in one. It might feel overwhelming, but it’s typically best to keep all the questions in one survey. Simply put, the benefits outweigh the headaches. Conducting one survey, as opposed to two or three different surveys, gives you one sample of respondents. With this, you can compare all of the survey answers to each other, because they were answered by the same group of people. Additionally, the more answers to questions you have per respondent, the better understanding you can gain for analyses like market segmentation and concept testing. Finally, cost alone is typically a good reason to execute one survey because it reduces the amount of total sample needed – and sample can be pricey! In general, though, if you feel like the survey is starting to go down too many different paths, getting too long, or targeting several different types of people, it might be time to consider splitting into multiple surveys. Remember, the goal is to have a survey that gives you actionable answers in the end, not one that gives you headaches and no answers to your problems.
For more information on understanding complex survey design, watch Patty’s webinar on demand: “What you really need to know about complex survey design".
If you have questions on complex survey design or other topics, feel free to reach out to Patty and she will be happy to help.
Tags: Recruiting Respondents