Congratulations on creating your survey! But before you dive into collecting responses, it’s crucial to prioritise testing. Effective testing can be the difference between success and failure in your survey research project. Let’s explore five key tests you should conduct before fully launching your survey.
1) Question Test
The importance of testing your questions cannot be overstated. Start by ensuring that your questions are functioning as intended. Review your original questionnaire or survey document to ensure question types, answer options, and question text is correct and matches. Does each question type align with its purpose? For example, should a single-choice question actually be a multiple-choice question? Additionally, verify if the “none of the above” option automatically excludes other options.
2) Direction Test
While basic surveys follow a linear structure, others may require routing based on respondent demographics/groups or specific answers to a previous question. Testing randomisation & rotation is also important to reduce bias and sometimes survey fatigue. To conduct a ‘direction’ test, create an Excel or Google Sheet listing the routes or audiences that require different survey routes. You and your colleagues should rigorously test the routes as if they belonged to the intended demographic/group. Are they encountering the expected questions? Should certain respondents be excluded? Provide a comments column for individuals to report issues encountered during the test (this isn’t just limited to routing; this could be randomisation, question flow, user experience or device issues). Alternatively, some survey platforms allow users to apply comments directly within the survey but I prefer to get an overview of things via a spreadsheet – that way I can mark if they’re done or not. Remember to thank respondents on the disqualification page, as you might want to engage them in the future.
3) UX Test
This is the sense check. Put yourself in the respondents’ shoes and evaluate their experience. Make sure at the start (or in prior communications) the purpose of the survey is clearly conveyed, and respondents understand their privacy rights and data protection. Consider the engagement level and interest generated by your question types. Does the survey start with easy qualifying questions and gradually progress to more complex ones? Ensure that answer options are comprehensive and aligned with respondent’s potential answers. Can participants skip open-ended questions if pressed for time (the longer the survey, the more likely respondents will disengage). Additionally, consider providing incentives to boost response rates and encourage more considered answers. It’s essential to ensure that your survey is also accessible to all potential respondents, including those with disabilities. Test the survey’s compatibility with screen readers, check for appropriate colour contrast, and verify that it can be navigated using keyboard-only inputs. By conducting accessibility testing, you promote inclusivity.
4) Device Test
With surveys being accessible on both mobile and desktop devices, it’s crucial to test their functionality across platforms. Most survey platforms offer device-specific views, allowing you to assess the respondent’s experience on desktop, mobile and tablets. Keep in mind that certain question types, such as card sorts or highlighting questions, may not work optimally on mobile devices. According to Kantar, 40% of survey starts occur on mobile devices, yet approximately 67% of surveys aren’t compatible or user-friendly on respondents’ preferred devices. Consider the question types you’re using and weigh them against the device profile of your sample.
5) The Soft Launch Test
After conducting thorough tests, you may believe you’re ready to go. However, it’s essential to perform a soft launch before reaching your full sample. A soft launch involves sending the survey to a limited number of respondents, typically 5-10% of your sample. Some may carry out a second soft launch, perhaps with 20% of the sample. A soft launch allows you to identify any design errors or routing issues that may have been missed during testing. Keep an eye out for high drop-off rates, which could indicate survey fatigue or technical problems. If available, consider running “dummy” responses using the survey platform’s features. Remember, real-world responses can present unforeseen challenges, making the soft launch an integral part of the process. Use the insights gained from the soft launch to make any final adjustments before launching to your full sample. Remember to schedule this step into your workflow.
If you have a research project in mind, get in touch at elliot@ferninsight.com or tell me more about your project here