Online data collection is undeniably at the heart of our business. We have over 650,000 people in our online panels and conduct over 7 million interviews a year. This is why it is absolutely vital for us to meet or even excel the respondents’ expectations when designing online questionnaires. Here are our most important design principles.

The Psychology of Survey Response

When designing a questionnaire, we always have the respondent experience in mind. This is why this article has to start with the psychology of survey response.

There is a lot of useful literature available, but we pretty much like the model developed by Roger Tourangeau (Tourangeau, R., Rips, L. C., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press). It is a general-purpose tool that helps to easily assess the quality of a questionnaire.

According to this model, the human mind has to complete four steps in order to give a suitable answer, whenever a question is asked. Let’s explain these steps by giving an example: How many Aspirin tablets do you take per month?

1. Comprehend the question by identifying cues: The respondent will have to understand what an Aspirin tablet is. If he is not sure whether a tablet computer or a medical tablet is meant, he could still deduce the right meaning from the context: the expression “taking a tablet per month” refers to medication.
2. Retrieve relevant information from memory: Once the respondent has understood the question, he can start to retrieve relevant information from his memory. When was the last time he has taken a tablet and when was the time before? If this is too hard to remember, he might take another approach and estimate how often he buys a package of Aspirin in average?
3. Integrate available information into judgment: Let’s say the respondent comes to the conclusion that he buys one package with 20 tablets per year. That would mean, that he takes about two tablets per month on average, which seems to be plausible to him.
4. Map judgment on response and give an answer: Depending on the question type, the respondent will now enter “2” into the open text field or select the right answer among the options given.

With the help of this model, you can try to anticipate the mental processes of your respondents for each and every question in your questionnaire. It really doesn’t matter whether it is an open ended question, a single select, a multiple choice or any other question type.

It’s needless to say, that completing these four steps will require concentration and eventually tire the respondents. The more questions you’ll ask in a row, the more you will use up the respondent’s attentiveness and concentration. In simple terms, survey fatigue is the product of the required mental effort and the length of the questionnaire – so you should either keep your questionnaire short or simple (or short and simple).

And that leads us to the last point: data quality. Everyone has an individual limit of cognitive burden he can cope with. If the level of fatigue exceeds the mental capacity, the brain will start looking for shortcuts and heuristics to reduce the cognitive burden. You could, for example, put less effort into the retrieval of information and use only the first thing that comes into your mind. Or you could simply select the “Don’t know”-answering option. In any case, this so-called satisficing behavior leads to less accurate answers and, in its most extreme forms, to non-sense data.

In summary, good questionnaires preserve the respondents’ attentiveness and concentration by reducing the cognitive burden of participating. Consequently they help to achieve a superior data quality. This is exactly what we are striving for when designing online questionnaires.

What you can do to improve your questionnaire

If you want to improve any given questionnaire, you can use basically two strategies: reducing the respondents’ mental effort for each single question or shorten the questionnaire in total.

Reducing the Mental Effort

Let’s start with the first strategy. In order to reduce the mental effort, you should optimize each of your questions for each of the four steps.

1. Use a neutral and comprehensible language. Be clear and unambiguous in your wording and don’t use interlaced questions or double negations.
2. Give cues to facilitate the retrieval of relevant information. These can be texts, illustrations or photos, but also previous answers from the respondent (“In a previous question, you’ve said that you don’t like the product design. What can we improve about it?”)
3. Don’t influence the respondent’s judgment. Possibly the hardest task, because every detail may have an impact. At least try to be cautious and avoid any active influence on the respondent (e.g. leading questions).
4. Provide a comprehensive set of answering options. Give a disjoint and comprehensive list of answering options or allow for honest feedback.

These improvements are pretty obvious! But there is another thing you should do: optimize the usability of your questionnaire. If the usability of your questionnaire is not intuitive, the respondent will need to put mental effort in filling it in. And that will of course harm the data quality of your survey.

Just preserving the tried and tested approaches from the past is not enough. Because you have so far never changed the layout of your questionnaire, doesn’t mean the data you get out of it is still high quality. The way people interact with the internet has changed and your questionnaire should always reflect the new usage patterns.

This is why we constantly update the layout and functionality of our survey engine. Among the many things we consider are the latest technical standards (i.e. HTML5 and CSS3), new devices and specifications (i.e. touchscreens) and changes in the usage behavior (i.e. swiping).

By the way: You will find some of the most common question types in our demo survey. Feel free to click through the different questions types and let us know, if you have any question or comment.

Shortening Questionnaires

Let’s have a look at the second strategy for improving a questionnaire: shortening it, in order to reduce the duration of the mental effort.

Unfortunately, very often online questionnaires are too lengthy. The absence of a costly interviewer makes it easy to add irrelevant questions, just for the sake of collecting more data once you’re already at it. This hasn’t really been considered a problem for a long time, but mobile questionnaires recently brought new life to this discussion.

We know that data quality usually decreases after 20 minutes (slightly depending on the topic and other factors) and that there is no effective remedy against it. Sometimes we get asked to raise the incentive. While this may motivate the respondent to complete a lengthy questionnaire, it is not suitable for boosting the mental capacity or reducing the cognitive burden. So you may preserve the willingness to participate, but incentives do not improve data quality.

So first of all, you should get rid of all irrelevant questions and battery items. That is probably the hardest task, but as a rule of thumb, you should only collect data that you will really need to answer your research question. Secondly, shorten the question texts themselves (Hint: A maximum of 140 characters like Twitter would be ideal.). If the texts are too long, respondents may not read them thoroughly and just guess the question by reading the answering options or looking for keywords.

Once you have shortened your questionnaire to a reasonable length, you should optimize the order of the questions. Assuming that the respondents’ attentiveness and concentration is still good at the beginning and decreases towards the end, the most important questions should be asked as early as possible. Conversely, put questions about the demographic background towards the end of the questionnaire, as respondents will be able to answer them correctly, even if they feel a certain fatigue.

The Case of Mobile Surveys

And that leads us to the last point. Do not exclude mobile users, just because the usability of your questionnaire is poor or the survey is too long, as this will bias the results of your study.

At first sight, mobile users have to put a higher effort in completing a questionnaire: their screen is considerably smaller, they have no keyboard available and might be in situations that partially consume their attention. The surprising truth is that mobile research very often goes hand in hand with an excellent data quality, because researchers actively design for an optimized survey experience. Following the principles above simply seems to be more plausible, when bringing the limitations of a mobile device to mind. Frankly spoken: There is no such a thing as mobile-friendly. Either a questionnaire is user friendly, or it isn’t.

Without going too much into the details here, we’d like to show you one last example, how survey design might develop in the future. Tinder is a very successful app and this has also to do with the intuitive usability of “rating by swiping”. We’ve developed a question type that works similar: if you swipe to the left you dislike the image (logo, brand, etc.), if you swipe to the right, you like it. This question type is highly user friendly and definitely requires little mental effort for filling it in.

The whole thing becomes really interesting, once you start measuring reaction times: How fast can respondents decide whether they like or dislike a concept. For the respondents no additional mental effort is required, but the researcher will definitely get richer insights on how consumers make their choices.

From this standpoint is seems to be very plausible that implicit methods will become more relevant, when more and more high quality data should be collected in shorter and shorter questionnaires.

What do you think?