We need better questionnaires

There is a lot of buzz in MRX about the need to move to shorter surveys as part of the “mobile first” design imperative. The logic is unassailable, but the execution has proven difficult, to say the least.

Let’s start by asking the question: how short? There is no consensus on that. LIghtspeed/GMI argues for an upper limit of 15 minutes, which sounds like a lot to me. At the other end of the spectrum, the folks at On Device Research say 15 questions is enough, noting that every question after that causes your completion rate to fall 5% – 10%. This seems like a pretty serious, if not unachievable, constraint.

All of this came to mind when I was reading a series of post from @lovestats on the Peanut Labs blog about questionnaire design (which she oddly refers to as “survey writing” but I already have harassed her about that). She looks at a handful of common biases associated with certain types of survey questions and provides some helpful advice on how to avoid them. This one on post purchase rationalization is a good example of why “mobile first” has been such a challenge. It’s easy to get an opinion on a product, but that opinion is not all that useful without knowing the context. Getting that context means more questions, which means that 15-question max goes out the window faster than a panel respondent straightlines a grid.

Whether by mobile, online, telephone, postal, or face-to-face, questionnaires are almost always too long in the sense that they ask questions that probably don’t need to be asked. All of us, at one time or another, have fallen prey to kitchen sink design principles. We all probably can also agree that shorter questionnaires are a must, but getting there is harder than it sounds.

Early in my career I had the opportunity to work (and take in an occasional baseball game) with Paul Sheatsley, one of the true pioneers of survey research. Paul used to say that to get good quality survey data you needed three things: a representative sample, a well-designed questionnaire, and well-trained interviewer. Interviewers have mostly disappeared in the world of MRX and sample quality is in danger of being redefined so that representativeness is no longer a requirement. Let’s try to do better with questionnaire design.

There are some good things happening. We are beginning to realize that our questions must be more succinct and answer categories limited to what can reasonably fit on a small screen. Vertical scrolling, once an anathema in online questionnaires, is making a comeback, reducing the need for server interactions that can make mobile questionnaires longer to complete than the same questionnaire online. Data fusion techniques are getting another serious look.

But at the same time there is the danger, as has happened with sampling, that we will view this as a technology problem rather than a scientific challenge. I expect we all have seen our share of software demos that serve up elaborate interfaces and self-described “engaging designs” but with perfectly lousy questions that either don’t measure what they purport to measure or are just plain confusing, ambiguous, or downright dumb. There are many ways to torture a survey respondent beyond asking too many questions.

What is missing from much of what goes on in MRX is an understanding of basic research skills. Methods may come and go, but the principles that underpin good research do not. That doesn’t mean that research has to be slow and expensive, only that it needs to be thoughtful and well informed about the best ways to go about our craft. We have the advantage of decades of high quality research and advice on questionnaire design, virtually all of it grounded in scientific principles. Yet remarkably few MR practitioners know it and use it. Or, to quote ESOMAR Director General Finn Raben who has said it more eloquently:

We must not forget that on other occasions the fundamental science behind good market research cannot always be circumvented. The core (old-fashioned, if you will), craft skills, sampling constraints, attention to detail and durability of proven research techniques are also “fit for purpose” in certain scenarios, and cannot always be emulated by newer fashions……after all, how crazy is it to buy a (new) pair of jeans that have been stone washed until they are almost ready to fall apart?

Yes, MR is changing but I also think we are beginning to see that those changes are more about evolution than revolution. As often as not, the challenge is to adapt rather than invent.


One thought on “We need better questionnaires

  1. Point taken, Reg. Common language isn’t always better than industry language. 🙂
    And I’m absolutely with you on ensuring we retain the fundamentals of science in our work. Just because we have so much cool technology that makes research happen more quickly doesn’t mean it’s also going to make our research more valid and reliable. Science first, cool second.

Leave a Reply

Your email address will not be published. Required fields are marked *