Here are four tips for writing omnibus survey questions to ensure that your effort is successful:

1.  Tailor your questions to the interview mode.  Some omnibus surveys are conducted by phone, and some are done online.  Each mode requires a different style of asking questions.  For example, having an agree/disagree scale with seven points would work well for an online survey.  But on the phone, each of those points would have to be labeled with words and read out loud to the respondent.

2.  Ask the right number of questions.  If you ask too many questions, you may be defeating the cost advantage of an omnibus.  But if you ask too few questions, you will not get enough depth and leverage to tell a story.  Usually you need points of contrast or context, so you will need to ask more than just one or two direct questions.  Plan on asking five to eight questions for an omnibus survey.

3.  Keep a broad focus.  Most U.S. omnibus surveys include 1,000 respondents representing the full adult population.  Ask questions that will apply to all or most of them so that you are taking advantage of the full sample size.

4.  Ask questions that relate to your campaign. Your goal is to highlight the product or service you are selling.  Sensational, funny, or outrageous questions and survey findings will sometimes give you a quick flash of attention, but not the kind of substantive leverage that will truly carry a message into multiple forums over the course of several months.

At Versta, we will work with you on all phases of the omnibus process, including concept development, design, drafting and revising the questions, and then analyzing  and reporting the findings in a way that helps you tell the story.  Have more questions?  Give us a call — we are happy to help.

—Joe Hopper, Ph.D.

In 2004, Sandra Rathod and Andrea la Bruna conducted experiments to examine the effects of survey length on response rates, drop-out rates, respondent fatigue, speed of answering, and data quality.  In 2009, researchers at Survey Sampling International replicated the experiments and presented their findings at the recent 2010 ARF Re-Think Conference.

Their findings confirm what we know from the research five years earlier:

1.  With longer surveys, respondents get more fatigued, pay less attention, and increase their speed of response as they progress through the survey

2.  With longer surveys, data quality declines as the survey length increases (questions are skipped, open-ends are less complete, less effort is devoted to questions at the end of the survey compared to the start)

3.  With longer surveys, respondents are more likely to cheat by answering untruthfully to avoid multiple follow-up questions

What is a “long” survey?  The consistent answer from this research and other research in years past is that surveys over 20 minutes are too long.  In our experience, you can nearly always get detailed data that gives you deep insights into your questions with surveys under 20 minutes.  It is just a matter of focusing on the right questions (and only those questions) and then skillfully designing the survey instrument to answer those questions.

Need help?  Give us a call.  We would be happy to help you find the right focus and an efficient research design that delivers high quality data to answer your critical questions.

—Joe Hopper, Ph.D.

Here are some great quotes we found to keep in mind as guiding principles:

A prudent question is one half of wisdom—Francis Bacon, 17^th century philosopher and scientist

If you do not ask the right questions, you do not get the right answers—Edward Hodnett, 20^th century poet and writer

The uncreative mind can spot wrong answers, but it takes a very creative mind to spot wrong questions—Sir Antony Jay, contemporary writer

Successful people ask better questions, and as a result, they get better answers—Anthony Robbins, contemporary self-help author and motivational speaker

As noted in the newsletter, one of the best ways to truly add value to the research that you do is to listen carefully to your internal clients and formulate the right questions.  Research is all about answering questions, which is, of course, all about asking questions.  Building fancy statistical models is fun and brainy and definitely worth bragging about at your next party, but few people in your organization care much about models.  They care about what answers those statistics provide to the fundamental business questions that keep them up at night.

Any questions? Give us a call. Struggling to find your questions? Take a look at our March 2010 Newsletter.  We are happy to help you find them.

—Joe Hopper, Ph.D.

Here is a helpful hint: Do not actually terminate respondents until after you have asked all screening questions.  Suppose you’re trying to reach women over age 30 who own dogs.  You’ll need three screening questions: gender, age, and dog ownership.  First you ask gender.  It’s tempting to have men immediately discontinue, and then after asking age have all those under 30 discontinue, and so on.  Do not do this.  Ask gender, age, and dog ownership of everyone.  You don’t need men in your survey, but knowing their age and whether they own dogs may be useful to you.  Likewise, you don’t need young women, but knowing whether they own dogs may be useful.  We see two common scenarios in which this information becomes extremely useful:

1.  You complete the study, present the findings, and somebody asks, “What if we expand our market to men?”  Having complete screening data lets you estimate how many men there over age 30 who own dogs, which is an important piece of exploring whether to expand the market.

2.  You decide to expand your target group while still in the field, which means relaxing the requirements of who qualifies for the survey.  “Let’s open up our survey to allow women who are age 25 and over,” your client suddenly says.  Having complete screening data tells you in advance how many more women will qualify for the survey and helps you estimate in advance the potential cost and benefit of relaxing this criterion.

Programming a survey to do this is easy, but unfortunately most people don’t.  Give your programmer clear instructions.  The additional cost of asking all screening questions to all respondents who want to take the survey is minimal, and the benefits are potentially huge.  It’s a smarter way to screen respondents.

Need additional help?  Versta Research would be happy to review your survey and assist in any way we can.

—Joe Hopper, Ph.D.

Unfortunately these companies are teaching customers to ignore them.  Their surveys are not about listening to customers, but about internal systems for benchmarking or trending.  Worse yet, sometimes nobody even pretends to listen.  The data from these surveys are ignored because there is too much of it, nobody knows what to make of it, or because it always says the same thing.  If they listened, they would notice that their customers are increasingly irritated.  We recently helped a client analyze such data and read customer comments like: “Why are you asking me this?  Do you really care? Why are you annoying me with a survey before I have even finished?”  Irritating your customers is too high a price to pay for this kind of research.

There are more efficient, more respectful, and more insightful ways to listen to your customers with survey research.  Here are a few simple guidelines:

• Define your “need to know” information, and don’t ask anything beyond that
• Know ahead of time how much data is needed, and put a limit on how much is enough
• Determine an end point for data collection, instead of gathering data forever just because it is cheap and easy
• Keep surveys relevant so that you are respectful of your customers’ time and goodwill
• Use sampling strategies so that each customer is surveyed no more than once or twice a year

Research and insight are important, but it is critical to understand the cost of teaching your customers to ignore you.  Be smart about your research, and you will get the insight you need.

If you are unsure, give Versta Research a call and we will help evaluate your current survey program to ensure that you are getting valuable information without undue burden on your customers.

–Joe Hopper, Ph.D.

I was reminded of it when I came across this survey about fear of needles.  It reports statistics on how many are fearful of needles and avoiding medical care as a result.  But it’s a story begging for a conclusion, which the press release authors and survey designers forgot to include.  It is a good example of highlighting problems, but not offering solutions.  If you’re a motivated reader of that story, you can fill in the conclusion by reading the paragraph at the bottom about the sponsoring healthcare company’s business, but how many people read that, and how often does such information make it into a story picked up by the press?

When developing surveys for news stories and communications outreach, it is important to describe the problem, document it, measure it, and quantify it.  Statistics are powerful ways to tell stories and highlight problems.  But it is just as important to take the next step and describe unmet needs and solutions.

Here are some ways Versta has learned to focus on solutions:

• Document problems that can be improved with your product
• Measure consumer frustration with current solutions (or lack of them)
• Ask about the perceived value of benefits that your product offers

Think of it like healthcare:  We want the diagnosis of what’s wrong, but we also need the prognosis and a treatment plan, all wrapped up in a story told through numbers.

–Joe Hopper, Ph.D.

But the problem is rarely “bad respondents” – instead the problem is lazy researchers.  When people discover that the survey they just agreed to take is boring, tedious, repetitive, or too long, they either quit altogether or they stop providing good answers.

It is our job to elicit good data through good design, and to keep respondents – the lifeblood of our work – satisfied and thoughtful.  In a recent article in Alert, a publication of the Market Research Association, Jackie Lorch, VP of Survey Sampling International suggested one way to do this:  Use wording within the questionnaire that “speaks” to respondents, thanks them, and encourages them, in the same way we might do during a one-on-one interview.  Here are four specific suggestions she offers (reproduced verbatim):

1. Acknowledge the considerable effort people are giving.  For example:

The survey is quite long, and there are many questions, which you will have to think quite hard about.

2. Thank people for their effort with specific words which go beyond the usual, casual “thank you for your opinion.”  For example:

Thank you for completing the questionnaire so far. We realize that some of the questions you’ve been answering are not easy, and have required some real thought, especially that last group.

3. Remind people that we know they have a choice to take part or to complete the whole questionnaire.  For example:

Please click on the “next” button when you are ready to continue.

4. Reassure people that survey-taking is an activity in which people like them participate – but at the same time there are some aspects which may seem a bit odd.  For example:

People all over the nation and from all walks of life are taking part. The questions may seem a bit strange at first.

Versta Research believes that to serve our clients, we also need to serve our research respondents.  They are the ones who provide the information and data that we transform into stories, providing the deep insight that clients need.

–Joe Hopper, Ph.D.

Research seems to show this is not the case.  Further, a study just published in the Summer 2009 issue of Public Opinion Quarterly outlines three ways that we can improve the quality of responses in on-line surveys:

• Make the boxes in which respondents type their answers larger

• Explain to respondents that their answers can exceed the size of the box

• Tell respondents how valuable their input is, and encourage thoughtful answers to each open-ended question

These simple techniques will encourage respondents to give more complete, detailed, and thoughtful answers to our questions, yielding rich and insightful data.

–Joe Hopper, Ph.D.