The article provides evidence that online surveys offer better data quality compared to telephone surveys and other modes of “oral” interviewing. Data from the computer surveys were superior in terms of:

1. Validity. The researchers used two different measures of the same thing, and found higher consistency between the two measures for respondents taking the computer-based survey. This is known as concurrent validity.
2. Satisficing. Respondents taking the computer-based survey worked harder to provide accurate answers, which showed up in more variation and differentiation among their answers, and fewer response-order effects.
3. Social desirability bias. Those taking computer surveys were more likely to share their true feelings about a controversial topic rather than reporting feelings that would make them look more sympathetic or politically correct.

The findings reinforce those reported in another POQ article published at the end of last year, which outlined the relative strengths and weaknesses of phone versus online surveys. They will also no doubt feed the ongoing controversy about usage of online survey panels (See “How Good Are Online Survey Panels?” and “Practical Statistics vs. Theoretical Statistics”).

Which should you do, a phone survey or an online survey? It depends on your objectives. We would be happy to help you think through your options and the strengths of each. Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

—Joe Hopper, Ph.D.

In 2004, Sandra Rathod and Andrea la Bruna conducted experiments to examine the effects of survey length on response rates, drop-out rates, respondent fatigue, speed of answering, and data quality.  In 2009, researchers at Survey Sampling International replicated the experiments and presented their findings at the recent 2010 ARF Re-Think Conference.

Their findings confirm what we know from the research five years earlier:

1.  With longer surveys, respondents get more fatigued, pay less attention, and increase their speed of response as they progress through the survey

2.  With longer surveys, data quality declines as the survey length increases (questions are skipped, open-ends are less complete, less effort is devoted to questions at the end of the survey compared to the start)

3.  With longer surveys, respondents are more likely to cheat by answering untruthfully to avoid multiple follow-up questions

What is a “long” survey?  The consistent answer from this research and other research in years past is that surveys over 20 minutes are too long.  In our experience, you can nearly always get detailed data that gives you deep insights into your questions with surveys under 20 minutes.  It is just a matter of focusing on the right questions (and only those questions) and then skillfully designing the survey instrument to answer those questions.

Need help?  Give us a call.  We would be happy to help you find the right focus and an efficient research design that delivers high quality data to answer your critical questions.

—Joe Hopper, Ph.D.

“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”

Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters.  Why?  Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works.

Their conclusion is particularly surprising because although statistical inference based on probability sampling has a solid theoretical underpinning, in practice pure probability samples are almost never achieved.  Nearly always we are faced with low response rates and non-response biases.  We work around this in practical ways (not always supported by extensive theory) to weight the data, adjust for biases, understand the sources of biases, add caveats to our findings, and so on.  And sure enough, our efforts tend to work, so we keep doing it.

There is substantial data to show that carefully managed non-probability online panels can be used to estimate certain population parameters depending on the nature of the study and how exact those estimates need to be.  Someday our academics and social theorists will help us understand why.  Here’s our theory: Social bodies are not composed of individual units (the assumption underlying inferential statistical theory)  but nodes of networks such that non-random entry points to measure social forces can provide as much information as a random selection of individuals.  We’ll leave it to the future theorists to tell us if we’re right.

In the meantime, there is a lot of research for which using non-probability online panels makes good sense (which the AAPOR report acknowledges), so we’ll keep doing it and we’ll keep extending it into new areas even if the theory can’t keep up.  And we’ll keep refining our techniques based on experience and practice to make it work better, and we’ll keep thinking about why it works and lending that insight to the work we do for you.

If you need help or expertise designing and executing your research based on probability or nonprobability samples, online panels or phone, let us know.  We know our statistical theory.  More importantly, we know our practical statistics.  We can help you sort through it all to ensure rigorous research and practical results.

—Joe Hopper, Ph.D.

The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report.  It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.

Here is a summary of their conclusions and recommendations (quoted verbatim):

• Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.
• There are times when a nonprobability online panel is an appropriate choice, as there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional methods.
• Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results.
• Panel companies can inform the public debate considerably by sharing more about their methods and data.
• Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.

AAPOR’s first conclusion is provocative, and sure to stir up a great deal of controversy.  It runs counter to what most research firms are doing these days.  In an article next week we will consider this conclusion in more depth, and offer a more nuanced perspective than AAPOR.  In the meantime and if you’re interested, you can read the full report.  If you wish to discuss it or need help interpreting it, please let us know as we would be happy to help.

—Joe Hopper, Ph.D.

Phone vs. Online Surveys: Strengths & Weaknesses

As reflected in the table above, the article is rather technical.  But there are two key summary points worth learning from the study:

One, online surveys fielded through panel providers may not fully represent the population.  But we know from other research that they can come close, so the key is to understand the sources of potential bias and adjust for them as needed.

Two, online surveys fielded through panel providers may result in more accurate data than other types of surveys.  There are two reasons for this.  First, online panelists are more engaged in the process and interested in the topics, so they exhibit less cognitive laziness.  Second, online surveys feel more private, so respondents provide answers that are more honest.  This is in contrast to phone surveys in which people usually want to be nice and therefore provide socially desirable answers, even if it’s not how they really feel.

Which should you do, a phone survey or an online survey?  And what type of sampling should you employ, probability or non-probability?  It depends on your objectives.  Each of these issues can matter a lot or not at all depending on the specific questions you are trying to answer.  We would be happy to help you think through your options and the strengths of each.  Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

-Joe Hopper, Ph.D.

Looking back, here are what we consider to be the five biggest changes that shaped current challenges faced by market research and opinion polling:

1. Online data collection using sample panels became the dominant mode of surveying, displacing a huge industry in telephone surveying.  Plus there has been a dramatic decline in households owning landlines, which makes the ideal of probability sampling extremely difficult to achieve.
2. Survey technology made data collection easy and cheap which means that a lot of market research became a commodity, resulting in industry consolidation and need to find new ways of adding value.
3. An explosion of too many surveys, which are everywhere, powered by simple tools like Survey Monkey.  Along with this there has been a dramatic decline in response rates, bringing the issue of quality to the fore like never before.
4. Statistical and mathematical expertise grew in demand.  There is so much data and information, and it is so easy to get.  But who knows what to do with it all?
5. “Audience-ready” reports became more valuable.  Newspapers laid off staff and market research departments downsized, demanding a level of professional involvement to ensure that research reports are easy to read, interpret, and ready to deliver to the audiences who need them most.

One product of these five trends is Versta Research’s focus on high-level expertise and academic brainpower so that you get smart, creative, and flexible market research.  Another is the skill and commitment we have in helping you communicate research to managers, clients, reporters, and the audiences you need to reach.

In our next post we’ll highlight what we predict will be the five biggest trends that will shape market research challenges in the decade to come.

–Joe Hopper, Ph.D.

So I’m always encouraged with yet more evidence that people do tell the truth on surveys.  Studies continue to show that surveys and polling are effective because (1) you need to ask only a small representative sample of people what they think to make statistical conclusions about the whole, and (2) the vast majority of people will tell you the truth about what they think.  It would seem that people are less concerned about privacy and keeping their opinions to themselves  than they are about being heard and making a difference.

The latest data demonstrating this was published in an industry magazine from the Marketing Research Association.  If a person volunteers, or is asked, to take a survey multiple times, what happens?  The vast majority will give you the same answers each time they take the survey, even on questions that ask for subjective ratings on 10 point scales.  If a person were lying or just racing through a survey with random answers to collect incentives, this would not happen.  It turns out that most people give thoughtful, truthful answers each time.

Of course you do not want the same person included in your survey more than once, and a good research vendor will do the careful work of ensuring it does not happen.  But when it does happen, it is fascinating that our efforts to get to the truth are so resilient. The study was done by Paul Johnson, a senior analyst at Western Wats.  If you would like a copy of the study, let me know and I will send you a copy (unfortunately the MRA keeps this content under lock and key, so I cannot include a web link here).

A corollary of this research will be a topic for another day, but here’s the gist:

If you want to know the truth about your product or your customers or your competition, just ask, because most people will tell you.

Versta Research can help you with what to ask, how to ask, whom to ask, and how many to ask, and chances are you will be delighted with how much insight you can gain through a systematic and rigorous effort of asking.

–Joe Hopper, Ph.D.

Unfortunately these companies are teaching customers to ignore them.  Their surveys are not about listening to customers, but about internal systems for benchmarking or trending.  Worse yet, sometimes nobody even pretends to listen.  The data from these surveys are ignored because there is too much of it, nobody knows what to make of it, or because it always says the same thing.  If they listened, they would notice that their customers are increasingly irritated.  We recently helped a client analyze such data and read customer comments like: “Why are you asking me this?  Do you really care? Why are you annoying me with a survey before I have even finished?”  Irritating your customers is too high a price to pay for this kind of research.

There are more efficient, more respectful, and more insightful ways to listen to your customers with survey research.  Here are a few simple guidelines:

• Define your “need to know” information, and don’t ask anything beyond that
• Know ahead of time how much data is needed, and put a limit on how much is enough
• Determine an end point for data collection, instead of gathering data forever just because it is cheap and easy
• Keep surveys relevant so that you are respectful of your customers’ time and goodwill
• Use sampling strategies so that each customer is surveyed no more than once or twice a year

Research and insight are important, but it is critical to understand the cost of teaching your customers to ignore you.  Be smart about your research, and you will get the insight you need.

If you are unsure, give Versta Research a call and we will help evaluate your current survey program to ensure that you are getting valuable information without undue burden on your customers.

–Joe Hopper, Ph.D.

Why would a market research firm use such a primitive tool for its own feedback?  Are we like the cobbler who can’t afford shoes for our own children?

No.  The reason is that our feedback survey is exceedingly simple.  We ask just two questions:  What was valuable about our work?  What do we need to improve?  (Actually, there is a third question – we’ll leave you in suspense until we complete our first engagement with you.)  We do not ask rating questions because we have no need to quantify the information, and we do not need to quantify because we do not have tens of thousands of clients.  All we need to know – from each client, personally and individually — is what we should keep doing, and what we need to fix.

Versta always uses and recommends the tools that get the job done most effectively.  While there are a lot of terrible do-it-yourself surveys on Survey Monkey, it can be a decent tool if you know how to use it and your needs are simple.

If you’re not sure about your needs, or would like some help with your own DIY project on Survey Monkey, give us a call and we would be happy to lend our expertise.

–Joe Hopper, Ph.D.

In years past, data collection was a slow, complicated, and expensive task.  If you had just a few survey questions, you could share the cost of data collection by cooperating with others who had just a few questions.  That was the birth of the omnibus.  Today, however, most surveys are conducted online, and the data collection process is automated, fast, and inexpensive.  The cost of a custom survey with just a few survey questions, just for you, fielded exactly when and how you want it,  is often no higher than an omnibus.

Sometimes an omnibus survey is the best way to go, but it all depends on the issues you’re trying to understand and the audience you’re trying to reach.  If you think an omnibus survey might be right for you, here are Versta’s recommendations about how to proceed:

• Focus on the issues you’re trying to understand and the budget you have to work with

• Ask us to recommend all options that answer your questions within your budget

• Do not worry about whether the study is an omnibus or not — an omnibus survey is a money-saving technique for the researcher that should be invisible to you

• If an omnibus survey makes sense, work through a firm like Versta Research – even if we field it through a big-name provider for you, it will cost less, and we can design it in a customized way that helps you turn data into stories

–Joe Hopper, Ph.D.