The article provides evidence that online surveys offer better data quality compared to telephone surveys and other modes of “oral” interviewing. Data from the computer surveys were superior in terms of:

1. Validity. The researchers used two different measures of the same thing, and found higher consistency between the two measures for respondents taking the computer-based survey. This is known as concurrent validity.
2. Satisficing. Respondents taking the computer-based survey worked harder to provide accurate answers, which showed up in more variation and differentiation among their answers, and fewer response-order effects.
3. Social desirability bias. Those taking computer surveys were more likely to share their true feelings about a controversial topic rather than reporting feelings that would make them look more sympathetic or politically correct.

The findings reinforce those reported in another POQ article published at the end of last year, which outlined the relative strengths and weaknesses of phone versus online surveys. They will also no doubt feed the ongoing controversy about usage of online survey panels (See “How Good Are Online Survey Panels?” and “Practical Statistics vs. Theoretical Statistics”).

Which should you do, a phone survey or an online survey? It depends on your objectives. We would be happy to help you think through your options and the strengths of each. Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

—Joe Hopper, Ph.D.

“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”

Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters.  Why?  Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works.

Their conclusion is particularly surprising because although statistical inference based on probability sampling has a solid theoretical underpinning, in practice pure probability samples are almost never achieved.  Nearly always we are faced with low response rates and non-response biases.  We work around this in practical ways (not always supported by extensive theory) to weight the data, adjust for biases, understand the sources of biases, add caveats to our findings, and so on.  And sure enough, our efforts tend to work, so we keep doing it.

There is substantial data to show that carefully managed non-probability online panels can be used to estimate certain population parameters depending on the nature of the study and how exact those estimates need to be.  Someday our academics and social theorists will help us understand why.  Here’s our theory: Social bodies are not composed of individual units (the assumption underlying inferential statistical theory)  but nodes of networks such that non-random entry points to measure social forces can provide as much information as a random selection of individuals.  We’ll leave it to the future theorists to tell us if we’re right.

In the meantime, there is a lot of research for which using non-probability online panels makes good sense (which the AAPOR report acknowledges), so we’ll keep doing it and we’ll keep extending it into new areas even if the theory can’t keep up.  And we’ll keep refining our techniques based on experience and practice to make it work better, and we’ll keep thinking about why it works and lending that insight to the work we do for you.

If you need help or expertise designing and executing your research based on probability or nonprobability samples, online panels or phone, let us know.  We know our statistical theory.  More importantly, we know our practical statistics.  We can help you sort through it all to ensure rigorous research and practical results.

—Joe Hopper, Ph.D.

The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report.  It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.

Here is a summary of their conclusions and recommendations (quoted verbatim):

• Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.
• There are times when a nonprobability online panel is an appropriate choice, as there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional methods.
• Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results.
• Panel companies can inform the public debate considerably by sharing more about their methods and data.
• Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.

AAPOR’s first conclusion is provocative, and sure to stir up a great deal of controversy.  It runs counter to what most research firms are doing these days.  In an article next week we will consider this conclusion in more depth, and offer a more nuanced perspective than AAPOR.  In the meantime and if you’re interested, you can read the full report.  If you wish to discuss it or need help interpreting it, please let us know as we would be happy to help.

—Joe Hopper, Ph.D.

Phone vs. Online Surveys: Strengths & Weaknesses

As reflected in the table above, the article is rather technical.  But there are two key summary points worth learning from the study:

One, online surveys fielded through panel providers may not fully represent the population.  But we know from other research that they can come close, so the key is to understand the sources of potential bias and adjust for them as needed.

Two, online surveys fielded through panel providers may result in more accurate data than other types of surveys.  There are two reasons for this.  First, online panelists are more engaged in the process and interested in the topics, so they exhibit less cognitive laziness.  Second, online surveys feel more private, so respondents provide answers that are more honest.  This is in contrast to phone surveys in which people usually want to be nice and therefore provide socially desirable answers, even if it’s not how they really feel.

Which should you do, a phone survey or an online survey?  And what type of sampling should you employ, probability or non-probability?  It depends on your objectives.  Each of these issues can matter a lot or not at all depending on the specific questions you are trying to answer.  We would be happy to help you think through your options and the strengths of each.  Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

-Joe Hopper, Ph.D.

How does it save you money?  The biggest cost in conducting a survey is getting it set up and then finding a statistically representative sample of respondents to answer the questions.  Once you’ve got them on the phone or online answering survey questions, it is costs little to ask them “just one more question.”  You want to avoid asking questions for more than fifteen minutes, but suppose that you have fifteen questions you want to ask, and Acme Sports has fifteen questions, and Best Wireless has another fifteen questions.  Rather than each of you conducting your own short survey and duplicating all the work (and cost) of set-up, sampling, and communicating with respondents, you can pool your resources to do one survey.  Each of you pays proportionally to how much of the survey is devoted to your questions.

Here is an example how an omnibus survey works (shown second) versus traditional custom surveys (shown first) — the costs shown are for illustrative purposes only:

Since Versta Research and other research firms have clients asking for short and inexpensive surveys all the time, we do the work of finding the other companies to partner with you on an omnibus.  Versta will advise you on all phases of the work:  How many questions to ask, what to ask, how to ask, and we’ll provide you with several cost and survey-length options so that you can find the right approach for your budget and strategic needs.

Have more questions?  Give us a call at 312-348-6089.  We would be happy to answer your questions and to advise on an approach that works for you.

-Joe Hopper, Ph.D.

Looking back, here are what we consider to be the five biggest changes that shaped current challenges faced by market research and opinion polling:

1. Online data collection using sample panels became the dominant mode of surveying, displacing a huge industry in telephone surveying.  Plus there has been a dramatic decline in households owning landlines, which makes the ideal of probability sampling extremely difficult to achieve.
2. Survey technology made data collection easy and cheap which means that a lot of market research became a commodity, resulting in industry consolidation and need to find new ways of adding value.
3. An explosion of too many surveys, which are everywhere, powered by simple tools like Survey Monkey.  Along with this there has been a dramatic decline in response rates, bringing the issue of quality to the fore like never before.
4. Statistical and mathematical expertise grew in demand.  There is so much data and information, and it is so easy to get.  But who knows what to do with it all?
5. “Audience-ready” reports became more valuable.  Newspapers laid off staff and market research departments downsized, demanding a level of professional involvement to ensure that research reports are easy to read, interpret, and ready to deliver to the audiences who need them most.

One product of these five trends is Versta Research’s focus on high-level expertise and academic brainpower so that you get smart, creative, and flexible market research.  Another is the skill and commitment we have in helping you communicate research to managers, clients, reporters, and the audiences you need to reach.

In our next post we’ll highlight what we predict will be the five biggest trends that will shape market research challenges in the decade to come.

–Joe Hopper, Ph.D.

So I’m always encouraged with yet more evidence that people do tell the truth on surveys.  Studies continue to show that surveys and polling are effective because (1) you need to ask only a small representative sample of people what they think to make statistical conclusions about the whole, and (2) the vast majority of people will tell you the truth about what they think.  It would seem that people are less concerned about privacy and keeping their opinions to themselves  than they are about being heard and making a difference.

The latest data demonstrating this was published in an industry magazine from the Marketing Research Association.  If a person volunteers, or is asked, to take a survey multiple times, what happens?  The vast majority will give you the same answers each time they take the survey, even on questions that ask for subjective ratings on 10 point scales.  If a person were lying or just racing through a survey with random answers to collect incentives, this would not happen.  It turns out that most people give thoughtful, truthful answers each time.

Of course you do not want the same person included in your survey more than once, and a good research vendor will do the careful work of ensuring it does not happen.  But when it does happen, it is fascinating that our efforts to get to the truth are so resilient. The study was done by Paul Johnson, a senior analyst at Western Wats.  If you would like a copy of the study, let me know and I will send you a copy (unfortunately the MRA keeps this content under lock and key, so I cannot include a web link here).

A corollary of this research will be a topic for another day, but here’s the gist:

If you want to know the truth about your product or your customers or your competition, just ask, because most people will tell you.

Versta Research can help you with what to ask, how to ask, whom to ask, and how many to ask, and chances are you will be delighted with how much insight you can gain through a systematic and rigorous effort of asking.

–Joe Hopper, Ph.D.

Why would a market research firm use such a primitive tool for its own feedback?  Are we like the cobbler who can’t afford shoes for our own children?

No.  The reason is that our feedback survey is exceedingly simple.  We ask just two questions:  What was valuable about our work?  What do we need to improve?  (Actually, there is a third question – we’ll leave you in suspense until we complete our first engagement with you.)  We do not ask rating questions because we have no need to quantify the information, and we do not need to quantify because we do not have tens of thousands of clients.  All we need to know – from each client, personally and individually — is what we should keep doing, and what we need to fix.

Versta always uses and recommends the tools that get the job done most effectively.  While there are a lot of terrible do-it-yourself surveys on Survey Monkey, it can be a decent tool if you know how to use it and your needs are simple.

If you’re not sure about your needs, or would like some help with your own DIY project on Survey Monkey, give us a call and we would be happy to lend our expertise.

–Joe Hopper, Ph.D.

In years past, data collection was a slow, complicated, and expensive task.  If you had just a few survey questions, you could share the cost of data collection by cooperating with others who had just a few questions.  That was the birth of the omnibus.  Today, however, most surveys are conducted online, and the data collection process is automated, fast, and inexpensive.  The cost of a custom survey with just a few survey questions, just for you, fielded exactly when and how you want it,  is often no higher than an omnibus.

Sometimes an omnibus survey is the best way to go, but it all depends on the issues you’re trying to understand and the audience you’re trying to reach.  If you think an omnibus survey might be right for you, here are Versta’s recommendations about how to proceed:

• Focus on the issues you’re trying to understand and the budget you have to work with

• Ask us to recommend all options that answer your questions within your budget

• Do not worry about whether the study is an omnibus or not — an omnibus survey is a money-saving technique for the researcher that should be invisible to you

• If an omnibus survey makes sense, work through a firm like Versta Research – even if we field it through a big-name provider for you, it will cost less, and we can design it in a customized way that helps you turn data into stories

–Joe Hopper, Ph.D.

But the problem is rarely “bad respondents” – instead the problem is lazy researchers.  When people discover that the survey they just agreed to take is boring, tedious, repetitive, or too long, they either quit altogether or they stop providing good answers.

It is our job to elicit good data through good design, and to keep respondents – the lifeblood of our work – satisfied and thoughtful.  In a recent article in Alert, a publication of the Market Research Association, Jackie Lorch, VP of Survey Sampling International suggested one way to do this:  Use wording within the questionnaire that “speaks” to respondents, thanks them, and encourages them, in the same way we might do during a one-on-one interview.  Here are four specific suggestions she offers (reproduced verbatim):

1. Acknowledge the considerable effort people are giving.  For example:

The survey is quite long, and there are many questions, which you will have to think quite hard about.

2. Thank people for their effort with specific words which go beyond the usual, casual “thank you for your opinion.”  For example:

Thank you for completing the questionnaire so far. We realize that some of the questions you’ve been answering are not easy, and have required some real thought, especially that last group.

3. Remind people that we know they have a choice to take part or to complete the whole questionnaire.  For example:

Please click on the “next” button when you are ready to continue.

4. Reassure people that survey-taking is an activity in which people like them participate – but at the same time there are some aspects which may seem a bit odd.  For example:

People all over the nation and from all walks of life are taking part. The questions may seem a bit strange at first.

Versta Research believes that to serve our clients, we also need to serve our research respondents.  They are the ones who provide the information and data that we transform into stories, providing the deep insight that clients need.

–Joe Hopper, Ph.D.