The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report.  It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.

Here is a summary of their conclusions and recommendations (quoted verbatim):

• Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.
• There are times when a nonprobability online panel is an appropriate choice, as there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional methods.
• Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results.
• Panel companies can inform the public debate considerably by sharing more about their methods and data.
• Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.

AAPOR’s first conclusion is provocative, and sure to stir up a great deal of controversy.  It runs counter to what most research firms are doing these days.  In an article next week we will consider this conclusion in more depth, and offer a more nuanced perspective than AAPOR.  In the meantime and if you’re interested, you can read the full report.  If you wish to discuss it or need help interpreting it, please let us know as we would be happy to help.

—Joe Hopper, Ph.D.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.

Phone vs. Online Surveys: Strengths & Weaknesses

As reflected in the table above, the article is rather technical.  But there are two key summary points worth learning from the study:

One, online surveys fielded through panel providers may not fully represent the population.  But we know from other research that they can come close, so the key is to understand the sources of potential bias and adjust for them as needed.

Two, online surveys fielded through panel providers may result in more accurate data than other types of surveys.  There are two reasons for this.  First, online panelists are more engaged in the process and interested in the topics, so they exhibit less cognitive laziness.  Second, online surveys feel more private, so respondents provide answers that are more honest.  This is in contrast to phone surveys in which people usually want to be nice and therefore provide socially desirable answers, even if it’s not how they really feel.

Which should you do, a phone survey or an online survey?  And what type of sampling should you employ, probability or non-probability?  It depends on your objectives.  Each of these issues can matter a lot or not at all depending on the specific questions you are trying to answer.  We would be happy to help you think through your options and the strengths of each.  Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

-Joe Hopper, Ph.D.

In years past, data collection was a slow, complicated, and expensive task.  If you had just a few survey questions, you could share the cost of data collection by cooperating with others who had just a few questions.  That was the birth of the omnibus.  Today, however, most surveys are conducted online, and the data collection process is automated, fast, and inexpensive.  The cost of a custom survey with just a few survey questions, just for you, fielded exactly when and how you want it,  is often no higher than an omnibus.

Sometimes an omnibus survey is the best way to go, but it all depends on the issues you’re trying to understand and the audience you’re trying to reach.  If you think an omnibus survey might be right for you, here are Versta’s recommendations about how to proceed:

• Focus on the issues you’re trying to understand and the budget you have to work with

• Ask us to recommend all options that answer your questions within your budget

• Do not worry about whether the study is an omnibus or not — an omnibus survey is a money-saving technique for the researcher that should be invisible to you

• If an omnibus survey makes sense, work through a firm like Versta Research – even if we field it through a big-name provider for you, it will cost less, and we can design it in a customized way that helps you turn data into stories

–Joe Hopper, Ph.D.

Research seems to show this is not the case.  Further, a study just published in the Summer 2009 issue of Public Opinion Quarterly outlines three ways that we can improve the quality of responses in on-line surveys:

• Make the boxes in which respondents type their answers larger

• Explain to respondents that their answers can exceed the size of the box

• Tell respondents how valuable their input is, and encourage thoughtful answers to each open-ended question

These simple techniques will encourage respondents to give more complete, detailed, and thoughtful answers to our questions, yielding rich and insightful data.

–Joe Hopper, Ph.D.

In a provocative new book entitled What Would Google Do? author Jeff Jarvis claims that “Middlemen Are Doomed” because the Internet has made information so easy, accessible, and cheap.  He argues that the Internet links buyers and sellers directly, and it allows buyers to get information and access to sellers instantaneously, rendering middlemen obsolete.  Market researchers are middlemen who operate between businesses who need information about their customers, and the people (their customers and prospects) being interviewed about needs, attitudes, and behaviors.  If Jarvis is right, what is the future for market research and public opinion polling?

The answer, it seems, will depend on how and whether we add value beyond collecting and delivering data.  If we focus on our business being research we add a great deal of value, and while the tools of our trade may evolve, our essential work remains.  Good researchers are experts in the entire process of how to formulate, collect, and analyze information to increase understanding.  We analyze problems, help clients ask the right questions to solve problems, recommend and implement methods of data collection to produce information that will answer the right questions, use interpretive or statistical tools to untangle and interpret the data, and ultimately transform all of this into a story that clients can use to make good decisions.  The Internet is not displacing the importance of these intermediary steps.  On the contrary, new technologies will make the work faster, better, and less expensive, further enhancing the value of what we do as researchers.

However, there are those in the market research industry who, over the last thirty years, have built businesses around data collection and delivery.  They make a lot of phone calls, send out lots of surveys, scan barcodes, collect data, tally it up, and send numbers and charts to clients.  As Jarvis suggested, the problem for these firms is that online technology now links consumers and businesses directly, it lets businesses ask survey questions, it tallies up the data, and puts the results directly in the hands of clients.  No doubt Jarvis is right that this world of market research middlemen is at its end.

Versta Research is committed to helping you deepen your understanding of the world.  The Internet puts the world to your doorstep; smart research helps you make sense of it.

–Joe Hopper, Ph.D.