But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

–Joe Hopper, Ph.D.

The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report.  It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.

Here is a summary of their conclusions and recommendations (quoted verbatim):

• Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.
• There are times when a nonprobability online panel is an appropriate choice, as there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional methods.
• Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results.
• Panel companies can inform the public debate considerably by sharing more about their methods and data.
• Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.

AAPOR’s first conclusion is provocative, and sure to stir up a great deal of controversy.  It runs counter to what most research firms are doing these days.  In an article next week we will consider this conclusion in more depth, and offer a more nuanced perspective than AAPOR.  In the meantime and if you’re interested, you can read the full report.  If you wish to discuss it or need help interpreting it, please let us know as we would be happy to help.

—Joe Hopper, Ph.D.