Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/tips-for-sampling-from-online-panels/ http://www.verstaresearch.com/blog/tips-for-sampling-from-online-panels/#comments Wed, 23 Nov 2011 15:09:06 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1754 Versta Research is a strong advocate for using online panels for surveys.  As telephone usage and technology have changed, phone surveys are increasingly difficult and expensive, and they are not necessarily more rigorous than other methods.

But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

Two researchers at the U.S. Census Bureau recently outlined an emerging innovation in survey research that could reverse the trend towards passive, boring, self-administered surveys that characterizes much online research.  The idea is to use internet avatars in real-time interviewing with survey respondents.

Beyond just the heightened interest of having an animated survey, the avatars would be programmed to register and interpret respondents’ verbal answers, facial expressions, and body language through webcams.

Suppose, for example, that a respondent answers a question with detailed information that answers a follow-up question as well.  The avatar would use natural language processing to insert that data into the subsequent question, and then avoid asking the follow-up.  Or if the respondent looks away from the screen and pauses for time longer than is typical, the avatar can offer a rephrased question or a reassuring comment to re-engage the participant and to put him or her at ease.  This type of innovation  could bring many of the advantages of live interviewing back into the realm of internet surveys, which are far more efficient in terms of time and cost.

The use of effective avatar interviewers is at least several years away, however, because it involves not only evolving internet technologies, but also advanced linguistic processing, facial and voice recognition technologies, and so on.  In fact, the sheer technological difficulty of truly replacing human interviewers reminds us of how absurd it is for research companies to make claims about technology replacing higher-order activities in the research process, such as providing analysis and insight.

At least for now, software and services with “actionable insight” buttons generate yet more mountains of data in need of human synthesis and interpretation.  If anything, the role for smart and experienced researchers who can turn all that data into a story is growing.  It is growing for researchers who work on the client side and who have direct accountability to the executives who need data-driven insights.  And it is growing for firms like Versta Research where the highest levels of intellectual and human capital are central to our work.

–Joe Hopper, Ph.D.

• Thirty percent of U.S. households do not have a landline telephone
• An additional 16% have a landline telephone, but never or rarely use it to receive calls
• The percentage of households without landlines is increasing by about five to six percentage points each year
• Half of young adults under age 30 have no landline in their homes
• Half of adult renters have no landline in their homes
• Nearly four out of ten Hispanic adults have no landline in their homes

For anyone doing random sample surveys of consumers, the implications are huge.  Methodological purists insist that only phone-based surveys are rigorous, but clearly the biases of phone-based research can be severe, and no doubt they often are.

Putting aside the issue of research methods, however, we believe a more significant issue over the next few years will be the explosive growth in smartphone usage.  Although not captured in government surveys that track phone usage and availability, the Pew center estimates that one-third of all U.S. adults currently own a smartphone, and industry analysts predict that by the end of 2011 half of all cell phones in the U.S. will be smartphones.

Why is this is so critical for researchers?  Because more than any other technology since the widespread adoption of the Internet, smartphones are changing how consumers behave.  Consumers get up in the morning and check media apps before brushing their teeth.  Then while commuting to work or waiting for their computer to boot up, they purchase coupons for goods and services they might never have bought otherwise.  They map and track their locations, and expect product features and services that can anticipate what they need.

Smartphones are important for reasons far beyond narrow methodological concerns about how to execute surveys.  They are important because they are changing the very nature of what we research: how people behave and think, and what they buy, believe, want, and aspire to.  Indeed, just as we nearly always ask our research respondents to tell us their gender, age, and income, we now often ask whether they have smartphones.

Beyond anything else, market research is about understanding people (the verstehen in Versta), which means our focus at Versta is always on the “what, how, and why” of what matters in peoples’ lives in a rapidly changing world.  Smartphones matter in a more profound way than cell phones ever did.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/smartphones-matter-more-than-cell-phones/feed/ 0 http://www.verstaresearch.com/blog/how-to-boost-response-rates-for-online-surveys/ http://www.verstaresearch.com/blog/how-to-boost-response-rates-for-online-surveys/#comments Wed, 29 Jun 2011 21:50:07 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1461 One of the surprises of multi-mode research is that offering people a choice of how they want to complete a survey (online, by mail, by phone, etc.) does not necessarily boost response rates.  An article in the most recent issue of Public Opinion Quarterly provides new evidence of this.  The study showed that even in a population with full access to both mail and Internet options (and full literacy in both modes), a full mail survey achieves a higher response rate than a web-based survey.  It also achieves a higher response rate than a choice of either mail or web.  So much for our recent article on The Myth of Too Many Choices!

But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates.  Here are some key takeaways from the research:

• A pre-notification postcard sent by mail will significantly improve response rates to your online survey

• Offering a token cash incentive (a couple dollars) sent by mail in advance will significantly improve response rates to your online survey

• As a final step, offering a mail survey to non-respondents will boost response rates even further

So why even bother with online surveys, you may wonder?  Indeed, even with all this, response rates are no better than conducting a mail-only survey.  The answer is that online surveys have a number of advantages worth keeping.  Programmed skip logic and constraints ensure no missing data or misinterpretation of skip patterns.  Data are entered into a database real-time, eliminating errors from manual data entry or optical scanning.  The process is faster, more efficient, and usually less expensive.

While many people overestimate the importance of response rates on a survey’s validity, there is no doubt that higher response rates are better in terms of cost, efficiency, and feasibility.  So if you are thinking about deploying an online survey to your customers or members, it is probably worth investing in some old fashioned mail-based outreach.

Need help?  We can be reached by phone (312-348-6089), or Internet, or mail (919 Forest Ave, Evanston, IL  60202).  Whichever way you contact us, you’ll get a 100% response rate within a few hours.

—Joe Hopper, Ph.D.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

When it comes to surveys for market research and public polling, smartphones have bigger and better screens, better technology, and web capabilities.  But more importantly, smartphones are becoming the “device of choice” for tasks that were once the domain of our desktop computers. For example, web-based e-mail via PCs is on the decline, while email through mobile devices is growing rapidly.  And of course unless we’re doing phone surveys, e-mail is currently the primary means by which researchers recruit survey respondents.

Leora Lawton, a colleague in AAPOR (the American Association of Public Opinion Research) recently reported back with some tips on survey design for smartphones that she synthesized from a talk by Mario Callegaro of Google.  In brief, she reports:

• Avoid long surveys
• Do not use question grids
• Scales should be a maximum of five points
• Show one question per page
• Put navigation buttons on top
• Use vertical, not horizontal, buttons
• Avoid logos, progress meters, help links, and copyrights that take up space
• Avoid drop-down menus that require JavaScript

Of course technologies change quickly, and enhanced capabilities and tools for effective and sophisticated smartphone surveys are in development.  But for now these are good ideas to keep in mind.

Need help thinking about the best approach for your research, whether it be telephone, in-person, online, or smartphone-based?  Feel free to give us a call; we would be happy to help.

—Joe Hopper, Ph.D.

The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report.  It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.

Here is a summary of their conclusions and recommendations (quoted verbatim):

• Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.
• There are times when a nonprobability online panel is an appropriate choice, as there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional methods.
• Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results.
• Panel companies can inform the public debate considerably by sharing more about their methods and data.
• Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.

AAPOR’s first conclusion is provocative, and sure to stir up a great deal of controversy.  It runs counter to what most research firms are doing these days.  In an article next week we will consider this conclusion in more depth, and offer a more nuanced perspective than AAPOR.  In the meantime and if you’re interested, you can read the full report.  If you wish to discuss it or need help interpreting it, please let us know as we would be happy to help.

—Joe Hopper, Ph.D.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.