Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/how-to-estimate-the-length-of-a-survey/ http://www.verstaresearch.com/blog/how-to-estimate-the-length-of-a-survey/#comments Wed, 07 Dec 2011 20:50:11 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1771 In Versta Research’s Winter 2011 Newsletter, published just this week, we describe a simple method for estimating how long it will take respondents to complete surveys.

Here we offer the “Versta Digest” version as a handy reference card.  Once you get the hang of it, you don’t need the examples and explanation.  You just need to know the rules.  We recommend reading the full article first, so you know what we’re talking about when it comes to “points.”  Then, when you need a refresher or a reference source, consult these rules:

• One point for each simple question or scaled response
• One point for every two response options in a multiple choice question
• One point for each row in a grid question
• Two points for any response that requires mental calculation
• Three points for every short response to an open-ended question
• One point for every three sentences of extra text that respondents must read

Then:

• Tally up the points
• Divide by 8 for online survey length (in minutes)
• Divide by 8 then multiply by 1.5 for phone survey length (in minutes)

The system is straightforward, easy to learn, and easy to execute.  It is a method that really works and that we have validated against hundreds of different types of surveys over the past several years.

Once you know how to determine survey length, you can think more strategically about the ideal survey length to optimize the value and content of a survey within your budget.  Call us at 312-348-6089 with any additional assistance you may need.

–Joe Hopper, Ph.D.

But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

–Joe Hopper, Ph.D.

One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits.  If these respondents are not providing thoughtful answers, the data are suspect.  All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.

But we should not rely on panel providers alone to ensure valid data.  Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases.  Here are typical indicators of potentially bad data:

1.  Speeding. Though people can legitimately whiz through surveys at varying speeds, we typically flag the fastest five percent for further investigation.

2.  Non-sense open ends.  People who have nothing to say will usually say that, so we flag respondents who type random letters, offer non-sense or vacuous answers, or skip answering entirely.

3.  Choosing all options on a screening question.  Often it  means the respondent was gaming the survey to get in, especially if some options logically exclude others.

4.  Failing quality check questions.  Usually we include a couple of questions that have only one correct response to flag respondents who are not paying attention.

5.  Inconsistent numeric values.  How long a person has worked in a profession or at a particular job, for example, must be consistent with a person’s age.

6.  Straight-lining and patterning.  If questions are laid out in grids, respondents who answer identically for all questions, or who move in a diagonal along the grid should be flagged for investigation.

7.  Logically inconsistent answers.  If attitude and behavior questions are logically related to each other (for example, multiple questions about concern for the environment), inconsistent responses may indicate bad data.

The customer who reminded us that online surveys face multiple hurdles had just gotten results from a survey that she discovered had included a respondent who took the survey 250 times.  Nobody from the research firm bothered to look at the data beyond feeding it into the data-tabulator-chart-maker-here-are-your-actionable-insights machine.

At Versta Research, our approach is the opposite.  Smart people look at your data at each step because there is no other way to turn data into a story that you can trust and then share with your management team with confidence.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

Two researchers at the U.S. Census Bureau recently outlined an emerging innovation in survey research that could reverse the trend towards passive, boring, self-administered surveys that characterizes much online research.  The idea is to use internet avatars in real-time interviewing with survey respondents.

Beyond just the heightened interest of having an animated survey, the avatars would be programmed to register and interpret respondents’ verbal answers, facial expressions, and body language through webcams.

Suppose, for example, that a respondent answers a question with detailed information that answers a follow-up question as well.  The avatar would use natural language processing to insert that data into the subsequent question, and then avoid asking the follow-up.  Or if the respondent looks away from the screen and pauses for time longer than is typical, the avatar can offer a rephrased question or a reassuring comment to re-engage the participant and to put him or her at ease.  This type of innovation  could bring many of the advantages of live interviewing back into the realm of internet surveys, which are far more efficient in terms of time and cost.

The use of effective avatar interviewers is at least several years away, however, because it involves not only evolving internet technologies, but also advanced linguistic processing, facial and voice recognition technologies, and so on.  In fact, the sheer technological difficulty of truly replacing human interviewers reminds us of how absurd it is for research companies to make claims about technology replacing higher-order activities in the research process, such as providing analysis and insight.

At least for now, software and services with “actionable insight” buttons generate yet more mountains of data in need of human synthesis and interpretation.  If anything, the role for smart and experienced researchers who can turn all that data into a story is growing.  It is growing for researchers who work on the client side and who have direct accountability to the executives who need data-driven insights.  And it is growing for firms like Versta Research where the highest levels of intellectual and human capital are central to our work.

–Joe Hopper, Ph.D.

But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates.  Here are some key takeaways from the research:

• A pre-notification postcard sent by mail will significantly improve response rates to your online survey

• Offering a token cash incentive (a couple dollars) sent by mail in advance will significantly improve response rates to your online survey

• As a final step, offering a mail survey to non-respondents will boost response rates even further

So why even bother with online surveys, you may wonder?  Indeed, even with all this, response rates are no better than conducting a mail-only survey.  The answer is that online surveys have a number of advantages worth keeping.  Programmed skip logic and constraints ensure no missing data or misinterpretation of skip patterns.  Data are entered into a database real-time, eliminating errors from manual data entry or optical scanning.  The process is faster, more efficient, and usually less expensive.

While many people overestimate the importance of response rates on a survey’s validity, there is no doubt that higher response rates are better in terms of cost, efficiency, and feasibility.  So if you are thinking about deploying an online survey to your customers or members, it is probably worth investing in some old fashioned mail-based outreach.

Need help?  We can be reached by phone (312-348-6089), or Internet, or mail (919 Forest Ave, Evanston, IL  60202).  Whichever way you contact us, you’ll get a 100% response rate within a few hours.

—Joe Hopper, Ph.D.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

This is not to say they are bad or fraudulent respondents.  Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions.  The problem with respondent error, it turns out, is poor survey design, which may involve biased or  ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.

Recent research shows that grid-style questions that look like this:

or this:

. . . hurt the reliability of answers from respondents.  The evidence for this conclusion comes from one of the leading academic authorities on survey design, Duane Alwin, at Penn State.  Here we quote from a Harvard reviewer who summarized the findings in a recent journal article:

“Of particular note is Alwin’s finding that the widespread survey practice of presenting items in batteries—sets of consecutive questions using the same response format—tends to yield less reliable responses than presenting them alone or in a series of topically related questions with differing response formats. He conjectures that “[s]imilarity of question content and response format may actually distract a respondent from giving full attention to what information is being asked” (p.180). Some might anticipate that similarity in response format would instead heighten reliability, by raising correlations among items in a battery. Alwin’s reliability estimates do not depend on within-occasion correlations between different items,however: the finding suggests that respondents vary across occasions in how they use a battery’s response format.”

In short, even if you have good respondents who are not “straight-lining” through your grids, they may be focusing more on the task of filling out the grid than on a thoughtful response to each item in the grid.  So instead, lay out your questions one at a time, like this:

And when you’re not sure what to do next, namely how to turn all of that now-reliable data into a story that you can really use, give Versta Research a call.

—Joe Hopper, Ph.D.

As newly-appointed director of market research for the American Marketing Association in Chicago, I oversee efforts to develop and organize events related to market research.  I also oversee efforts to measure, track, and analyze customer satisfaction for all other events and activities sponsored by the AMA.  For the past few events, the AMA collected feedback via an online survey sent by e-mail immediately after the events.  They were also using a less-than-stellar do-it-yourself survey tool that promises to deliver “actionable insights!” at the click of a button.

The results were abysmal.  For every 50 attendees, we were lucky to get one response, which meant our final sample sizes ranged from one to four respondents.  Even a magic “actionable insights” button can’t do much with that.  For good reasons, we did not want to send out multiple e-mail reminders, and there was no money or staff to reach out with telephone calls.

So what’s a market researcher to do?  First, we on the market research committee joked about doing multivariate analysis with a sample of n=1.  OK, not so hilarious.  Then we decided that for the next MR event that we were organizing, we would go back to basics.  The plan:  Distribute surveys by hand during the last ten minutes of the event; make an announcement about it; explain why feedback is important; tell people how the data will be used; ask them to do it; and collect their surveys as they headed out the door.

We considered offering a raffle incentive as well, but we didn’t need to.  We got responses back from two-thirds of attendees.  True, we lost some advantages of doing an online survey.  The process was less anonymous, and we were stuck having to enter all the data into a computer by hand.  But we got robust data that we can work with, analyze, and develop a story to share with our client.

Sometimes paper surveys make a lot of sense, and in this case it made sense because our respondent base was  right there, a captive audience in one room.  Evolving technologies don’t always replace the old.  Telephones, computers, the Internet, e-mail, social media, and smartphones are not necessarily replacing paper surveys, but rather opening new opportunities and finding their appropriate niches.  Each mode has an important and valuable place in market research.  The key is to know when to use each and for what purposes, because all you really care about is getting information to make smart decisions.

Need help sorting through all the options?  Feel free to give us a call.  We can help you optimize your research plan by choosing the best methods, whether old or new, and then putting the best minds to work in answering your critical questions.

—Joe Hopper, Ph.D.