Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/cell-phones-may-double-your-survey-costs/ http://www.verstaresearch.com/blog/cell-phones-may-double-your-survey-costs/#comments Tue, 09 Aug 2011 14:13:02 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1528

These days most researchers agree that if you want to do a random sample phone survey of the U.S. population, you ought to include cell phones.  More than one-quarter of the population do not have landline telephones at home.  Those who do have landline telephones are less likely than ever to answer them, and less likely than ever to participate in surveys.

But it is not easy to include cell phones.  The sampling protocols and the post-stratification weighting become more complicated.  You need to account for a higher probability of cell phone owners being in your sample, because most of them also have landlines.  You can’t use automated or predictive dialing to call cell phone numbers.  You can’t target geography as well, because area codes and exchanges have become mobile.  And people get mad at you if they have to pay for incoming calls, so you need to offer cash.

What’s the bottom line effect on costs for a survey that includes cell phones?  A recent study sponsored by the American Association for Public Opinion Research (AAPOR) documents the following:

The cost of fieldwork for cell phone interviews is double if you don’t screen out those who also have landlines, and more than double (2.6 times higher) if you do screen them out.

Of course if you are surveying the whole population, only a portion of your sampling and interviews will be cell-phone based.  Currently, we recommend that 20% to 40% of interviews be cell-based.  But there are additional professional costs to remember as well, such as purchasing, managing, merging, weighting, and analyzing different types and sources of sample, and training interviewers to work with different sources and types of respondents.

There are still plenty of phone surveys being done that do not include cell-phones, and for many types of studies landline-only surveys produce information that is good enough for what needs to be learned.  But it is getting increasingly difficult for these surveys to achieve true representation and surely their days numbered.

Feel free to give us a call if you need help figuring out the best approach for your research.  We can advise you on the most cost-effective, feasible, and rigorous approaches to getting the data, stories, and level of understanding you need.

—Joe Hopper, Ph.D.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

This is not to say they are bad or fraudulent respondents.  Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions.  The problem with respondent error, it turns out, is poor survey design, which may involve biased or  ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.

Recent research shows that grid-style questions that look like this:

or this:

. . . hurt the reliability of answers from respondents.  The evidence for this conclusion comes from one of the leading academic authorities on survey design, Duane Alwin, at Penn State.  Here we quote from a Harvard reviewer who summarized the findings in a recent journal article:

“Of particular note is Alwin’s finding that the widespread survey practice of presenting items in batteries—sets of consecutive questions using the same response format—tends to yield less reliable responses than presenting them alone or in a series of topically related questions with differing response formats. He conjectures that “[s]imilarity of question content and response format may actually distract a respondent from giving full attention to what information is being asked” (p.180). Some might anticipate that similarity in response format would instead heighten reliability, by raising correlations among items in a battery. Alwin’s reliability estimates do not depend on within-occasion correlations between different items,however: the finding suggests that respondents vary across occasions in how they use a battery’s response format.”

In short, even if you have good respondents who are not “straight-lining” through your grids, they may be focusing more on the task of filling out the grid than on a thoughtful response to each item in the grid.  So instead, lay out your questions one at a time, like this:

And when you’re not sure what to do next, namely how to turn all of that now-reliable data into a story that you can really use, give Versta Research a call.

—Joe Hopper, Ph.D.

This shift in telephone usage is critically important to survey research, because telephone surveys have set a gold standard for rigorous research over the last two decades.  Given the trends, most survey researchers would agree that cell phone sample must now be included in the most rigorous research designs for an RDD survey.  But doing so introduces new difficulties in design, sampling, weighting, and cost, and our collective experiences are not yet sufficient for the industry to  agree on a set of “best practices” when it comes to including cell phones in survey research.

Our leading industry association, AAPOR (The American Association of Public Opinion Research) recently published a comprehensive update from the AAPOR Cell Phone Task Force that offers a good overview of the issues, complications, guidelines, and considerations every researcher should take into account when designing a telephone survey of the general population.  Briefly, it covers and highlights the following issues:

Coverage and Sampling: It is increasingly difficult to reach young men and minorities via landlines, so cell phone coverage is critical.  But using dual and overlapping sampling frames (both landline and cell phone) introduces new methodological complications in sampling.

Nonresponse: Response rates are lower among cell phones users, though response rates among landline users continue to decline as well.

Measurement: Data gathered via cell phone interviews is generally not of lower quality (despite concerns about audio quality, talking in public places, and distractions from multitasking).

Weighting: If dual and overlapping sampling frames are used (to include both landlines and cell phones) then data need to be statistically weighted to account for the complicated differences in probabilities that each person in the population will be recruited into the survey.

Legal and Ethical Issues: Including cell phones in surveying introduces a new set of legal, regulatory, and ethical considerations regarding auto-dialers, texting, caller ID, and do-not-call lists.

Operational Issues: Cell phone interviewing requires a unique set of protocols for recruiting, call-backs, and interviewing, all of which require special training for the people conducting the interviews.

Cost Issues: Including cell phones at least doubles the cost of a survey, and many times may triple or quadruple the cost.

Should you include cell phones in your survey?  It depends on who you are trying to reach and for what purposes.  When it comes to effective surveying, there are no absolutely right answers about cell phones vs. landlines, just as there are no absolutely right answers about online vs. phone surveys.  Moreover, the rapidly changing landscape of how people communicate means that the best answer today may be different in six months.

Need some help thinking through the options and figuring out the best approach for your research?  Feel free to give us a call.  We have decades of experience and are committed to helping our clients take full advantage of important changes and innovations in research.

—Joe Hopper, Ph.D.

It turns out that I may just lack the survey-taking gene.  No joke.  New research of genetic and fraternal twins shows that our willingness to participate in research is shaped in part by our genes.  The research was led by Lori Foster Thompson, an associate professor of psychology at North Carolina State University, and is soon to be published in the Journal of Organizational Behavior.  As reported on Science Daily:

For the study, the researchers sent out a survey to over 1,000 sets of twins — some fraternal, some identical — and then measured who did and did not respond. The researchers were interested in whether the response behavior of one twin accurately predicted the behavior of the other twin. “We found that the behavior of one identical twin was a good predictor for the other,” Foster Thompson says, “but that the same did not hold true for fraternal twins.

“Because all of the sets of twins were raised in the same household, the only distinguishing variable between identical and fraternal twin sets is the fact that identical twins are genetically identical and fraternal twins are not.”

Understanding why some people avoid surveys is important because systematic non-participation can be a source of error in research known as “non-response bias.”  It is critical to understand the sources and potential effects of skewed samples.  Of course what this research does not tell us is whether  genetics has any effect on the outcomes we are trying to measure.  If the genetic survey-takers are no different from the non-survey-takers on the key issues we care about in our research, then it does not matter.

It  is ironic that some of us have trouble relating to the willingness of respondents who are the lifeblood of our work.  But it is also a good thing that helps us anticipate all the potential difficulties that good research overcomes.  Here at Versta Research, the happy-survey-takers advise on the best pitch for those who love it, while those who are reluctant, like me, advise on the barriers we may need to overcome.  In the end, you get research designed to understand precisely and accurately the full population of people you care about, whatever their genetic dispositions may be.

—Joe Hopper, Ph.D.

The authors looked at various types of biases that creep into business managers’ decisions when based on data presented to them.  They did this by conducting experiments with business school students and managers who are members of the American Marketing Association.  Some were presented with numeric data in tables, while others were presented with data in charts or graphs.  All tables, charts, and graphs were clear and well-designed.

The results?  When compared to an optimal decision based on a purely rational assessment of the data, decisions were typically biased, and “graphical formats that followed existing recommendation for the appropriate display of data did not reduce . . . biases compared with data presented in tables.”  Moreover, “neither real-world experience nor explicit training reduced these biases.”

In some ways this is surprising, because graphing data can often help us better (and more quickly) grasp its meaning.  On the other hand it is not surprising.  Graphs can be so visually compelling that they might hinder purely rational assessments.

In our view (and the authors’ as well) the research does not invalidate the need for effective data visualization.  A good chart can be a compelling piece of story.  But it is not the story, any more than a table of numbers can be the story.  A good chart is a communication tool.  So are good tables, and so are good sentences that weave together a compelling story.

At Versta Research we use chart and tables in about equal proportions (sometimes tables work better) and we always integrate them into a clear story with appropriate statistical modeling to lend support.  The solution to overcoming bias is not fancier graphics, but rather a clear presentation of information with a compelling synthesis and assessment of what it means.

—Joe Hopper, Ph.D.

For example, in some cultures people are especially reluctant to give negative answers, which exaggerates the positive-response bias we are accustomed to seeing in the U.S.  Even seemingly “factual” questions can be subject to measurement error and biases that make cross-cultural comparisons potentially difficult.

To help, a team of academic researchers led by the University of Michigan’s Survey Research Center and the University of Nebraska’s Survey Research and Methodology Program have just released a new set of guidelines for best practices in cross-cultural surveys.  They address a full range of topics, including questionnaire design, translation, adaptation and quality.  The guidelines are excellent (and voluminous – 638 pages) and provide an essential primer on problems and solutions for comparative survey research across cultures and countries.

The guidelines are also a good reminder that researchers should always be sensitive to cultural issues, biases, and variations even within their own cultures.  Statistically significant gender variation may reflect important differences between men and women, or it may reflect basic cultural differences in how men and women answer survey questions.  Remember to keep both possibilities in mind, and think about ways to query your data to uncover the most plausible answer.

If you’re not sure about how culture and context affect your plan for research or the data you are analyzing, we would be happy to offer you our best thinking.  Please don’t hesitate to call us at 312-348-6089.

—Joe Hopper, Ph.D.

The article provides evidence that online surveys offer better data quality compared to telephone surveys and other modes of “oral” interviewing. Data from the computer surveys were superior in terms of:

1. Validity. The researchers used two different measures of the same thing, and found higher consistency between the two measures for respondents taking the computer-based survey. This is known as concurrent validity.
2. Satisficing. Respondents taking the computer-based survey worked harder to provide accurate answers, which showed up in more variation and differentiation among their answers, and fewer response-order effects.
3. Social desirability bias. Those taking computer surveys were more likely to share their true feelings about a controversial topic rather than reporting feelings that would make them look more sympathetic or politically correct.

The findings reinforce those reported in another POQ article published at the end of last year, which outlined the relative strengths and weaknesses of phone versus online surveys. They will also no doubt feed the ongoing controversy about usage of online survey panels (See “How Good Are Online Survey Panels?” and “Practical Statistics vs. Theoretical Statistics”).

Which should you do, a phone survey or an online survey? It depends on your objectives. We would be happy to help you think through your options and the strengths of each. Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

—Joe Hopper, Ph.D.

Keeping your survey efforts honest is critical.  Sincere efforts to document issues through rigorous surveys succeed, and when done correctly, people are influenced by numbers.  But trying to shock or manipulate with sensationalist, biased, and phony data gets you only so far.  For a great survey, keep it honest, rigorous, and valid, and carefully control for bias.  We, at Versta, can help you succeed with this.

-Joe Hopper, Ph.D.