Here we offer the “Versta Digest” version as a handy reference card.  Once you get the hang of it, you don’t need the examples and explanation.  You just need to know the rules.  We recommend reading the full article first, so you know what we’re talking about when it comes to “points.”  Then, when you need a refresher or a reference source, consult these rules:

• One point for each simple question or scaled response
• One point for every two response options in a multiple choice question
• One point for each row in a grid question
• Two points for any response that requires mental calculation
• Three points for every short response to an open-ended question
• One point for every three sentences of extra text that respondents must read

Then:

• Tally up the points
• Divide by 8 for online survey length (in minutes)
• Divide by 8 then multiply by 1.5 for phone survey length (in minutes)

The system is straightforward, easy to learn, and easy to execute.  It is a method that really works and that we have validated against hundreds of different types of surveys over the past several years.

Once you know how to determine survey length, you can think more strategically about the ideal survey length to optimize the value and content of a survey within your budget.  Call us at 312-348-6089 with any additional assistance you may need.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

Two researchers at the U.S. Census Bureau recently outlined an emerging innovation in survey research that could reverse the trend towards passive, boring, self-administered surveys that characterizes much online research.  The idea is to use internet avatars in real-time interviewing with survey respondents.

Beyond just the heightened interest of having an animated survey, the avatars would be programmed to register and interpret respondents’ verbal answers, facial expressions, and body language through webcams.

Suppose, for example, that a respondent answers a question with detailed information that answers a follow-up question as well.  The avatar would use natural language processing to insert that data into the subsequent question, and then avoid asking the follow-up.  Or if the respondent looks away from the screen and pauses for time longer than is typical, the avatar can offer a rephrased question or a reassuring comment to re-engage the participant and to put him or her at ease.  This type of innovation  could bring many of the advantages of live interviewing back into the realm of internet surveys, which are far more efficient in terms of time and cost.

The use of effective avatar interviewers is at least several years away, however, because it involves not only evolving internet technologies, but also advanced linguistic processing, facial and voice recognition technologies, and so on.  In fact, the sheer technological difficulty of truly replacing human interviewers reminds us of how absurd it is for research companies to make claims about technology replacing higher-order activities in the research process, such as providing analysis and insight.

At least for now, software and services with “actionable insight” buttons generate yet more mountains of data in need of human synthesis and interpretation.  If anything, the role for smart and experienced researchers who can turn all that data into a story is growing.  It is growing for researchers who work on the client side and who have direct accountability to the executives who need data-driven insights.  And it is growing for firms like Versta Research where the highest levels of intellectual and human capital are central to our work.

–Joe Hopper, Ph.D.

• Thirty percent of U.S. households do not have a landline telephone
• An additional 16% have a landline telephone, but never or rarely use it to receive calls
• The percentage of households without landlines is increasing by about five to six percentage points each year
• Half of young adults under age 30 have no landline in their homes
• Half of adult renters have no landline in their homes
• Nearly four out of ten Hispanic adults have no landline in their homes

For anyone doing random sample surveys of consumers, the implications are huge.  Methodological purists insist that only phone-based surveys are rigorous, but clearly the biases of phone-based research can be severe, and no doubt they often are.

Putting aside the issue of research methods, however, we believe a more significant issue over the next few years will be the explosive growth in smartphone usage.  Although not captured in government surveys that track phone usage and availability, the Pew center estimates that one-third of all U.S. adults currently own a smartphone, and industry analysts predict that by the end of 2011 half of all cell phones in the U.S. will be smartphones.

Why is this is so critical for researchers?  Because more than any other technology since the widespread adoption of the Internet, smartphones are changing how consumers behave.  Consumers get up in the morning and check media apps before brushing their teeth.  Then while commuting to work or waiting for their computer to boot up, they purchase coupons for goods and services they might never have bought otherwise.  They map and track their locations, and expect product features and services that can anticipate what they need.

Smartphones are important for reasons far beyond narrow methodological concerns about how to execute surveys.  They are important because they are changing the very nature of what we research: how people behave and think, and what they buy, believe, want, and aspire to.  Indeed, just as we nearly always ask our research respondents to tell us their gender, age, and income, we now often ask whether they have smartphones.

Beyond anything else, market research is about understanding people (the verstehen in Versta), which means our focus at Versta is always on the “what, how, and why” of what matters in peoples’ lives in a rapidly changing world.  Smartphones matter in a more profound way than cell phones ever did.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/smartphones-matter-more-than-cell-phones/feed/ 0 http://www.verstaresearch.com/blog/cell-phones-may-double-your-survey-costs/ http://www.verstaresearch.com/blog/cell-phones-may-double-your-survey-costs/#comments Tue, 09 Aug 2011 14:13:02 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1528

These days most researchers agree that if you want to do a random sample phone survey of the U.S. population, you ought to include cell phones.  More than one-quarter of the population do not have landline telephones at home.  Those who do have landline telephones are less likely than ever to answer them, and less likely than ever to participate in surveys.

But it is not easy to include cell phones.  The sampling protocols and the post-stratification weighting become more complicated.  You need to account for a higher probability of cell phone owners being in your sample, because most of them also have landlines.  You can’t use automated or predictive dialing to call cell phone numbers.  You can’t target geography as well, because area codes and exchanges have become mobile.  And people get mad at you if they have to pay for incoming calls, so you need to offer cash.

What’s the bottom line effect on costs for a survey that includes cell phones?  A recent study sponsored by the American Association for Public Opinion Research (AAPOR) documents the following:

The cost of fieldwork for cell phone interviews is double if you don’t screen out those who also have landlines, and more than double (2.6 times higher) if you do screen them out.

Of course if you are surveying the whole population, only a portion of your sampling and interviews will be cell-phone based.  Currently, we recommend that 20% to 40% of interviews be cell-based.  But there are additional professional costs to remember as well, such as purchasing, managing, merging, weighting, and analyzing different types and sources of sample, and training interviewers to work with different sources and types of respondents.

There are still plenty of phone surveys being done that do not include cell-phones, and for many types of studies landline-only surveys produce information that is good enough for what needs to be learned.  But it is getting increasingly difficult for these surveys to achieve true representation and surely their days numbered.

Feel free to give us a call if you need help figuring out the best approach for your research.  We can advise you on the most cost-effective, feasible, and rigorous approaches to getting the data, stories, and level of understanding you need.

—Joe Hopper, Ph.D.

But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates.  Here are some key takeaways from the research:

• A pre-notification postcard sent by mail will significantly improve response rates to your online survey

• Offering a token cash incentive (a couple dollars) sent by mail in advance will significantly improve response rates to your online survey

• As a final step, offering a mail survey to non-respondents will boost response rates even further

So why even bother with online surveys, you may wonder?  Indeed, even with all this, response rates are no better than conducting a mail-only survey.  The answer is that online surveys have a number of advantages worth keeping.  Programmed skip logic and constraints ensure no missing data or misinterpretation of skip patterns.  Data are entered into a database real-time, eliminating errors from manual data entry or optical scanning.  The process is faster, more efficient, and usually less expensive.

While many people overestimate the importance of response rates on a survey’s validity, there is no doubt that higher response rates are better in terms of cost, efficiency, and feasibility.  So if you are thinking about deploying an online survey to your customers or members, it is probably worth investing in some old fashioned mail-based outreach.

Need help?  We can be reached by phone (312-348-6089), or Internet, or mail (919 Forest Ave, Evanston, IL  60202).  Whichever way you contact us, you’ll get a 100% response rate within a few hours.

—Joe Hopper, Ph.D.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process.  We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.

But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing.  That’s an unusually high ratio of “suspends” as we call them.  So what was the problem?  Were we just getting lousy respondents who did not want to seriously participate in a survey?  Was the survey was too difficult, tedious, boring, or confusing?  One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.

The story in this data: Something is wrong with your survey

Nearly every respondent who quit got close to finishing and then dropped off at exactly the same point, which was odd because the most difficult questions were earlier in the survey.  In fact, the question where most ended up quitting was an interesting drag-and-drop interactive exercise.  Ah, that was the problem.  The programming for the interactive piece was flawed and respondents were being kicked out.

It wasn’t without a good deal of angst that the programming team tested, re-tested, and confirmed the error.  Everybody involved in this effort resisted: the sampling provider, the programmers, the survey tool developers, the questionnaire designers—they have all done this work hundreds of times, so there can’t be anything wrong with their piece of it, right?

Let the data speak.  It will tell you where the mistakes are. There are lots of places and moments where things may go wrong.  If the top people responsible for the project do not have immediate visibility into the data, they are unable to suggest smart solutions, and bad research will just keep happening.

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

This is not to say they are bad or fraudulent respondents.  Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions.  The problem with respondent error, it turns out, is poor survey design, which may involve biased or  ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.

Recent research shows that grid-style questions that look like this:

or this:

. . . hurt the reliability of answers from respondents.  The evidence for this conclusion comes from one of the leading academic authorities on survey design, Duane Alwin, at Penn State.  Here we quote from a Harvard reviewer who summarized the findings in a recent journal article:

“Of particular note is Alwin’s finding that the widespread survey practice of presenting items in batteries—sets of consecutive questions using the same response format—tends to yield less reliable responses than presenting them alone or in a series of topically related questions with differing response formats. He conjectures that “[s]imilarity of question content and response format may actually distract a respondent from giving full attention to what information is being asked” (p.180). Some might anticipate that similarity in response format would instead heighten reliability, by raising correlations among items in a battery. Alwin’s reliability estimates do not depend on within-occasion correlations between different items,however: the finding suggests that respondents vary across occasions in how they use a battery’s response format.”

In short, even if you have good respondents who are not “straight-lining” through your grids, they may be focusing more on the task of filling out the grid than on a thoughtful response to each item in the grid.  So instead, lay out your questions one at a time, like this:

And when you’re not sure what to do next, namely how to turn all of that now-reliable data into a story that you can really use, give Versta Research a call.

—Joe Hopper, Ph.D.