Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research.  Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.

This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls.  We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do.

The webinar (scheduled for January 26, 2012 with archived replay available afterwards) is being co-sponsored by the American Association of Public Opinion Research, and will be led by Claudia Deane, associate director for public opinion and survey research at the Kaiser Family Foundation.  It is designed for non-researchers, and will focus on:

• The science of how polling and survey research works
• How to determine the legitimacy of a poll and the rigor of survey research
• How to assess the quality and usefulness of survey questions

If you miss the webinar or hunger for more, Versta Research has (and can point you toward) additional resources to help PR professionals, marketing professionals, and market researchers understand, keep abreast of, and communicate the importance of research.  We are one of the few research firms in the industry with a background in university teaching and academic research striving to bring that expertise to the practical worlds of communications campaigns and marketing insight.  Please give us a call and we would be happy to help you further.

–Joe Hopper, Ph.D.

One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

What does Versta Research recommend when it comes to sample size?  Well, the answer always depends on the type of study, the objectives of the study, the audience to whom it will be presented, and of course budget. We can make legitimate statistical calculations based on any sample size, but there are dramatic trade-offs in precision and cost no matter what sample size is chosen.

Here, however, are some general guidelines:

If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision.  The benefit of doing this will almost never exceed the cost of including that many more people in your sample.

The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.

With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point.  But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points.  Another twenty five, fifty, or one hundred respondents can make an important difference when you are looking at small samples.

Most sample sizes will range from about 100 and 1,200 and within this range, for each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates.  So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people.  Some of the healthcare research we do relies on samples sizes of fifty to one-hundred.  Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200.

Unfortunately there is no magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand.  But there is a magic phone number worth trying:  (312) 348-6089.  Versta Research has a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget.  As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/an-interactive-graph-for-choosing-sample-size/feed/ 0 http://www.verstaresearch.com/blog/online-surveys-have-same-accuracy-as-phone/ http://www.verstaresearch.com/blog/online-surveys-have-same-accuracy-as-phone/#comments Thu, 26 May 2011 13:24:37 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1379 A new study presented by two professors from Harvard University and the University of Massachusetts at Amherst was probably one of the liveliest and potentially disruptive presentations at least week’s annual meeting of the American Association of Public Opinion Research (AAPOR) in Phoenix.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

Finding Gold in Your Data Mine

The truth, however, is that while technology and tools can multiply our capabilities and help us work better and faster, they cannot discover meaningful patterns or find hidden insights. Only smart people can do that.  The reason is that market research data only become meaningful within a context of questions that need to be answered, or stories that need to be told.  Tools and technology cannot supply that context.

We are working with a client who has been struggling for the last five months to find a story in survey data.  They commissioned the survey to generate data for a whitepaper for presentation to business level clients and prospects.  They’ve been staring at tables and banner tabs, pie charts and bar charts, correlations and gap analyses.  But squeeze the data as they might, the story will not emerge.

Here was our counsel:  Before you can find a story in the data, be more explicit about the context that will bring it to life.  So we asked each person on the team to write three or four dream headlines that they would like to see come out of the research.  Our instructions were as follows:

1.  Focus on headlines that would be most useful to communicate to your audience

2.  Do not look at the survey or the data—forget about what you think it says

3.  Do not go back to any of your previous documents or thinking—do it top of mind, quickly

4.  Do not worry about making your headlines pretty or accurate—make it a brain dump of your dream headlines

Of course we cannot guarantee that the data will support every claim they want to make.  But we can guarantee that with so many data points and ways of linking data in even the smallest of data sets, there are compelling ways to make that data support a story that is on target and relevant.

How do you find the gold in your data mine?  By providing a frame of reference in which the data becomes gold.  Take three steps backward to consider the critical (and specific) questions you need to have answered and outline the relevant stories that would be useful to your audiences.  Then start mining your data for answers to those questions and for data points, contradictions, anomalies, and surprising patterns that relate to your stories.

—Joe Hopper, Ph.D.

Last week, Allstate found itself in this unhappy situation when it released findings from “research” showing accident rates based on drivers’ astrological signs. The findings were absurd, showing that Virgos were 700% more likely to get into car accidents than Scorpios.  The worst part was that some consumers believed it meant Allstate was analyzing astrological signs as part of its insurance underwriting process.  Allstate scored a big hit for media placement, but found itself retracting the research and apologizing soon afterwards.

Regrettably, there are a lot of silly surveys, presumably to support PR.  The PR folks and the research agencies proffering this kind of “research” clearly miss the whole point of doing research for PR, which is to document problems and solutions.

Here’s our suggestion for how to do better.  The key to great research for PR is to focus on your core message and then use data to support it.  Forget about grabbing media headlines with sexy and attention-getting statistics.  Honest and compelling information about issues that concern people will provide a story with authenticity you can leverage.  Here are four steps we suggest:

• Articulate a clear message that supports your business objectives
• Brainstorm relevant and important problems and solutions that are related to that message
• Write “dream headlines” around those problems and solutions using hypothetical facts and figures to support the message
• Conduct real research to uncover and document those problems and solutions

Allstate didn’t do this, admitting that their effort was designed to be “entertaining” rather than substantive.  Had they thought about their message first and how research can be used support it, we doubt they would have tried to grab attention in the way they did.

For additional tips, read our recent article in Public Relations Tactics entitled “How to Create Surveys for PR Stories.”  If you are contemplating research for PR and want to ensure avoiding the pitfalls of silly surveys, feel free to give us a call.  We can advise you and your team on an optimal approach for conducting rigorous, substantive, and media-worthy studies to support your PR efforts.

—Joe Hopper, Ph.D.

Should you state a survey’s margin of error in your press release when pitching a story to the media?  In our view, the answer is no.  Why not?  Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.

But how many readers of your news story understand what sampling error is?  For that matter, how many researchers understand what sampling error is?  It was only after four college semesters of advanced mathematical statistics that I finally “got it.”  More importantly, how many readers understand that there are many other potential sources of survey error?  Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%.  (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)

In short, margins of error are misleading because they deal with only one source of error.  They convey a false sense of accuracy.  And they should not be used.  This idea is not always popular among colleagues and clients.  But take comfort:  One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above.  Here we quote their methodological statement that accompanies every press release and report they issue:

All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading. All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100% response rates. These are only theoretical because no published polls come close to this ideal.

We agree 100% with this statement, and advise our clients accordingly.  In our view, the polling industry  has an obligation to re-educate the media about this issue as well.  Transparency requires that you (and we) report samples size, composition, and source.  But transparency also requires a willingness to do away with misleading statements, no matter how much a reporter wants them.

—Joe Hopper, Ph.D.

There are three tiers of survey research common in public relations, only two of which can truly optimize your PR:

Serious Surveys.  Beyond providing rich material for news releases, serious surveys are designed to establish true thought leadership.  They are leveraged for publication through white papers, conference presentations, or peer-reviewed journals.  The surveys we conducted and published about ulcerative colitis are excellent examples, and for the last three years have been used by the client to demonstrate a deep understanding of the issues faced by patients and physicians.

Solid Surveys. Solid surveys answer interesting questions or document issues that people care about, with solutions that people can use.  The primary goal is media placement, but a solid survey can be leveraged further.  For example, one client we worked for got significant story placements by surveying people about skin cancer myths and reasons for not taking simple precautions like wearing sunscreen.  Another gets ongoing coverage for a survey of IT professionals about future trends in the marketplace.

Silly Surveys. These are unfortunately not designed to answer authentic questions, but to grab attention in ways that may not even be relevant to your business.  We saw one survey not long ago that measured which song titles, from among five or six listed, captured people’s optimism or pessimism about the decade ahead.  The client?  An office products company.  Another recent survey announced the percentage of women who would give up TV, cell phones, computers, or sex in exchange for losing 10 pounds.  Who cares?  These are not meaningful comparisons; they’re just silly.

To be fair, silly surveys can attract good media, but there are costs.  First, they diminish your (and our) long-term credibility as journalists start to ask whether they’re reading yet another gimmick survey or whether your latest effort offers something of real value.  Second, there is an opportunity cost.  You could be investing in information that not only gets media attention, but is leveraged in multiple ways and for many years.  We have a client for whom we did a solid survey back in 2004, which they are still using on their website to engage consumers in 2010.

If you’re in the business of PR, we suggest leaving the silly surveys to Facebook polls.  Instead, focus on building a credible foundation with expert research and data analysis.  Versta Research can help you with this.  We can help you answer questions with expertise, turn data into stories, and ultimately help you communicate those stories to the audiences you need to reach.

—Joe Hopper, Ph.D.