Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research.  Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.

This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls.  We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do.

The webinar (scheduled for January 26, 2012 with archived replay available afterwards) is being co-sponsored by the American Association of Public Opinion Research, and will be led by Claudia Deane, associate director for public opinion and survey research at the Kaiser Family Foundation.  It is designed for non-researchers, and will focus on:

• The science of how polling and survey research works
• How to determine the legitimacy of a poll and the rigor of survey research
• How to assess the quality and usefulness of survey questions

If you miss the webinar or hunger for more, Versta Research has (and can point you toward) additional resources to help PR professionals, marketing professionals, and market researchers understand, keep abreast of, and communicate the importance of research.  We are one of the few research firms in the industry with a background in university teaching and academic research striving to bring that expertise to the practical worlds of communications campaigns and marketing insight.  Please give us a call and we would be happy to help you further.

–Joe Hopper, Ph.D.

One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

Last week, theater critics roundly panned the show, calling it among the worst Broadway shows ever. The New York Times described is as “so grievously broken in every respect that it is beyond repair.”  But days after these negative reviews, a market research firm was brought in, soliciting volunteers to help fix the show:

If you love Broadway shows, we would like to invite you to be a test audience member and participate in either Act 1 (first part of show) or Act 2 (second part of show) of the all new Broadway musical “Spider-Man: Turn Off the Dark.”

We will be asking you to fill out a survey after the performance, plus stay for a 15 minute discussion in the Spider-Man V.I.P. room. In turn for your feedback and participation you will receive a Spider-Man goodie bag worth over $60.00, on us.

Can market research really rescue the show?  Possibly.  High quality research, both qualitative (such as focus groups) and quantitative (such as surveys) can provide deep insights into how customers and audiences think, perceive, react, and respond to products, services, and opportunities.

If market research can help rescue the show, here are two suggestions we offer that will be critical to keep in mind:

1.  Keep the focus groups focused. Too often clients or managers want their customers to tell them what would make them love their product.  That’s not something customers can reliably tell you.  They can tell you what they care about, what products they purchase and why, what needs and problems they are trying to solve, and what frustrations they face.  Our advice generally: Keep the focus group participants focused on the things that matter to them, then connect the insights gained to the products being offered.

2.  Ask only about things that can be fixed. For all the talk about “actionable” research, there is a great deal of research that offers supposedly actionable ideas that clients or managers simply cannot and will not act upon.  How does the audience feel about Bono’s music for the show?  Unless management is prepared to dump Bono or have him re-write (again), don’t ask.  The key is to determine during the research design which specific decisions can and cannot be made, and then to focus the research on collecting data that lends specific insight to those decisions.

Asking market research to help rescue a $65 million investment at the last minute seems like a tall order, but fortunately great research does not take super-human powers.  At Versta Research we rely on brains, experience, thoughtfulness, and a commitment to working closely with clients to ensure that we deliver insightful stories that are understood and acted upon.

So what will become of Spider-Man?  Will he die under the crushing weight of debt and harsh reviews?  Or will the critics groan in agony as Americans happily pay for the worst show on earth?  Stay tuned for the next exciting episode of “Can a Focus Group Save Spider-Man?”

—Joe Hopper, Ph.D.

Last week, Allstate found itself in this unhappy situation when it released findings from “research” showing accident rates based on drivers’ astrological signs. The findings were absurd, showing that Virgos were 700% more likely to get into car accidents than Scorpios.  The worst part was that some consumers believed it meant Allstate was analyzing astrological signs as part of its insurance underwriting process.  Allstate scored a big hit for media placement, but found itself retracting the research and apologizing soon afterwards.

Regrettably, there are a lot of silly surveys, presumably to support PR.  The PR folks and the research agencies proffering this kind of “research” clearly miss the whole point of doing research for PR, which is to document problems and solutions.

Here’s our suggestion for how to do better.  The key to great research for PR is to focus on your core message and then use data to support it.  Forget about grabbing media headlines with sexy and attention-getting statistics.  Honest and compelling information about issues that concern people will provide a story with authenticity you can leverage.  Here are four steps we suggest:

• Articulate a clear message that supports your business objectives
• Brainstorm relevant and important problems and solutions that are related to that message
• Write “dream headlines” around those problems and solutions using hypothetical facts and figures to support the message
• Conduct real research to uncover and document those problems and solutions

Allstate didn’t do this, admitting that their effort was designed to be “entertaining” rather than substantive.  Had they thought about their message first and how research can be used support it, we doubt they would have tried to grab attention in the way they did.

For additional tips, read our recent article in Public Relations Tactics entitled “How to Create Surveys for PR Stories.”  If you are contemplating research for PR and want to ensure avoiding the pitfalls of silly surveys, feel free to give us a call.  We can advise you and your team on an optimal approach for conducting rigorous, substantive, and media-worthy studies to support your PR efforts.

—Joe Hopper, Ph.D.

Editing is key, however.  Who wants to read something in the halting, choppy, in-eloquent speech that most of us use in talking?  A killer quote is one that is short, direct, pithy, and on point.  Achieving this requires a journalistic standard for presenting quotes.  What does that mean?  Here are some steps:

• Transform the long run-on streams of words into sentences
• Remove the um’s, ah’s, like’s, you-know’s and other verbal fillers
• Cut out false starts and small tangents that do not contribute to intent of the sentence
• Do not change the words they use, or the order in which they use them

The goal is to focus on the intent and meaning of what they were saying.  Then clean it up so that it says exactly what they meant, and really did say.  As a check, imagine going back to your respondent a few minutes after they spoke and asking “Here’s what I’ve written down as what you said.  Is this right?”  If they would likely agree, then you’ve got it.

Here is an example from a project we completed this week:

Before: From what I’ve, you know, I’ve understood, that with Acme it’s very easy and very easy to navigate. That’s what the differences are between the companies is how easy it is to get in there and actually find what you’re looking for, and from what I’m . . . my feedback I’ve gotten about Acme, is that it is very easy and people feel comfortable getting in there and going through the process online.

After:  From what I’ve understood, Acme is very easy to navigate.  That’s the difference between companies: How easy it is to get in there and actually find what you are looking for.  That’s the feedback I’ve gotten about Acme.

If you are curious about whether our “journalistic standard” is appropriate, here’s a fascinating radio story that ran several years ago, which NPR re-ran this week in memory of John Solomon, the reporter who wrote it.

Radio interviews, it turns out, are similarly edited.  And according to NPR, they have never had a person interviewed on their show complain about how they were edited.

Need help grabbing your audience and telling a compelling research story?  Give us a call.  We focus on the rigors of research, and then turning data into stories so that your research gets heard and used, and so that it really makes a difference.

—Joe Hopper, Ph.D.

Should you state a survey’s margin of error in your press release when pitching a story to the media?  In our view, the answer is no.  Why not?  Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.

But how many readers of your news story understand what sampling error is?  For that matter, how many researchers understand what sampling error is?  It was only after four college semesters of advanced mathematical statistics that I finally “got it.”  More importantly, how many readers understand that there are many other potential sources of survey error?  Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%.  (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)

In short, margins of error are misleading because they deal with only one source of error.  They convey a false sense of accuracy.  And they should not be used.  This idea is not always popular among colleagues and clients.  But take comfort:  One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above.  Here we quote their methodological statement that accompanies every press release and report they issue:

All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading. All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100% response rates. These are only theoretical because no published polls come close to this ideal.

We agree 100% with this statement, and advise our clients accordingly.  In our view, the polling industry  has an obligation to re-educate the media about this issue as well.  Transparency requires that you (and we) report samples size, composition, and source.  But transparency also requires a willingness to do away with misleading statements, no matter how much a reporter wants them.

—Joe Hopper, Ph.D.

In an op-ed column two days ago, Stuart Rothenberg, a prominent political (and non-partisan) commentator argued that partisan pollsters (those who work directly for either Democratic or Republican candidates) do a better job than presumably objective third party pollsters.  Why?  Because they have to get it right.  Their campaign strategies depend on it.  Quoting Mr. Rothenberg:

“Nonpartisan pollsters simply have less at stake in getting the right numbers in their polls. Sure, there is personal pride involved, but usually those pollsters are conducting surveys for local media outlets that simply want numbers to put in their newspapers or on local TV news. Often, these nonpartisan pollsters were established to promote an educational institution’s visibility.  Campaign pollsters have much more at stake. Their numbers drive campaign strategy, with victory or defeat of their candidate hanging in the balance. Their numbers have to be correct.”

One irony is that media outlets often pride themselves on their rigorous rules for reporting surveys, with outcomes that are quite the opposite.  Sometimes they reject important stories that rely on research using online methods, for example, despite evidence that online surveys can produce better data.  And all too often they publish silly surveys, as if merely adhering to so-called rigorous protocols guarantees good research.  “Some observers assume the worst [methodological] offenders are campaign pollsters, who presumably cook their numbers to show whatever they want to show. Not true,” says Rothenberg, who cites a number of recent studies suggesting less-than-rigorous work from the likes of Gallup and USA Today, especially when compared to what several partisan pollsters are reporting.

Partisan pollsters, Rothenberg suggests, will do whatever is required to ensure that their numbers reflect exactly what they are trying to measure.  “Survey research depends on the dedication of the pollster to get it right.”

We agree, and Rothenberg’s argument aligns with our approach to working with clients no matter what the nature of our research.  In other words, we take a decidedly partisan approach, which has nothing to do with politics or being Democrats or Republicans.  It has to do with thinking about and executing our work as advocates for your interests who are investing in your success.  Yes, research needs all the objectivity and methodological rigor we can bring to it.  But it also needs our “partisan” dedication to getting it right, because the success or failure of your strategies depend on it.

—Joe Hopper, Ph.D.

There are three tiers of survey research common in public relations, only two of which can truly optimize your PR:

Serious Surveys.  Beyond providing rich material for news releases, serious surveys are designed to establish true thought leadership.  They are leveraged for publication through white papers, conference presentations, or peer-reviewed journals.  The surveys we conducted and published about ulcerative colitis are excellent examples, and for the last three years have been used by the client to demonstrate a deep understanding of the issues faced by patients and physicians.

Solid Surveys. Solid surveys answer interesting questions or document issues that people care about, with solutions that people can use.  The primary goal is media placement, but a solid survey can be leveraged further.  For example, one client we worked for got significant story placements by surveying people about skin cancer myths and reasons for not taking simple precautions like wearing sunscreen.  Another gets ongoing coverage for a survey of IT professionals about future trends in the marketplace.

Silly Surveys. These are unfortunately not designed to answer authentic questions, but to grab attention in ways that may not even be relevant to your business.  We saw one survey not long ago that measured which song titles, from among five or six listed, captured people’s optimism or pessimism about the decade ahead.  The client?  An office products company.  Another recent survey announced the percentage of women who would give up TV, cell phones, computers, or sex in exchange for losing 10 pounds.  Who cares?  These are not meaningful comparisons; they’re just silly.

To be fair, silly surveys can attract good media, but there are costs.  First, they diminish your (and our) long-term credibility as journalists start to ask whether they’re reading yet another gimmick survey or whether your latest effort offers something of real value.  Second, there is an opportunity cost.  You could be investing in information that not only gets media attention, but is leveraged in multiple ways and for many years.  We have a client for whom we did a solid survey back in 2004, which they are still using on their website to engage consumers in 2010.

If you’re in the business of PR, we suggest leaving the silly surveys to Facebook polls.  Instead, focus on building a credible foundation with expert research and data analysis.  Versta Research can help you with this.  We can help you answer questions with expertise, turn data into stories, and ultimately help you communicate those stories to the audiences you need to reach.

—Joe Hopper, Ph.D.