Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research.  Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.

This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls.  We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do.

The webinar (scheduled for January 26, 2012 with archived replay available afterwards) is being co-sponsored by the American Association of Public Opinion Research, and will be led by Claudia Deane, associate director for public opinion and survey research at the Kaiser Family Foundation.  It is designed for non-researchers, and will focus on:

• The science of how polling and survey research works
• How to determine the legitimacy of a poll and the rigor of survey research
• How to assess the quality and usefulness of survey questions

If you miss the webinar or hunger for more, Versta Research has (and can point you toward) additional resources to help PR professionals, marketing professionals, and market researchers understand, keep abreast of, and communicate the importance of research.  We are one of the few research firms in the industry with a background in university teaching and academic research striving to bring that expertise to the practical worlds of communications campaigns and marketing insight.  Please give us a call and we would be happy to help you further.

–Joe Hopper, Ph.D.

One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

What does Versta Research recommend when it comes to sample size?  Well, the answer always depends on the type of study, the objectives of the study, the audience to whom it will be presented, and of course budget. We can make legitimate statistical calculations based on any sample size, but there are dramatic trade-offs in precision and cost no matter what sample size is chosen.

Here, however, are some general guidelines:

If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision.  The benefit of doing this will almost never exceed the cost of including that many more people in your sample.

The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.

With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point.  But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points.  Another twenty five, fifty, or one hundred respondents can make an important difference when you are looking at small samples.

Most sample sizes will range from about 100 and 1,200 and within this range, for each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates.  So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people.  Some of the healthcare research we do relies on samples sizes of fifty to one-hundred.  Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200.

Unfortunately there is no magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand.  But there is a magic phone number worth trying:  (312) 348-6089.  Versta Research has a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget.  As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/an-interactive-graph-for-choosing-sample-size/feed/ 0 http://www.verstaresearch.com/blog/online-surveys-have-same-accuracy-as-phone/ http://www.verstaresearch.com/blog/online-surveys-have-same-accuracy-as-phone/#comments Thu, 26 May 2011 13:24:37 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1379 A new study presented by two professors from Harvard University and the University of Massachusetts at Amherst was probably one of the liveliest and potentially disruptive presentations at least week’s annual meeting of the American Association of Public Opinion Research (AAPOR) in Phoenix.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys.   The news is mixed.  While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great.  The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:

Declining Satisfaction with Surveys

In addition to the chart above, key statistics outlined in the article include:

• Nearly nine out of ten (87%) feel that public opinion polls, on the whole, are a good thing

• Four out of five (79%) say that the last time they were interviewed for research, it was a pleasant experience

• Two-thirds (68%) agree that research surveys help manufacturers or other organizations produce better products and services for consumers

• About half (54%) feel that the research industry serves a useful purpose.

• Just one third (34%) trust pollsters to tell the truth, though twice as many (64%) believe that most polls give results that are “pretty nearly right”

• Just one-quarter (29%) think that survey organizations can be trusted to protect their privacy

• One-quarter (25%) feel that answering questions in polls or research surveys is a waste of time

Clearly the research industry (and our clients) can do much to improve.  Here are two quick ideas as a start.  (1)  Next time you are tempted to send yet another survey to your customers, ask whether it’s purpose is to benefit them or you—if it’s the latter, don’t send it.  (2) Next time you are tempted to send out a 45-minute survey with grid after grid of repetitive measures, work more carefully with the people who will use the research to specify which of those measures can truly be acted upon.

A more careful and thoughtful approach to research will benefit you and your customers, and it will boost the confidence that our publics have in research and polling.  Versta Research would be delighted to work with you in making this happen.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/what-people-think-of-surveys/feed/ 0 http://www.verstaresearch.com/blog/allstate%e2%80%99s-pr-misstep-with-a-silly-study/ http://www.verstaresearch.com/blog/allstate%e2%80%99s-pr-misstep-with-a-silly-study/#comments Fri, 04 Feb 2011 13:46:37 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1110 Research can be powerful for PR efforts because new data and analysis often help anchor a story with facts and figures that have a deep interest and appeal to readers.  But if the research is not clearly designed to support key messages in your PR plan, you may find yourself apologizing to managers for a campaign that failed.  Worse yet, you may find yourself apologizing to an audience who got exactly the wrong message.

Last week, Allstate found itself in this unhappy situation when it released findings from “research” showing accident rates based on drivers’ astrological signs. The findings were absurd, showing that Virgos were 700% more likely to get into car accidents than Scorpios.  The worst part was that some consumers believed it meant Allstate was analyzing astrological signs as part of its insurance underwriting process.  Allstate scored a big hit for media placement, but found itself retracting the research and apologizing soon afterwards.

Regrettably, there are a lot of silly surveys, presumably to support PR.  The PR folks and the research agencies proffering this kind of “research” clearly miss the whole point of doing research for PR, which is to document problems and solutions.

Here’s our suggestion for how to do better.  The key to great research for PR is to focus on your core message and then use data to support it.  Forget about grabbing media headlines with sexy and attention-getting statistics.  Honest and compelling information about issues that concern people will provide a story with authenticity you can leverage.  Here are four steps we suggest:

• Articulate a clear message that supports your business objectives
• Brainstorm relevant and important problems and solutions that are related to that message
• Write “dream headlines” around those problems and solutions using hypothetical facts and figures to support the message
• Conduct real research to uncover and document those problems and solutions

Allstate didn’t do this, admitting that their effort was designed to be “entertaining” rather than substantive.  Had they thought about their message first and how research can be used support it, we doubt they would have tried to grab attention in the way they did.

For additional tips, read our recent article in Public Relations Tactics entitled “How to Create Surveys for PR Stories.”  If you are contemplating research for PR and want to ensure avoiding the pitfalls of silly surveys, feel free to give us a call.  We can advise you and your team on an optimal approach for conducting rigorous, substantive, and media-worthy studies to support your PR efforts.

—Joe Hopper, Ph.D.

AAPOR and the ASA are offering a great way for non-statisticians to learn more about sampling.  AAPOR is the American Association of Public Opinion Research, and the ASA is the American Statistical Association.  In February they are hosting an introductory webinar on sampling for non-statisticians that that we recommend.  It is being taught by a senior statistician at NORC, and will cover such topics as:

• The logic of sampling
• The concept of a sampling frame
• The difference between a sample and a census
• The process of sampling
• Different types of samples
• Probability vs. non-probability sampling
• Sampling error and non-sampling error
• Sample size considerations
• Post-sampling steps

The webinar is described as “a course to expose non-statisticians to sampling so that they are able to read and understand articles or documents describing sampling designs and communicate with statisticians about their research needs.”

The work that we do at Versta Research always starts with the art of asking questions (so we know what our research must “do”) then moves into the rigors of research design, which typically involves the science of sampling, then back again into the art and science of turning data into stories.  If you have always had a nagging feeling that you ought to know just a little bit more about that middle piece—the science of sampling—without having to get a four year degree in statistics, here’s your chance.

—Joe Hopper, Ph.D.

Should you state a survey’s margin of error in your press release when pitching a story to the media?  In our view, the answer is no.  Why not?  Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.

But how many readers of your news story understand what sampling error is?  For that matter, how many researchers understand what sampling error is?  It was only after four college semesters of advanced mathematical statistics that I finally “got it.”  More importantly, how many readers understand that there are many other potential sources of survey error?  Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%.  (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)

In short, margins of error are misleading because they deal with only one source of error.  They convey a false sense of accuracy.  And they should not be used.  This idea is not always popular among colleagues and clients.  But take comfort:  One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above.  Here we quote their methodological statement that accompanies every press release and report they issue:

All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading. All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100% response rates. These are only theoretical because no published polls come close to this ideal.

We agree 100% with this statement, and advise our clients accordingly.  In our view, the polling industry  has an obligation to re-educate the media about this issue as well.  Transparency requires that you (and we) report samples size, composition, and source.  But transparency also requires a willingness to do away with misleading statements, no matter how much a reporter wants them.

—Joe Hopper, Ph.D.