Even if you are not involved in political polling, it is worth paying attention to the methods and best practices of political pollsters.  One reason is that few other areas of research offer a way to completely validate one’s methods.  Pollsters are using sampling and survey methods to predict the behaviors of a much larger population.  Then in just one day that population behaves, we get a near-perfect count of exactly how they behaved, and we know whether the methods worked.

Several industry colleagues have recently been debating the merits of calculating and reporting “margins of error” in political polling, and pointed us to some surprising data from The New York Times:

[The New York Times has compiled] a database consisting of thousands of primary and caucus polls dating back to the 1970s.  Each poll contains numbers for several candidates, so there are a total of about 17,000 observations. How often does a candidate’s actual vote total fall within the theoretical margin of error?  The answer is, not very often. In theory, a candidate’s actual vote total should fall outside the margin of error only 5 percent of the time [given that political polls report margins of error using a 95% confidence interval]. In reality, the candidate’s vote total was outside the margin of error 65 percent of the time! Part of this is because the database includes some polls conducted months before the actual voting took place. But even if you restrict the analysis to polls conducted within the final week of the campaign, about 40 percent of the vote totals fell outside the margin of error — eight times more often than is supposed to happen if you could take the margin of error at face value.

This does not mean that the polls were wrong, predicting wins for losing candidates and vice versa.  Rather, it means that the estimates were not as precise as the stated margins of error would have a reader believe.

The problem is that “margins of error” are based on a statistical theories that almost never line up with the messy reality of our world.  Margins of error make a number of assumptions which are rarely true in practice, including:

• Respondents are selected through simple random sampling

• All those sampled participate in the survey

• Sampling error is the only source of survey error

Indeed, Versta Research usually recommends to clients who publish survey research that they not report margins of error because the concept (and the calculations) are seriously misleading and flawed.

Calculating margins of error and looking at statistical significance should be used not because they give accurate or “scientific” predictions, but because they provide useful summary measures of how much variability there is in the data given the sample size and other critical factors that can affect one’s estimates.  At Versta Research, this helps us better interpret data and better assess what matters.  That, in turn, allows us to tell a story with the data that does not overreach or misrepresent what is going on.

–Joe Hopper, Ph.D.

One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

What does Versta Research recommend when it comes to sample size?  Well, the answer always depends on the type of study, the objectives of the study, the audience to whom it will be presented, and of course budget. We can make legitimate statistical calculations based on any sample size, but there are dramatic trade-offs in precision and cost no matter what sample size is chosen.

Here, however, are some general guidelines:

If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision.  The benefit of doing this will almost never exceed the cost of including that many more people in your sample.

The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.

With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point.  But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points.  Another twenty five, fifty, or one hundred respondents can make an important difference when you are looking at small samples.

Most sample sizes will range from about 100 and 1,200 and within this range, for each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates.  So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people.  Some of the healthcare research we do relies on samples sizes of fifty to one-hundred.  Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200.

Unfortunately there is no magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand.  But there is a magic phone number worth trying:  (312) 348-6089.  Versta Research has a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget.  As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/an-interactive-graph-for-choosing-sample-size/feed/ 0 http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/ http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/#comments Thu, 28 Apr 2011 13:32:38 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1306 As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys.  A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion.  The insanely popular survey question used to calculate Net Promoter Scores is a good example:

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

As of this writing, Versta is one of 67 prominent survey and polling organizations currently supporting the initiative. The initiative is designed to create protocols and recognition for regular disclosure of methods when survey organizations conduct public polls.

Trust in survey research and public polls is critical not only for the public good, but for the important work that we do for our clients.  Our clients rely on us to give them accurate data and truthful assessments. They can’t make good decisions without that. And they rely on us to ensure that when they share data with the public, it is backed by the most rigorous and defensible methods available in the social sciences. Transparency is one means of ensuring that all organizations are continually held to the highest standards of rigor and accuracy.

AAPOR began the initiative in 2009 “to encourage routine disclosure of methodological information from polls and surveys whose findings are released to the public.” The organization lists the following as key elements of the efforts now underway:

• The association’s public recognition of excellence in transparency;
• A system for collecting and archiving disclosure information;
• Education for organizations in how to make transparency a routine part of their work;
• Outreach to survey sponsors, users of survey data and the public to reinforce the value of transparency and to publicize the organizations that participate in the initiative;
• Collaboration with other associations and academic disciplines in these activities.

One benefit of working with an organization like Versta Research is that you can fully trust the outcomes of the research, and you can trust that we have the expertise to defend it.  You can also trust that we work diligently within the industry to protect the value of our collective investments in good research.

—Joe Hopper, Ph.D.

Here are four tips for writing omnibus survey questions to ensure that your effort is successful:

1.  Tailor your questions to the interview mode.  Some omnibus surveys are conducted by phone, and some are done online.  Each mode requires a different style of asking questions.  For example, having an agree/disagree scale with seven points would work well for an online survey.  But on the phone, each of those points would have to be labeled with words and read out loud to the respondent.

2.  Ask the right number of questions.  If you ask too many questions, you may be defeating the cost advantage of an omnibus.  But if you ask too few questions, you will not get enough depth and leverage to tell a story.  Usually you need points of contrast or context, so you will need to ask more than just one or two direct questions.  Plan on asking five to eight questions for an omnibus survey.

3.  Keep a broad focus.  Most U.S. omnibus surveys include 1,000 respondents representing the full adult population.  Ask questions that will apply to all or most of them so that you are taking advantage of the full sample size.

4.  Ask questions that relate to your campaign. Your goal is to highlight the product or service you are selling.  Sensational, funny, or outrageous questions and survey findings will sometimes give you a quick flash of attention, but not the kind of substantive leverage that will truly carry a message into multiple forums over the course of several months.

At Versta, we will work with you on all phases of the omnibus process, including concept development, design, drafting and revising the questions, and then analyzing  and reporting the findings in a way that helps you tell the story.  Have more questions?  Give us a call — we are happy to help.

—Joe Hopper, Ph.D.

But new technologies have made data collection more efficient and less expensive, so omnibus surveys are not always the best option.  For many clients, a fully customized survey might be less expensive than an omnibus survey.

Here are the two most important factors that affect whether an omnibus survey will make sense for you:

1.  The number of questions you want to ask. If you have a small number of questions – typically no more than five or six – then an omnibus survey can be a good option.  Omnibus surveys are designed to aggregate just a few questions from many different clients, and are priced accordingly.  If you have more than five or six questions, then fielding a short custom survey can often be priced as competitively as an omnibus, and you gain some significant advantages.

2.  Who you want to survey. Omnibus surveys work only if many clients want to get survey responses from the same group of people.  As such, an omnibus survey can be a good option if the population you want to study is broadly defined.  In fact, most omnibus surveys are “general population” surveys, which means they rely on a sample of the entire U.S. adult population.  If you start narrowing your target group (such as moms with children under the age of 5), then a custom survey that recruits those specific people is likely to be better.

You need both conditions for an omnibus survey to make sense.  In other words, you need to have just a handful of questions, and you need to have a broadly defined group as your target.  Many surveys meet these conditions, and when they do, they save you money.

If you are not sure whether an omnibus survey makes sense for you, we would be happy to price both options for you, outlining the advantages and disadvantages of each.  We’ll advise you on the optimal approach given your objectives.  Feel free to give us a call (312-348-6089).

-Joe Hopper, Ph.D.

How does it save you money?  The biggest cost in conducting a survey is getting it set up and then finding a statistically representative sample of respondents to answer the questions.  Once you’ve got them on the phone or online answering survey questions, it is costs little to ask them “just one more question.”  You want to avoid asking questions for more than fifteen minutes, but suppose that you have fifteen questions you want to ask, and Acme Sports has fifteen questions, and Best Wireless has another fifteen questions.  Rather than each of you conducting your own short survey and duplicating all the work (and cost) of set-up, sampling, and communicating with respondents, you can pool your resources to do one survey.  Each of you pays proportionally to how much of the survey is devoted to your questions.

Here is an example how an omnibus survey works (shown second) versus traditional custom surveys (shown first) — the costs shown are for illustrative purposes only:

Since Versta Research and other research firms have clients asking for short and inexpensive surveys all the time, we do the work of finding the other companies to partner with you on an omnibus.  Versta will advise you on all phases of the work:  How many questions to ask, what to ask, how to ask, and we’ll provide you with several cost and survey-length options so that you can find the right approach for your budget and strategic needs.

Have more questions?  Give us a call at 312-348-6089.  We would be happy to answer your questions and to advise on an approach that works for you.

-Joe Hopper, Ph.D.