Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research.  Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.

This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls.  We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do.

The webinar (scheduled for January 26, 2012 with archived replay available afterwards) is being co-sponsored by the American Association of Public Opinion Research, and will be led by Claudia Deane, associate director for public opinion and survey research at the Kaiser Family Foundation.  It is designed for non-researchers, and will focus on:

• The science of how polling and survey research works
• How to determine the legitimacy of a poll and the rigor of survey research
• How to assess the quality and usefulness of survey questions

If you miss the webinar or hunger for more, Versta Research has (and can point you toward) additional resources to help PR professionals, marketing professionals, and market researchers understand, keep abreast of, and communicate the importance of research.  We are one of the few research firms in the industry with a background in university teaching and academic research striving to bring that expertise to the practical worlds of communications campaigns and marketing insight.  Please give us a call and we would be happy to help you further.

–Joe Hopper, Ph.D.

Even if you are not involved in political polling, it is worth paying attention to the methods and best practices of political pollsters.  One reason is that few other areas of research offer a way to completely validate one’s methods.  Pollsters are using sampling and survey methods to predict the behaviors of a much larger population.  Then in just one day that population behaves, we get a near-perfect count of exactly how they behaved, and we know whether the methods worked.

Several industry colleagues have recently been debating the merits of calculating and reporting “margins of error” in political polling, and pointed us to some surprising data from The New York Times:

[The New York Times has compiled] a database consisting of thousands of primary and caucus polls dating back to the 1970s.  Each poll contains numbers for several candidates, so there are a total of about 17,000 observations. How often does a candidate’s actual vote total fall within the theoretical margin of error?  The answer is, not very often. In theory, a candidate’s actual vote total should fall outside the margin of error only 5 percent of the time [given that political polls report margins of error using a 95% confidence interval]. In reality, the candidate’s vote total was outside the margin of error 65 percent of the time! Part of this is because the database includes some polls conducted months before the actual voting took place. But even if you restrict the analysis to polls conducted within the final week of the campaign, about 40 percent of the vote totals fell outside the margin of error — eight times more often than is supposed to happen if you could take the margin of error at face value.

This does not mean that the polls were wrong, predicting wins for losing candidates and vice versa.  Rather, it means that the estimates were not as precise as the stated margins of error would have a reader believe.

The problem is that “margins of error” are based on a statistical theories that almost never line up with the messy reality of our world.  Margins of error make a number of assumptions which are rarely true in practice, including:

• Respondents are selected through simple random sampling

• All those sampled participate in the survey

• Sampling error is the only source of survey error

Indeed, Versta Research usually recommends to clients who publish survey research that they not report margins of error because the concept (and the calculations) are seriously misleading and flawed.

Calculating margins of error and looking at statistical significance should be used not because they give accurate or “scientific” predictions, but because they provide useful summary measures of how much variability there is in the data given the sample size and other critical factors that can affect one’s estimates.  At Versta Research, this helps us better interpret data and better assess what matters.  That, in turn, allows us to tell a story with the data that does not overreach or misrepresent what is going on.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

These days most researchers agree that if you want to do a random sample phone survey of the U.S. population, you ought to include cell phones.  More than one-quarter of the population do not have landline telephones at home.  Those who do have landline telephones are less likely than ever to answer them, and less likely than ever to participate in surveys.

But it is not easy to include cell phones.  The sampling protocols and the post-stratification weighting become more complicated.  You need to account for a higher probability of cell phone owners being in your sample, because most of them also have landlines.  You can’t use automated or predictive dialing to call cell phone numbers.  You can’t target geography as well, because area codes and exchanges have become mobile.  And people get mad at you if they have to pay for incoming calls, so you need to offer cash.

What’s the bottom line effect on costs for a survey that includes cell phones?  A recent study sponsored by the American Association for Public Opinion Research (AAPOR) documents the following:

The cost of fieldwork for cell phone interviews is double if you don’t screen out those who also have landlines, and more than double (2.6 times higher) if you do screen them out.

Of course if you are surveying the whole population, only a portion of your sampling and interviews will be cell-phone based.  Currently, we recommend that 20% to 40% of interviews be cell-based.  But there are additional professional costs to remember as well, such as purchasing, managing, merging, weighting, and analyzing different types and sources of sample, and training interviewers to work with different sources and types of respondents.

There are still plenty of phone surveys being done that do not include cell-phones, and for many types of studies landline-only surveys produce information that is good enough for what needs to be learned.  But it is getting increasingly difficult for these surveys to achieve true representation and surely their days numbered.

Feel free to give us a call if you need help figuring out the best approach for your research.  We can advise you on the most cost-effective, feasible, and rigorous approaches to getting the data, stories, and level of understanding you need.

—Joe Hopper, Ph.D.

What does Versta Research recommend when it comes to sample size?  Well, the answer always depends on the type of study, the objectives of the study, the audience to whom it will be presented, and of course budget. We can make legitimate statistical calculations based on any sample size, but there are dramatic trade-offs in precision and cost no matter what sample size is chosen.

Here, however, are some general guidelines:

If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision.  The benefit of doing this will almost never exceed the cost of including that many more people in your sample.

The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.

With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point.  But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points.  Another twenty five, fifty, or one hundred respondents can make an important difference when you are looking at small samples.

Most sample sizes will range from about 100 and 1,200 and within this range, for each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates.  So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people.  Some of the healthcare research we do relies on samples sizes of fifty to one-hundred.  Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200.

Unfortunately there is no magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand.  But there is a magic phone number worth trying:  (312) 348-6089.  Versta Research has a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget.  As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/an-interactive-graph-for-choosing-sample-size/feed/ 0 http://www.verstaresearch.com/blog/online-surveys-have-same-accuracy-as-phone/ http://www.verstaresearch.com/blog/online-surveys-have-same-accuracy-as-phone/#comments Thu, 26 May 2011 13:24:37 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1379 A new study presented by two professors from Harvard University and the University of Massachusetts at Amherst was probably one of the liveliest and potentially disruptive presentations at least week’s annual meeting of the American Association of Public Opinion Research (AAPOR) in Phoenix.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys.   The news is mixed.  While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great.  The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:

Declining Satisfaction with Surveys

In addition to the chart above, key statistics outlined in the article include:

• Nearly nine out of ten (87%) feel that public opinion polls, on the whole, are a good thing

• Four out of five (79%) say that the last time they were interviewed for research, it was a pleasant experience

• Two-thirds (68%) agree that research surveys help manufacturers or other organizations produce better products and services for consumers

• About half (54%) feel that the research industry serves a useful purpose.

• Just one third (34%) trust pollsters to tell the truth, though twice as many (64%) believe that most polls give results that are “pretty nearly right”

• Just one-quarter (29%) think that survey organizations can be trusted to protect their privacy

• One-quarter (25%) feel that answering questions in polls or research surveys is a waste of time

Clearly the research industry (and our clients) can do much to improve.  Here are two quick ideas as a start.  (1)  Next time you are tempted to send yet another survey to your customers, ask whether it’s purpose is to benefit them or you—if it’s the latter, don’t send it.  (2) Next time you are tempted to send out a 45-minute survey with grid after grid of repetitive measures, work more carefully with the people who will use the research to specify which of those measures can truly be acted upon.

A more careful and thoughtful approach to research will benefit you and your customers, and it will boost the confidence that our publics have in research and polling.  Versta Research would be delighted to work with you in making this happen.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/what-people-think-of-surveys/feed/ 0 http://www.verstaresearch.com/blog/survey-says-call-me-on-my-cell-phone/ http://www.verstaresearch.com/blog/survey-says-call-me-on-my-cell-phone/#comments Thu, 30 Dec 2010 14:41:00 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1004 The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes.  So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population.  Does it matter, given that surveys rarely interview everyone anyway?  Probably.  If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.

This shift in telephone usage is critically important to survey research, because telephone surveys have set a gold standard for rigorous research over the last two decades.  Given the trends, most survey researchers would agree that cell phone sample must now be included in the most rigorous research designs for an RDD survey.  But doing so introduces new difficulties in design, sampling, weighting, and cost, and our collective experiences are not yet sufficient for the industry to  agree on a set of “best practices” when it comes to including cell phones in survey research.

Our leading industry association, AAPOR (The American Association of Public Opinion Research) recently published a comprehensive update from the AAPOR Cell Phone Task Force that offers a good overview of the issues, complications, guidelines, and considerations every researcher should take into account when designing a telephone survey of the general population.  Briefly, it covers and highlights the following issues:

Coverage and Sampling: It is increasingly difficult to reach young men and minorities via landlines, so cell phone coverage is critical.  But using dual and overlapping sampling frames (both landline and cell phone) introduces new methodological complications in sampling.

Nonresponse: Response rates are lower among cell phones users, though response rates among landline users continue to decline as well.

Measurement: Data gathered via cell phone interviews is generally not of lower quality (despite concerns about audio quality, talking in public places, and distractions from multitasking).

Weighting: If dual and overlapping sampling frames are used (to include both landlines and cell phones) then data need to be statistically weighted to account for the complicated differences in probabilities that each person in the population will be recruited into the survey.

Legal and Ethical Issues: Including cell phones in surveying introduces a new set of legal, regulatory, and ethical considerations regarding auto-dialers, texting, caller ID, and do-not-call lists.

Operational Issues: Cell phone interviewing requires a unique set of protocols for recruiting, call-backs, and interviewing, all of which require special training for the people conducting the interviews.

Cost Issues: Including cell phones at least doubles the cost of a survey, and many times may triple or quadruple the cost.

Should you include cell phones in your survey?  It depends on who you are trying to reach and for what purposes.  When it comes to effective surveying, there are no absolutely right answers about cell phones vs. landlines, just as there are no absolutely right answers about online vs. phone surveys.  Moreover, the rapidly changing landscape of how people communicate means that the best answer today may be different in six months.

Need some help thinking through the options and figuring out the best approach for your research?  Feel free to give us a call.  We have decades of experience and are committed to helping our clients take full advantage of important changes and innovations in research.

—Joe Hopper, Ph.D.

The idea seems crazy, but as reported in Science, researchers at Carnegie Mellon University have found that certain kinds of twitter data can give them a good read on public sentiment. The looked at things like positive or negative comments about President Obama and found results that aligned with traditional polls. Comments about finances and savings aligned with consumer confidence polls. What does the polling industry think? Here’s what one colleague says: “I believe that I am now going to run to a dark corner of the house and cower in fear of what may come.”

To be sure, we are not recommending that you rely on social media to accurately measure overall public opinion. No way. But in our view, these findings may lead to new insights about how people, individually and collectively, behave and think, and how research — both academic and practical — can harness these new forms of data to measure markets and the social world.  The findings may also force us to re-think our theories of statistical inference that rely on random sampling. There is much research to be done before we will know, but possibilities for social media someday offering insights that are as statistically valid as our current methods is intriguing and nothing to cower about.

Thinking about measuring social media as part of your research plan? We would be happy to advise you. We can help you explore new options in research while ensuring that your research and findings remain rigorous and defensible.

—Joe Hopper, Ph.D.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.