The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes. So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population. Does it matter, given that surveys rarely interview everyone anyway? Probably. If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.
Archive for the ‘Public Polls’ Category
Survey Says: Call Me on My Cell Phone
Thursday, December 30th, 2010
Sampling for Dummies
Friday, December 10th, 2010The idea of sampling is not hard to grasp, but methodological discussions about sampling can quickly move into the higher reaches of mathematics and probability that confuse even researchers who are not statisticians. Ever wonder what we are talking about when we refer to “probability samples?” Or the rationale for not reporting margins of error? Or why there is no such thing as a “statistically significant sample size?”
AAPOR and the ASA are offering a great way for non-statisticians to learn more about sampling. AAPOR is the American Association of Public Opinion Research, and the ASA is the American Statistical Association. In February they are hosting an introductory webinar on sampling for non-statisticians that that we recommend. It is being taught by a senior statistician at NORC, and will cover such topics as: (more…)
Eliminate Your Margin of Error
Thursday, October 14th, 2010
Should you state a survey’s margin of error in your press release when pitching a story to the media? In our view, the answer is no. Why not? Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.
But how many readers of your news story understand what sampling error is? For that matter, how many researchers understand what sampling error is? It was only after four college semesters of advanced mathematical statistics that I finally “got it.” More importantly, how many readers understand that there are many other potential sources of survey error? Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%. (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)
In short, margins of error are misleading because they deal with only one source of error. They convey a false sense of accuracy. And they should not be used. This idea is not always popular among colleagues and clients. But take comfort: One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above. Here we quote their methodological statement that accompanies every press release and report they issue: (more…)
Why You Need a Partisan Pollster
Friday, September 17th, 2010
In an op-ed column two days ago, Stuart Rothenberg, a prominent political (and non-partisan) commentator argued that partisan pollsters (those who work directly for either Democratic or Republican candidates) do a better job than presumably objective third party pollsters. Why? Because they have to get it right. Their campaign strategies depend on it. Quoting Mr. Rothenberg: (more…)
How to Stop Fraudulent Polls
Friday, September 10th, 2010With the sad proliferation of silly surveys, non-scientific interest polls, and downright fraudulent polls, the research industry is stepping up with a number of key initiatives to combat the trend. Versta Research is part of that effort, and in August announced that we are part of the Transparency Initiative being developed by the American Association for Public Opinion Research (AAPOR).
As of this writing, Versta is one of 67 prominent survey and polling organizations currently supporting the initiative. The initiative is designed to create protocols and recognition for regular disclosure of methods when survey organizations conduct public polls. (more…)
Trouble for Phone Surveys: Nobody Talks
Thursday, June 3rd, 2010In the old days (decades ago), phone surveys had limited utility because many people had no phone service in their homes. When that changed, phone surveys became ubiquitous because they allowed researchers better control over the process. Data quality improved. Now increasing numbers of people have moved to cell phones only, which has been a significant challenge for the survey industry. The numbers are staggering:
The number of people without home access to landline telephones is increasing.
Almost half of adults under age 30 live in a household with only wireless telephone service.
Writing Successful Omnibus Survey Questions
Friday, May 14th, 2010Choosing an omnibus survey can be a simple approach to getting survey data, because it is usually inexpensive and fast, and involves asking just a few questions. But there is sometimes a downside to simplicity: You have just a few questions to get that nugget of data you’re hoping to use as a news hook or to provide insight to your client. If your key questions are off target, you can’t turn to other content in your survey to find something usable.
Here are four tips for writing omnibus survey questions to ensure that your effort is successful: (more…)
Practical Statistics vs. Theoretical Statistics
Friday, April 9th, 2010If something works and it keeps on working but you don’t know exactly why it works, what would you do? Our view is that you should keep doing it. Not everyone agrees with us. The American Association of Public Opinion Research (AAPOR) convened a task force to study online survey panels, and released their report last month (we posted a summary of findings last week). To us, the most jarring statement in the report was this:
“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”
Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters. Why? Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works. (more…)