The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes. So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population. Does it matter, given that surveys rarely interview everyone anyway? Probably. If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.
Posts Tagged ‘Sampling’
Survey Says: Call Me on My Cell Phone
Thursday, December 30th, 2010
Sampling for Dummies
Friday, December 10th, 2010The idea of sampling is not hard to grasp, but methodological discussions about sampling can quickly move into the higher reaches of mathematics and probability that confuse even researchers who are not statisticians. Ever wonder what we are talking about when we refer to “probability samples?” Or the rationale for not reporting margins of error? Or why there is no such thing as a “statistically significant sample size?”
AAPOR and the ASA are offering a great way for non-statisticians to learn more about sampling. AAPOR is the American Association of Public Opinion Research, and the ASA is the American Statistical Association. In February they are hosting an introductory webinar on sampling for non-statisticians that that we recommend. It is being taught by a senior statistician at NORC, and will cover such topics as: (more…)
Sampling for Concept Testing and Innovation
Friday, October 22nd, 2010Many assume that rigorous market research always works with random samples or probability samples. This assumption is not true. There are many studies that require purposive sampling instead. Purposive sampling involves finding people with specific characteristics or qualities, even if they do not fully represent the whole population, because these specific people can provide unique insights or data that are difficult to get otherwise.
An article in the October 2010 issue of the Journal of Marketing Research offers research to this point. The authors identify a unique profile of consumers who are measurably better at developing, testing, and reacting to new product ideas. The personal qualities these consumers possess include (quoting from the article): (more…)
Eliminate Your Margin of Error
Thursday, October 14th, 2010
Should you state a survey’s margin of error in your press release when pitching a story to the media? In our view, the answer is no. Why not? Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.
But how many readers of your news story understand what sampling error is? For that matter, how many researchers understand what sampling error is? It was only after four college semesters of advanced mathematical statistics that I finally “got it.” More importantly, how many readers understand that there are many other potential sources of survey error? Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%. (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)
In short, margins of error are misleading because they deal with only one source of error. They convey a false sense of accuracy. And they should not be used. This idea is not always popular among colleagues and clients. But take comfort: One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above. Here we quote their methodological statement that accompanies every press release and report they issue: (more…)
Genetics Affect Whether You Take Surveys
Wednesday, September 22nd, 2010Despite having been on the front lines of social research for twenty five years, interviewing respondents personally and eliciting data through surveys, I still feel somewhat surprised and disbelieving that people really want to participate in research. But they do. Sometimes eagerly. Almost always truthfully. Surely, my surprise stems from my own reluctance to fill out surveys.
It turns out that I may just lack the survey-taking gene. No joke. New research of genetic and fraternal twins shows that our willingness to participate in research is shaped in part by our genes. The research was led by Lori Foster Thompson, an associate professor of psychology at North Carolina State University, and is soon to be published in the Journal of Organizational Behavior. As reported on Science Daily: (more…)
Practical Statistics vs. Theoretical Statistics
Friday, April 9th, 2010If something works and it keeps on working but you don’t know exactly why it works, what would you do? Our view is that you should keep doing it. Not everyone agrees with us. The American Association of Public Opinion Research (AAPOR) convened a task force to study online survey panels, and released their report last month (we posted a summary of findings last week). To us, the most jarring statement in the report was this:
“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”
Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters. Why? Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works. (more…)
How Good Are Online Survey Panels?
Thursday, April 1st, 2010Ten years ago, surveys through online panels were rare. Mostly we conducted telephone surveys. Today it is the other way around. With online survey panels being a $2 billion industry in the U.S., there is now a wealth of comparative data to measure and assess the implications of using online surveys as a substitute for phone surveys.
The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report. It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.
Here is a summary of their conclusions and recommendations (quoted verbatim): (more…)
How to Conduct a Telephone Survey for Gold Standard Research
Thursday, February 25th, 2010Telephone surveys are still considered the gold standard for rigorous public opinion polling and market research. The reason is that virtually every household in the U.S. can be reached by telephone, and therefore we have careful methods of determining the probability that any individual person is included in a sample to be surveyed. Knowing this probability is at the core of statistical inference, which makes mathematical purists very happy.
Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population: (more…)