Asking people to fill out long, tiresome, and boring surveys is a scourge of the research, polling, and survey industry. (Another is asking them to fill out a survey every time they interact with you — see There Are Too Many Surveys.) Asking people to fill out long surveys teaches them to avoid surveys in the future, and indeed we see survey participation rates continuing to decline. But more importantly if you are the one who needs to rely on survey data, long surveys result in measurably lower data quality. (more…)
Posts Tagged ‘survey respondents’
How Long Should a Survey Be?
Friday, April 23rd, 2010Practical Statistics vs. Theoretical Statistics
Friday, April 9th, 2010If something works and it keeps on working but you don’t know exactly why it works, what would you do? Our view is that you should keep doing it. Not everyone agrees with us. The American Association of Public Opinion Research (AAPOR) convened a task force to study online survey panels, and released their report last month (we posted a summary of findings last week). To us, the most jarring statement in the report was this:
“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”
Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters. Why? Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works. (more…)
When to Kick Out a Survey Respondent
Thursday, March 4th, 2010Nearly every survey begins with screening questions to ensure that only the people you are trying to reach are included in the survey. For example, if you are conducting a survey of women, you need to ask about gender and kick out the men. And because every question costs money, you want to qualify respondents quickly and terminate those who do not belong.
Here is a helpful hint: Do not actually terminate respondents until after you have asked all screening questions. (more…)
Recent Findings on Phone vs. Online Surveys
Friday, February 5th, 2010A research article the Winter 2009 issue of Public Opinion Quarterly provided some useful comparisons of data quality between phone surveys and various types of online surveys. The findings are based on an experiment that fielded identical questionnaires via three survey modes, and, not surprisingly, there are strengths and weaknesses to each type of survey.
Phone vs. Online Surveys: Strengths & Weaknesses
As reflected in the table above, the article is rather technical. But there are two key summary points worth learning from the study: (more…)
People Don’t Lie on Surveys
Thursday, December 24th, 2009For some reason I’m always a little surprised that people tell the truth on surveys. I like to think of my “healthy skepticism” as a professional asset, because it forces us to check and double check, corroborate and triangulate. Before we commit to the findings of a research effort, we need to feel 100% sure we’re right and that our findings are based on solid data. (more…)
There Are Too Many Surveys
Thursday, November 12th, 2009You might think that a research firm specializing in surveys would be glad to see a world in which customer feedback surveys are everywhere. Not so. I take part in a lot of webinars, and unfortunately I am asked to complete a survey (at least one) every time I attend. Some websites ask me to evaluate my experience every time I go there. Some companies ask me to evaluate my customer service call every time I call with a question or complaint. The companies asking me to complete these surveys are convinced by pitches like this: “The real power of [our survey tool] can only be unlocked through a commitment to continuous listening. Listening to your visitors is truly a process, not an event. Continuous surveying can help you to establish benchmarks and trend your performance on key metrics.” (more…)
Better Data through Better Survey Design
Monday, September 14th, 2009Market researchers complain a lot when they get data back from surveys and see that people have been “speeding” through their surveys or that people are not giving thoughtful responses.
But the problem is rarely “bad respondents” – instead the problem is lazy researchers. When people discover that the survey they just agreed to take is boring, tedious, repetitive, or too long, they either quit altogether or they stop providing good answers.