It’s our job to deliver bad news as well as good news, right? To tell clients what they’re doing wrong so they can fix their problems and leap to the next level of profitability, right? Why would they spend money collecting data if they just wanted to hear how much customers love them? In fact, why would they want to hear how much customers love them, if the research says otherwise?
A recent study published in the Journal of Consumer Research suggests some answers that may surprise you. (more…)
I have never been a media poll watcher or politics junkie who tracks every new poll saying which candidate is favored to win. It is hard to see the point of spending so much time and money predicting the outcome of an event that will be known with certainty within a matter of days, weeks, or months. But elections are an amazing way to see survey research methods in action, and there are few opportunities to have those methods validated so quickly, accurately, and unforgivingly than political polling.
What did we learn from the polls this election season? (more…)
Figure 1. The U.S. Census Bureau's current questions about race and ethnicity. Click for full-size image.
Fifteen years ago the U.S. Census Bureau removed “Hispanic” from its question about race and instead created a stand-alone question to assess Hispanic ethnicity. It’s the reason we often advise our clients to ask about race and ethnicity separately, using questions similar to the Census Bureau’s, which are shown in Figure 1.
Well, chances are good that this will be soon be revised back to include Hispanic in a single question about race. The Census Bureau just concluded a large-scale study to assess the reliability and validity of its race and ethnicity measures, along with several potential alternatives. It involved questionnaires and re-interviews with a sample of nearly half a million households, plus 67 focus groups with nearly 800 people. (more…)
During a presidential election year there is no escaping the flurry of public opinion polling and the intense scrutiny that surveys get from the media. But love it or hate it, there are excellent reasons to pay close attention to this year’s political polling.
Way back in 1944, Edwards Deming published an article in the American Sociological Review that could be required reading for anybody who does research today. He outlined all potential (and unfortunately, common) sources of error in survey research.
Apparently our contemporary obsession with sample sizes, random samples, response rates, and margins of error is not so new. In outlining all sources of error, Demining wanted to emphasize that “sampling errors, even for small samples, are often the least of the errors present.”
So despite some old-fashioned language and defunct technologies (Versta Research has never fielded a survey via telegraph!) we feel it is worth reproducing here what Deming called the thirteen factors “affecting the ultimate usefulness of a survey” as all of them apply as much today as they did 68 years ago:
Is this YOUR distracted survey respondent?
A couple weeks ago we presented new data showing that response rates continue to decline. You can now expect that a typical, rigorously executed phone survey will yield a response rate in the single digits.
Scientific evidence over the last decade has shown that high response rates do not necessarily yield more accurate surveys. In fact, it turns out that high response rates can actually hurt the accuracy of surveys.
Survey response rates are now staggeringly low—in the single digits. A typical response rate for a relatively high-budget, carefully executed phone survey is merely 9%, down from 36% just fifteen years ago. Here are the numbers from research conducted earlier this year by the Pew Center:
Survey Response Rates Continue to Decline
If you want to throw money at a survey and try really hard to boost your response rate (the high-effort survey shown in the chart above), you can likely get up to 20% to 25%. But you will need to:
I recently saw a press release about a study showing that only 19.5% of news release headlines are optimized for SEO. It brought to mind all kinds of issues about how best to report numbers in press releases. In particular it highlighted the important issue of whether specific numbers are meaningful and whether they communicate a misleading sense of precision.
For example, when a survey reports a margin of error to any decimal place, it suggests a level of precision that is misleading. Do a quick search, and you’ll find press releases reporting margins of sampling error such as +/- 4.8%, +/- 10.5%, or +/- 1.85%. These numbers are based on sample size formulas that assume perfect random sampling and one hundred percent response rates, which are almost never achieved. (more…)
Most people who take surveys want to share their opinions, which is important for researchers hoping to get a few nuggets of data from willing respondents. The trouble is, if a survey it not written carefully, a respondent’s urgent desire to share their feelings may bias their answers to other questions.
Two marketing professors at Northwestern University’s Kellogg School of Management recently published a paper in the AMA’s Journal of Marketing Research identifying and documenting this unique kind of bias. They call it “response substitution.” (more…)
Data quality will suffer if respondents are bored with long surveys
It is hard to resist the temptation of asking “just one more question” when you’ve got an engaged respondent answering your survey questions online or on the phone. But it is crucial to do so because plenty of research shows that longer surveys result in bad data. Survey respondents may be willing to answer just one more question, but at some point the quality of information you get from them declines. Survey respondents become inattentive and offer lazy answers, or worse, they offer quick random answers just to get the survey over with.
At Versta Research we have a few rules of thumb for survey length based on (1) academic and industry research measuring data quality, (2) conversations with colleagues and suppliers throughout the industry, and (3) our ongoing experience of what works and what does not work. The maximum survey lengths we typically recommend are: (more…)