Posts Tagged ‘bias’
Wednesday, October 31st, 2012
By elephants, we mean Republicans. Or maybe you have too many Democrats. Maybe it keeps going back and forth, which is the problem that Gallup sometimes has. In the spirit of learning all we can from election season polling, this week we focus on whom to include (or exclude) in your research, analysis, and market projections.
The issue is showcased right now as political polls attempt to measure voter preference and predict the election outcome. Is voter preference really as volatile and open to persuasion as the polls sometimes suggest? Probably not. A 2004 research article in Public Opinion Quarterly carefully documented that much of the volatility in Gallup’s polls results from how they screen respondents and weight their data. (more…)
Tags: bias, concept testing, data analysis, Market Research, Public Polls, Sampling, statistics
Posted in Data Analysis & Analytics, Market Research, Methods & Tools, New Products and Innovation, Sampling, Survey Design | Comments Off
Thursday, August 23rd, 2012
Way back in 1944, Edwards Deming published an article in the American Sociological Review that could be required reading for anybody who does research today. He outlined all potential (and unfortunately, common) sources of error in survey research.
Apparently our contemporary obsession with sample sizes, random samples, response rates, and margins of error is not so new. In outlining all sources of error, Demining wanted to emphasize that “sampling errors, even for small samples, are often the least of the errors present.”
So despite some old-fashioned language and defunct technologies (Versta Research has never fielded a survey via telegraph!) we feel it is worth reproducing here what Deming called the thirteen factors “affecting the ultimate usefulness of a survey” as all of them apply as much today as they did 68 years ago:
(more…)
Tags: bias, margin of error, Online Surveys, response rates, Sampling, survey, Survey Design
Posted in Data Analysis & Analytics, Data Collection, Market Research, Public Polls, Sampling, Survey Design | Comments Off
Wednesday, April 25th, 2012
Most people who take surveys want to share their opinions, which is important for researchers hoping to get a few nuggets of data from willing respondents. The trouble is, if a survey it not written carefully, a respondent’s urgent desire to share their feelings may bias their answers to other questions.
Two marketing professors at Northwestern University’s Kellogg School of Management recently published a paper in the AMA’s Journal of Marketing Research identifying and documenting this unique kind of bias. They call it “response substitution.” (more…)
Tags: bias, customer satisfaction, satisfaction research, survey, survey respondents
Posted in Market Research, Methods & Tools, Survey Design, Survey Tips | Comments Off
Wednesday, April 11th, 2012
How big of a sample size do you really need? A recent article in the New York Times cited the following statistics:
- A small Voice of the Customer (VoC) research company called Mindshare Technologies collects satisfaction data from 175,000 respondents every day. That’s 60 million in a year.
- ForeSee, a small customer experience analytics firm fielded 15 million surveys in 2011.
These numbers are believable. I get a pop-up survey from ForeSee at least two or three times a week.
And it is absurd. Granted, these companies (and hundreds of other similar firms) are collecting surveys for multiple clients. But almost certainly, nobody needs to collect that much survey data from that many survey respondents. Why not? (more…)
Tags: bias, data, Data Collection, data mining, insight, Market Research, response rates, Sampling, satisfaction research, survey respondents, survey technology
Posted in Data Analysis & Analytics, Data Collection, Market Research, Methods & Tools, Sampling | Comments Off
Wednesday, January 18th, 2012
Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays. But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions. So it is helpful when our academic colleagues manage to document and codify the art and science of what we do. (more…)
Tags: bias, Internet, Market Research, Online Surveys, Sampling, statistics, Survey Design
Posted in Data Analysis & Analytics, Data Collection, Market Research, Online Surveys, Resources and Recommendations, Sampling, Survey Design | Comments Off
Tuesday, August 9th, 2011
These days most researchers agree that if you want to do a random sample phone survey of the U.S. population, you ought to include cell phones. More than one-quarter of the population do not have landline telephones at home. Those who do have landline telephones are less likely than ever to answer them, and less likely than ever to participate in surveys.
But it is not easy to include cell phones. The sampling protocols and the post-stratification weighting become more complicated. You need to account for a higher probability of cell phone owners being in your sample, because most of them also have landlines. You can’t use automated or predictive dialing to call cell phone numbers. You can’t target geography as well, because area codes and exchanges have become mobile. And people get mad at you if they have to pay for incoming calls, so you need to offer cash.
What’s the bottom line effect on costs for a survey that includes cell phones? A recent study sponsored by the American Association for Public Opinion Research (AAPOR) documents the following: (more…)
Tags: bias, Data Collection, Market Research, phone surveys, population, public opinion, Public Polls, research, Sampling
Posted in Data Collection, Future Trends, Market Research, Public Polls, Sampling | Comments Off
Thursday, May 26th, 2011
A new study presented by two professors from Harvard University and the University of Massachusetts at Amherst was probably one of the liveliest and potentially disruptive presentations at least week’s annual meeting of the American Association of Public Opinion Research (AAPOR) in Phoenix.
Why? Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling. The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes: (more…)
Tags: bias, census, Data Collection, Internet, journalism, media, news, Online Surveys, phone surveys, public opinion, Public Polls, Public Relations, Sampling
Posted in Data Collection, Methods & Tools, Online Surveys, Public Polls, Public Relations, Sampling | Comments Off
Saturday, April 16th, 2011
Among the many sources of potential error that can affect surveys are respondents themselves. They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.
This is not to say they are bad or fraudulent respondents. Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions. The problem with respondent error, it turns out, is poor survey design, which may involve biased or ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.
Recent research shows that grid-style questions that look like this:
or this: (more…)
Tags: bias, Data Collection, data quality, Market Research, Online Surveys, Survey Design, survey respondents
Posted in Data Collection, Market Research, Methods & Tools, Online Surveys, Survey Design, Survey Tips | Comments Off
Thursday, December 30th, 2010
The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes. So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population. Does it matter, given that surveys rarely interview everyone anyway? Probably. If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.
(more…)
Tags: bias, Data Collection, data quality, ethics, Market Research, phone surveys, public opinion, Public Polls, research, Sampling, Survey Design, survey respondents
Posted in Data Collection, Methods & Tools, Public Polls, Sampling | Comments Off
Wednesday, September 22nd, 2010
Despite having been on the front lines of social research for twenty five years, interviewing respondents personally and eliciting data through surveys, I still feel somewhat surprised and disbelieving that people really want to participate in research. But they do. Sometimes eagerly. Almost always truthfully. Surely, my surprise stems from my own reluctance to fill out surveys.
It turns out that I may just lack the survey-taking gene. No joke. New research of genetic and fraternal twins shows that our willingness to participate in research is shaped in part by our genes. The research was led by Lori Foster Thompson, an associate professor of psychology at North Carolina State University, and is soon to be published in the Journal of Organizational Behavior. As reported on Science Daily: (more…)
Tags: bias, Sampling, survey respondents
Posted in Data Collection, Market Research, Sampling | Comments Off