We like this cartoon because it highlights the unrealized potential of really smart research, but also the potential perils of research gone bad.
The cartoon brings to mind three lessons worth pondering:
The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes. So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population. Does it matter, given that surveys rarely interview everyone anyway? Probably. If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.
Survey respondents and other participants in our research studies are the lifeblood of the polling and market research industry. Therefore it is critical that we promote and nurture public participation in research, and that we reward, respect, and thank our respondents.
So whatever you do, please don’t take Dogbert’s approach to conducting a survey:
Versta Research believes strongly in protecting the privacy of all research respondents. We adhere to the strictest standards of ethics and privacy as outlined by The Council of American Survey Research Organizations (CASRO), the American Association of Public Opinion Research (AAPOR), and other industry organizations to which we belong. The following two paragraphs nicely summarize our views and approach: (more…)
Should you state a survey’s margin of error in your press release when pitching a story to the media? In our view, the answer is no. Why not? Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.
But how many readers of your news story understand what sampling error is? For that matter, how many researchers understand what sampling error is? It was only after four college semesters of advanced mathematical statistics that I finally “got it.” More importantly, how many readers understand that there are many other potential sources of survey error? Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%. (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)
In short, margins of error are misleading because they deal with only one source of error. They convey a false sense of accuracy. And they should not be used. This idea is not always popular among colleagues and clients. But take comfort: One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above. Here we quote their methodological statement that accompanies every press release and report they issue: (more…)
With the sad proliferation of silly surveys, non-scientific interest polls, and downright fraudulent polls, the research industry is stepping up with a number of key initiatives to combat the trend. Versta Research is part of that effort, and in August announced that we are part of the Transparency Initiative being developed by the American Association for Public Opinion Research (AAPOR).
As of this writing, Versta is one of 67 prominent survey and polling organizations currently supporting the initiative. The initiative is designed to create protocols and recognition for regular disclosure of methods when survey organizations conduct public polls. (more…)
We feel strongly that people who give us information through surveys, in-depth interviews, or focus group deserve a promise that this will never happen to them:
One of the tenets of rigorous market research is that respondent confidentiality is key. Why? Primarily because it benefits you as a client. To make smart decisions, you need honest and thoughtful information from your customers and constituents. In most cases, your customers want to give us that information (because they want you to do a better job) as long as it won’t come back to haunt them.
Versta Research adheres to the ethics guidelines for privacy as outlined by CASRO, AAPOR, and other industry organizations to which we belong. The following two paragraphs nicely summarize our views and approach: (more…)
Bad pollsters give the market research industry a bad name, so it is encouraging when smart people figure out clever ways of ratting them out. What is a bad pollster? One who makes up data to support an agenda, or who asks biased questions to get preferred answers. The only good reason for doing research or public opinion polling is to learn or share something new. All else is suspect.
Two researchers recently came up with methods of testing whether polling data is legitimate in a case where a research firm is accused of falsifying publicly released data. (more…)
