Friends often solicit from me quick advice about conducting do-it-yourself customer satisfaction surveys. What questions should they ask? How many questions should they ask? What measures and scales should they use? And, of course, shouldn’t they be using NPS (Net Promoter Score) like everyone else?
I tell them that, by far, the most useful question they can ask is an open ended question that would be something like this: (more…)
In response to last week’s newsletter, Is Your Research Good Enough for The New York Times?, which discussed hurdles of getting online survey research reported by some news organizations, a customer reminded us that online surveys can be difficult to sell internally as well. Too many people have been burned by junk data from online surveys.
One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits. If these respondents are not providing thoughtful answers, the data are suspect. All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.
But we should not rely on panel providers alone to ensure valid data. Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases. Here are typical indicators of potentially bad data:
An issue we continually struggle with at Versta Research is how to automate the research process and leverage new technologies without losing the essence of what good research does. Good research does not report data, build charts, or generate dashboards. It learns, answers new questions, interprets data, and helps users focus on information and findings that are relevant to their needs.
The last couple of weeks we have been working with a group that specializes in coding and tabulating text responses to open-ended questions on surveys. They have tools and technology that undoubtedly make the process easier and more efficient (we have used those tools, and they are impressive). They are also have a singular focus and expertise that is supposed to help streamline the process, cut costs, and improve speed and efficiency.
The results have been mediocre at best, even with human coders working the technology and making the critical decisions. (more…)
Asking people to fill out long, tiresome, and boring surveys is a scourge of the research, polling, and survey industry. (Another is asking them to fill out a survey every time they interact with you — see There Are Too Many Surveys.) Asking people to fill out long surveys teaches them to avoid surveys in the future, and indeed we see survey participation rates continuing to decline. But more importantly if you are the one who needs to rely on survey data, long surveys result in measurably lower data quality. (more…)
Good open-ended questions and thoughtful responses to these questions can yield amazingly rich and insightful data. But given the generally poor quality of responses to open-ended question in mail surveys, some have speculated that open-ended questions in online surveys would suffer a similar fate.
Research seems to show this is not the case. Further, a study just published in the Summer 2009 issue of Public Opinion Quarterly outlines three ways that we can improve the quality of responses in on-line surveys: