Friends often solicit from me quick advice about conducting do-it-yourself customer satisfaction surveys. What questions should they ask? How many questions should they ask? What measures and scales should they use? And, of course, shouldn’t they be using NPS (Net Promoter Score) like everyone else?
I tell them that, by far, the most useful question they can ask is an open ended question that would be something like this: (more…)
In response to last week’s newsletter, Is Your Research Good Enough for The New York Times?, which discussed hurdles of getting online survey research reported by some news organizations, a customer reminded us that online surveys can be difficult to sell internally as well. Too many people have been burned by junk data from online surveys.
One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits. If these respondents are not providing thoughtful answers, the data are suspect. All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.
But we should not rely on panel providers alone to ensure valid data. Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases. Here are typical indicators of potentially bad data:
One of the surprises of multi-mode research is that offering people a choice of how they want to complete a survey (online, by mail, by phone, etc.) does not necessarily boost response rates. An article in the most recent issue of Public Opinion Quarterly provides new evidence of this. The study showed that even in a population with full access to both mail and Internet options (and full literacy in both modes), a full mail survey achieves a higher response rate than a web-based survey. It also achieves a higher response rate than a choice of either mail or web. So much for our recent article on The Myth of Too Many Choices!
But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates. Here are some key takeaways from the research: (more…)
We have always been big fans of the Net Promoter Score (NPS) metric because it has convinced many firms to begin using customer satisfaction measurement scales that work better and that are tied to what people do rather than what people think. Eleven point scales (with points zero to ten) allow for optimal variation. They are intuitive and appealing: people quickly grasp the idea of rating something on a zero to ten scale, and are familiar with the idea from grade school. They also have a neutral mid-point, which is important for many customer satisfaction and loyalty studies.
But NPS questions do not make sense in many situations. Here’s one we saw last week—it’s a survey sent by Amazon to sellers who call regarding complicated issues with how their products are being displayed on the website or how payments are being transferred:
A Poor Use of the Net Promoter Question
One reason that some people dislike surveys (okay, I may be projecting) is that too many surveys have the Me Generation attitude:
Enough about you, the customer, and what you need. What about ME? Do you like me? How much do you like me? Would you recommend me to your friend? Please let me know, because we need to track our satisfaction scores. It helps us build our metrics and our dashboards. And if you like me enough, I get a big bonus. Hurray for me!
Who are your customer satisfaction surveys really about?
As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys. A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion. The insanely popular survey question used to calculate Net Promoter Scores is a good example:
“How likely is it that you would recommend Acme Solutions to a friend or colleague? Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”
There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal. The NPS scale is good because it has eleven points with meaningful endpoints and a meaningful midpoint. Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.
But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult. They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:
A Poor Fit: Pie Charts and Numeric Scales
Among the many sources of potential error that can affect surveys are respondents themselves. They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.
This is not to say they are bad or fraudulent respondents. Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions. The problem with respondent error, it turns out, is poor survey design, which may involve biased or ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.
Recent research shows that grid-style questions that look like this:
or this: (more…)
After spending $65 million and previewing the show over sixty times since the end of last year, the producers of the new Spider-Man musical in New York are turning to focus groups and surveys in hopes that market research can do something…anything…to save the amazing Spider-Man from destruction.
Last week, theater critics roundly panned the show, calling it among the worst Broadway shows ever. The New York Times described is as “so grievously broken in every respect that it is beyond repair.” But days after these negative reviews, a market research firm was brought in, soliciting volunteers to help fix the show: (more…)
The technology to field surveys via mobile devices has been around for a while, but has not yet gained much traction (and for good reasons). But with smartphones now proliferating at a remarkable pace, we may be in for a change. Deloitte released their 2011 IT and technology predictions last week, arguing that smartphones are likely to account for almost half of computer spending during the year. (more…)
Contrary what a methodological purist in the social sciences might recommend, we often design survey questions with scales that have no neutral midpoint or don’t know option. For example, we sometimes use scales that look like this:
Rather than this:
Or this:
Why? Because survey respondents tend to over-use neutral midpoints or don’t know options. (more…)