Friends often solicit from me quick advice about conducting do-it-yourself customer satisfaction surveys.  What questions should they ask?  How many questions should they ask?  What measures and scales should they use?  And, of course, shouldn’t they be using NPS (Net Promoter Score) like everyone else?

I tell them that, by far, the most useful question they can ask is an open ended question that would be something like this:

If there were one thing you would like us to improve, what would it be?

You can tailor the wording to your specific market, service, product, or situation.  Best of all, you do not need to code and tabulate all the responses, although you could and might want to.  You just need to read through them and act on them if it makes sense.

I offer this advice with conviction based on the experience of a business colleague who implemented an ongoing web satisfaction survey ten years ago.  I suggested he include a question like this.  He has told me several times over the years it is the only data he regularly reviews and acts upon.  Almost invariably a customer will point him towards a functional glitch on his website or offer a simple suggestion that would make it easier for customers to buy.

He asks other survey questions as well, measuring satisfaction across several dimensions.  And surely a robust effort to analyze that data might tell him what drives satisfaction and how to boost his levels of satisfaction from 85% to, say, 90%.  But what he really wants to know day in and day out is what he can fix to keep improving, and whether any of his customers have a great idea he never thought of that he can easily incorporate.  He gets that with this one question.

His experience is a testament to the idea that one question can provide a great deal of useful insight.  Sometimes one question is all you need.  If so, you’re all set.  If not, give us a call.  We’ll advise you on the next nine questions you need on your survey as well.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/the-one-question-you-need-on-your-survey/feed/ 0 http://www.verstaresearch.com/blog/7-ways-to-spot-bad-data/ http://www.verstaresearch.com/blog/7-ways-to-spot-bad-data/#comments Wed, 05 Oct 2011 20:57:24 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1638 In response to last week’s newsletter, Is Your Research Good Enough for The New York Times?, which discussed hurdles of getting online survey research reported by some news organizations, a customer reminded us that online surveys can be difficult to sell internally as well.  Too many people have been burned by junk data from online surveys.

One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits.  If these respondents are not providing thoughtful answers, the data are suspect.  All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.

But we should not rely on panel providers alone to ensure valid data.  Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases.  Here are typical indicators of potentially bad data:

1.  Speeding. Though people can legitimately whiz through surveys at varying speeds, we typically flag the fastest five percent for further investigation.

2.  Non-sense open ends.  People who have nothing to say will usually say that, so we flag respondents who type random letters, offer non-sense or vacuous answers, or skip answering entirely.

3.  Choosing all options on a screening question.  Often it  means the respondent was gaming the survey to get in, especially if some options logically exclude others.

4.  Failing quality check questions.  Usually we include a couple of questions that have only one correct response to flag respondents who are not paying attention.

5.  Inconsistent numeric values.  How long a person has worked in a profession or at a particular job, for example, must be consistent with a person’s age.

6.  Straight-lining and patterning.  If questions are laid out in grids, respondents who answer identically for all questions, or who move in a diagonal along the grid should be flagged for investigation.

7.  Logically inconsistent answers.  If attitude and behavior questions are logically related to each other (for example, multiple questions about concern for the environment), inconsistent responses may indicate bad data.

The customer who reminded us that online surveys face multiple hurdles had just gotten results from a survey that she discovered had included a respondent who took the survey 250 times.  Nobody from the research firm bothered to look at the data beyond feeding it into the data-tabulator-chart-maker-here-are-your-actionable-insights machine.

At Versta Research, our approach is the opposite.  Smart people look at your data at each step because there is no other way to turn data into a story that you can trust and then share with your management team with confidence.

–Joe Hopper, Ph.D.

But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates.  Here are some key takeaways from the research:

• A pre-notification postcard sent by mail will significantly improve response rates to your online survey

• Offering a token cash incentive (a couple dollars) sent by mail in advance will significantly improve response rates to your online survey

• As a final step, offering a mail survey to non-respondents will boost response rates even further

So why even bother with online surveys, you may wonder?  Indeed, even with all this, response rates are no better than conducting a mail-only survey.  The answer is that online surveys have a number of advantages worth keeping.  Programmed skip logic and constraints ensure no missing data or misinterpretation of skip patterns.  Data are entered into a database real-time, eliminating errors from manual data entry or optical scanning.  The process is faster, more efficient, and usually less expensive.

While many people overestimate the importance of response rates on a survey’s validity, there is no doubt that higher response rates are better in terms of cost, efficiency, and feasibility.  So if you are thinking about deploying an online survey to your customers or members, it is probably worth investing in some old fashioned mail-based outreach.

Need help?  We can be reached by phone (312-348-6089), or Internet, or mail (919 Forest Ave, Evanston, IL  60202).  Whichever way you contact us, you’ll get a 100% response rate within a few hours.

—Joe Hopper, Ph.D.

But NPS questions do not make sense in many situations.  Here’s one we saw last week—it’s a survey sent by Amazon to sellers who call regarding complicated issues with how their products are being displayed on the website or how payments are being transferred:

A Poor Use of the Net Promoter Question

The problem with this question is that any honest respondent will almost certainly answer “very unlikely” and give a score of zero.  The support representatives are anonymous, so it is impossible to recommend them.  If you call or email Amazon, you cannot request a specific representative, making the issue moot.  And even if you could recommend a specific representative, how often would you have the opportunity to make such a recommendation?  Few sellers talk to other sellers about the display and payment problems they are having.  So all in all, how likely do you think most sellers are to recommend the specific person they talked with?  If we have any faith in the reliability and validity of survey data, it should be about zero.

The hapless support representative likely got a lousy score (there goes her bonus!) not because she did a poor job, but because whoever is in charge of this tracking survey didn’t think very hard about NPS, how to use it and when to use it.  More generally, somebody didn’t think very hard about how to do smart research.

Staying on top of recent innovations, important industry trends, and best practices is critical.  We can help you with that, with suggestions about the right ways to apply them so that you get valid insights.  Your customers and your service reps will thank you.  And the next time your manager and executive team rate you, they will likely recommend you for the next important research project or satisfaction tracking study (score 9 or 10).

—Joe Hopper, Ph.D.

Enough about you, the customer, and what you need.  What about ME?  Do you like me?  How much do you like me?  Would you recommend me to your friend?  Please let me know, because we need to track our satisfaction scores.  It helps us build our metrics and our dashboards.  And if you like me enough, I get a big bonus.  Hurray for me!

Who are your customer satisfaction surveys really about?

I was reminded of this recently when I asked a client what his objectives were in surveying customers, and what he hoped to learn.  He responded that he needed to survey his customers because he was expected to.  Surveying customers was a documented best practice, and he needed the scores for his customer service award entry.

We suggest an opposite approach.  Focus as much as you can on your customers, what they need, and why they need it.  And to get your customers engaged in the survey, tell them how it will help them.

Here is a nice example.  This past weekend I attended the American Marketing Association’s annual three-day leadership summit.  It brought together roughly 200 marketing leaders and chapter heads from around the country to talk about marketing excellence and how to meet the needs of marketing professionals.

The organizers of this event asked us to give them feedback via a post-event survey.  They told us why it was important (it gives them critical information to improve the event for next year) and importantly, they told us specific improvements they had made based on previous survey results. “Last year you said you wanted time to converse over afternoon snacks.  So this year we brought back the snacks.”  This may sound trivial, but it spoke volumes.  It said, “We listen. We care. We’re doing this for you.”

All your surveys should focus on your customers, not on you.  Research should be driven by critical business questions, not methods or best practices.  A simple effort to tell your customers specifically how you have improved a product or service for them based on their previous feedback will improve response rates, and most importantly, it will shift your thinking about what research is and why you are doing it.

Need help?  Working with Versta Research puts a welcome burden on somebody else to worry about you, your best practices, and your award entry.  We will keep you focused on what really matters, which is your customers, who are far more interested in what they will get for desert than they are in helping you build your dashboard.

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

This is not to say they are bad or fraudulent respondents.  Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions.  The problem with respondent error, it turns out, is poor survey design, which may involve biased or  ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.

Recent research shows that grid-style questions that look like this:

or this:

. . . hurt the reliability of answers from respondents.  The evidence for this conclusion comes from one of the leading academic authorities on survey design, Duane Alwin, at Penn State.  Here we quote from a Harvard reviewer who summarized the findings in a recent journal article:

“Of particular note is Alwin’s finding that the widespread survey practice of presenting items in batteries—sets of consecutive questions using the same response format—tends to yield less reliable responses than presenting them alone or in a series of topically related questions with differing response formats. He conjectures that “[s]imilarity of question content and response format may actually distract a respondent from giving full attention to what information is being asked” (p.180). Some might anticipate that similarity in response format would instead heighten reliability, by raising correlations among items in a battery. Alwin’s reliability estimates do not depend on within-occasion correlations between different items,however: the finding suggests that respondents vary across occasions in how they use a battery’s response format.”

In short, even if you have good respondents who are not “straight-lining” through your grids, they may be focusing more on the task of filling out the grid than on a thoughtful response to each item in the grid.  So instead, lay out your questions one at a time, like this:

And when you’re not sure what to do next, namely how to turn all of that now-reliable data into a story that you can really use, give Versta Research a call.

—Joe Hopper, Ph.D.

Last week, theater critics roundly panned the show, calling it among the worst Broadway shows ever. The New York Times described is as “so grievously broken in every respect that it is beyond repair.”  But days after these negative reviews, a market research firm was brought in, soliciting volunteers to help fix the show:

If you love Broadway shows, we would like to invite you to be a test audience member and participate in either Act 1 (first part of show) or Act 2 (second part of show) of the all new Broadway musical “Spider-Man: Turn Off the Dark.”

We will be asking you to fill out a survey after the performance, plus stay for a 15 minute discussion in the Spider-Man V.I.P. room. In turn for your feedback and participation you will receive a Spider-Man goodie bag worth over $60.00, on us.

Can market research really rescue the show?  Possibly.  High quality research, both qualitative (such as focus groups) and quantitative (such as surveys) can provide deep insights into how customers and audiences think, perceive, react, and respond to products, services, and opportunities.

If market research can help rescue the show, here are two suggestions we offer that will be critical to keep in mind:

1.  Keep the focus groups focused. Too often clients or managers want their customers to tell them what would make them love their product.  That’s not something customers can reliably tell you.  They can tell you what they care about, what products they purchase and why, what needs and problems they are trying to solve, and what frustrations they face.  Our advice generally: Keep the focus group participants focused on the things that matter to them, then connect the insights gained to the products being offered.

2.  Ask only about things that can be fixed. For all the talk about “actionable” research, there is a great deal of research that offers supposedly actionable ideas that clients or managers simply cannot and will not act upon.  How does the audience feel about Bono’s music for the show?  Unless management is prepared to dump Bono or have him re-write (again), don’t ask.  The key is to determine during the research design which specific decisions can and cannot be made, and then to focus the research on collecting data that lends specific insight to those decisions.

Asking market research to help rescue a $65 million investment at the last minute seems like a tall order, but fortunately great research does not take super-human powers.  At Versta Research we rely on brains, experience, thoughtfulness, and a commitment to working closely with clients to ensure that we deliver insightful stories that are understood and acted upon.

So what will become of Spider-Man?  Will he die under the crushing weight of debt and harsh reviews?  Or will the critics groan in agony as Americans happily pay for the worst show on earth?  Stay tuned for the next exciting episode of “Can a Focus Group Save Spider-Man?”

—Joe Hopper, Ph.D.

When it comes to surveys for market research and public polling, smartphones have bigger and better screens, better technology, and web capabilities.  But more importantly, smartphones are becoming the “device of choice” for tasks that were once the domain of our desktop computers. For example, web-based e-mail via PCs is on the decline, while email through mobile devices is growing rapidly.  And of course unless we’re doing phone surveys, e-mail is currently the primary means by which researchers recruit survey respondents.

Leora Lawton, a colleague in AAPOR (the American Association of Public Opinion Research) recently reported back with some tips on survey design for smartphones that she synthesized from a talk by Mario Callegaro of Google.  In brief, she reports:

• Avoid long surveys
• Do not use question grids
• Scales should be a maximum of five points
• Show one question per page
• Put navigation buttons on top
• Use vertical, not horizontal, buttons
• Avoid logos, progress meters, help links, and copyrights that take up space
• Avoid drop-down menus that require JavaScript

Of course technologies change quickly, and enhanced capabilities and tools for effective and sophisticated smartphone surveys are in development.  But for now these are good ideas to keep in mind.

Need help thinking about the best approach for your research, whether it be telephone, in-person, online, or smartphone-based?  Feel free to give us a call; we would be happy to help.

—Joe Hopper, Ph.D.

Contrary what a methodological purist in the social sciences might recommend, we often design survey questions with scales that have no neutral midpoint or don’t know option. For example, we sometimes use scales that look like this:

Rather than this:

Or this:

Why? Because survey respondents tend to over-use neutral midpoints or don’t know options. Often their opinions are weak (or ambivalent) and it is easier to pick the middle option rather than thinking through the best answer that truly represents their opinion. A large majority of these fence-sitters can and will choose a non-neutral option if skillfully probed with additional questions, and the outcome will be nearly the same as if the neutral option were not offered in the first place.

There are, however, many situations in which it is important to include a neutral midpoint. For example, in campaign polling we often want to know not only the direction of opinion, but the intensity as well. Why? Because identifying (and targeting) the pool of fence-sitters, especially the ones who lean to our side, is critical.

Another situation when a neutral midpoint is important is when neutrality itself is theoretically important—that is, when it reflects a truly meaningful idea rather than just uncertainty or ambivalence.

Here is an example from a survey we just did. Respondents were asked to review the content of a website and then evaluate the usefulness of that website. As part of the survey, they were asked:

In this case, no difference does not reflect uncertainty or ambivalence, but something important and meaningful.  In fact, our client would like to see most people reporting no difference.

As always, good research design must be driven by the unique objectives and specific questions that need to be answered.  With response scales, that means thinking about the most appropriate way to construct a measure for every question rather than relying on a library of boilerplates or thoughtlessly applying standard rules to every situation.

Need help deciding upon the best scales to use for your research questions?  Versta Research would be happy to advise you so that any neutral, don’t know, or no difference options you include will make a huge difference in the insights you get.

—Joe Hopper, Ph.D.