Archive for the ‘Survey Design’ Category
Wednesday, January 18th, 2012
Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays. But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions. So it is helpful when our academic colleagues manage to document and codify the art and science of what we do. (more…)
Tags: bias, Internet, Market Research, Online Surveys, Sampling, statistics, Survey Design
Posted in Data Analysis & Analytics, Data Collection, Market Research, Online Surveys, Resources and Recommendations, Sampling, Survey Design | No Comments »
Wednesday, December 7th, 2011
In Versta Research’s Winter 2011 Newsletter, published just this week, we describe a simple method for estimating how long it will take respondents to complete surveys.
Here we offer the “Versta Digest” version as a handy reference card. Once you get the hang of it, you don’t need the examples and explanation. You just need to know the rules. We recommend reading the full article first, so you know what we’re talking about when it comes to “points.” Then, when you need a refresher or a reference source, consult these rules: (more…)
Tags: Data Collection, Online Surveys, phone surveys, research, Survey Design
Posted in Data Collection, Methods & Tools, Online Surveys, Survey Design | No Comments »
Wednesday, November 16th, 2011
Friends often solicit from me quick advice about conducting do-it-yourself customer satisfaction surveys. What questions should they ask? How many questions should they ask? What measures and scales should they use? And, of course, shouldn’t they be using NPS (Net Promoter Score) like everyone else?
I tell them that, by far, the most useful question they can ask is an open ended question that would be something like this: (more…)
Tags: insight, open-ends, satisfaction research, Survey Design
Posted in Survey Design, Survey Tips | No Comments »
Thursday, November 3rd, 2011
MaxDiff is a powerful method and it is increasingly popular among market researchers. But it is not always the best choice for measuring the importance of attributes, and here’s why.
Suppose you want to measure the importance of 12 attributes for a new product or service. If you know ahead of time that consumers are going to say that all 12 are extremely important to them, then MaxDiff is an excellent method for differentiating among the attributes so you can focus on the top two or three that matter most.
But what if you don’t know that all 12 attributes are extremely important? Maybe none of them are. Maybe they run the gamut from unimportant to extremely important. The problem with MaxDiff is that it only tells you the importance of attributes relative to each other, but it won’t tell you whether the attributes are important. The MaxDiff model will assign ratio-level numbers so that you can rank and quantify the importance of each attribute vis-à-vis the others. But it will not anchor the attributes in a meaningful way. (more…)
Tags: conjoint, Market Research, MaxDiff, Survey Design
Posted in Data Analysis & Analytics, Market Research, Methods & Tools, Survey Design | No Comments »
Thursday, June 2nd, 2011
We have always been big fans of the Net Promoter Score (NPS) metric because it has convinced many firms to begin using customer satisfaction measurement scales that work better and that are tied to what people do rather than what people think. Eleven point scales (with points zero to ten) allow for optimal variation. They are intuitive and appealing: people quickly grasp the idea of rating something on a zero to ten scale, and are familiar with the idea from grade school. They also have a neutral mid-point, which is important for many customer satisfaction and loyalty studies.
But NPS questions do not make sense in many situations. Here’s one we saw last week—it’s a survey sent by Amazon to sellers who call regarding complicated issues with how their products are being displayed on the website or how payments are being transferred:
A Poor Use of the Net Promoter Question
(more…)
Tags: measurement, satisfaction research, Survey Design, tracking studies
Posted in Market Research, Methods & Tools, Survey Design, Survey Tips | No Comments »
Thursday, May 12th, 2011
We are often surprised by the number of senior researchers in the market research industry who never touch raw data. Often they don’t even have the tools, since “data processing” is outsourced to lower levels or other countries. It is surprising because we almost always engage in work where getting into the data and puzzling over anomalies or hypotheses yields much deeper insight.
Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process. We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.
But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing. That’s an unusually high ratio of “suspends” as we call them. So what was the problem? Were we just getting lousy respondents who did not want to seriously participate in a survey? Was the survey was too difficult, tedious, boring, or confusing? One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.
The story in this data: Something is wrong with your survey
(more…)
Tags: data, Data Collection, data quality, insight, research, stories, survey, survey respondents
Posted in Charts and Data Visualization, Data Analysis & Analytics, Data Collection, Methods & Tools, Survey Design | No Comments »
Thursday, April 28th, 2011
As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys. A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion. The insanely popular survey question used to calculate Net Promoter Scores is a good example:
“How likely is it that you would recommend Acme Solutions to a friend or colleague? Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”
There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal. The NPS scale is good because it has eleven points with meaningful endpoints and a meaningful midpoint. Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.
But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult. They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:
A Poor Fit: Pie Charts and Numeric Scales
(more…)
Tags: charts, communication, Data Collection, journalism, media, news, omnibus, Public Relations, stories, Survey Design, visualizing data
Posted in Charts and Data Visualization, Data Collection, Omnibus Surveys, Presenting Research, Public Relations, Survey Design, Survey Tips, Turning Data into Stories | No Comments »
Saturday, April 16th, 2011
Among the many sources of potential error that can affect surveys are respondents themselves. They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.
This is not to say they are bad or fraudulent respondents. Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions. The problem with respondent error, it turns out, is poor survey design, which may involve biased or ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.
Recent research shows that grid-style questions that look like this:
or this: (more…)
Tags: bias, Data Collection, data quality, Market Research, Online Surveys, Survey Design, survey respondents
Posted in Data Collection, Market Research, Methods & Tools, Online Surveys, Survey Design, Survey Tips | No Comments »
Wednesday, March 30th, 2011
One resource that we give far too little thought in market research are the people who offer their time and thoughts about the stuff we are researching. They are truly the lifeblood of market research.
An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys. The news is mixed. While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great. The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:
Declining Satisfaction with Surveys
In addition to the chart above, key statistics outlined in the article include: (more…)
Tags: Data Collection, data quality, Market Research, public opinion, Public Polls, Survey Design, survey respondents
Posted in Data Collection, Future Trends, Market Research, Public Polls, Public Relations, Survey Design | No Comments »
Friday, March 18th, 2011
Learning the math behind market research is not easy because there is no programmatic way to master it as a body of learning. It is not like algebra, geometry, calculus, or statistics in high school or college. It is complex and multifaceted and draws upon nearly every area of theoretical mathematics, but it must be continually adapted to the needs and practical problems of measuring and predicting customer behaviors and attitudes.
So it requires both (1) a rigorous foundation in mathematics and (2) years of experience to understand how it gets re-worked and applied to the real-life questions of market research. And even the rigorous foundation needs to be continually reinforced and expanded as the scope of our capabilities expands. Twenty years ago, who would have thought that Bayesian statistics and Monte Carlo simulations would become so central to our work? (more…)
Tags: Market Research, mathematics, statistics
Posted in Data Analysis & Analytics, Market Research, Methods & Tools, Survey Design | No Comments »
