A good chart is the best way to understand the law of diminishing returns when it comes to sample size. So for our June 2011 newsletter we built an interactive graph for choosing sample size. It’s cool, educational, and useful. Moreover, it will show you just how mind boggling the numbers behind sampling can be. It may even give you more sympathy for the majority of people who just don’t “get it” or believe it when it comes to statistical sampling.
Archive for the ‘Data Collection’ Category
An Interactive Graph for Choosing Sample Size
Thursday, June 9th, 2011Online Surveys Have Same Accuracy as Phone
Thursday, May 26th, 2011A new study presented by two professors from Harvard University and the University of Massachusetts at Amherst was probably one of the liveliest and potentially disruptive presentations at least week’s annual meeting of the American Association of Public Opinion Research (AAPOR) in Phoenix.
Why? Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling. The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes: (more…)
How Data Can Highlight Mistakes
Thursday, May 12th, 2011We are often surprised by the number of senior researchers in the market research industry who never touch raw data. Often they don’t even have the tools, since “data processing” is outsourced to lower levels or other countries. It is surprising because we almost always engage in work where getting into the data and puzzling over anomalies or hypotheses yields much deeper insight.
Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process. We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.
But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing. That’s an unusually high ratio of “suspends” as we call them. So what was the problem? Were we just getting lousy respondents who did not want to seriously participate in a survey? Was the survey was too difficult, tedious, boring, or confusing? One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.
The story in this data: Something is wrong with your survey
Advice for PR Surveys: Avoid Numeric Scales
Thursday, April 28th, 2011As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys. A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion. The insanely popular survey question used to calculate Net Promoter Scores is a good example:
“How likely is it that you would recommend Acme Solutions to a friend or colleague? Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”
There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal. The NPS scale is good because it has eleven points with meaningful endpoints and a meaningful midpoint. Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.
But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult. They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:
A Poor Fit: Pie Charts and Numeric Scales
Reasons to Avoid Grid-Format Questions
Saturday, April 16th, 2011Among the many sources of potential error that can affect surveys are respondents themselves. They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.
This is not to say they are bad or fraudulent respondents. Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions. The problem with respondent error, it turns out, is poor survey design, which may involve biased or ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.
Recent research shows that grid-style questions that look like this:
or this: (more…)
What People Think of Surveys
Wednesday, March 30th, 2011One resource that we give far too little thought in market research are the people who offer their time and thoughts about the stuff we are researching. They are truly the lifeblood of market research.
An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys. The news is mixed. While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great. The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:
Declining Satisfaction with Surveys
In addition to the chart above, key statistics outlined in the article include: (more…)
When to Use Paper Surveys
Thursday, March 3rd, 2011
One might think that with evolving technologies, the old-fashioned ways of doing research, like using paper and pencil, might disappear. But it turns out that paper surveys are not dead, and here is a nice example that dramatically illustrates their value.
As newly-appointed director of market research for the American Marketing Association in Chicago, I oversee efforts to develop and organize events related to market research. I also oversee efforts to measure, track, and analyze customer satisfaction for all other events and activities sponsored by the AMA. For the past few events, the AMA collected feedback via an online survey sent by e-mail immediately after the events. They were also using a less-than-stellar do-it-yourself survey tool that promises to deliver “actionable insights!” at the click of a button.
The results were abysmal. (more…)
Tips for Surveys on Smartphones
Thursday, January 27th, 2011The technology to field surveys via mobile devices has been around for a while, but has not yet gained much traction (and for good reasons). But with smartphones now proliferating at a remarkable pace, we may be in for a change. Deloitte released their 2011 IT and technology predictions last week, arguing that smartphones are likely to account for almost half of computer spending during the year. (more…)
Make it Real with Adaptive Conjoint
Thursday, January 13th, 2011In a recent survey we fielded among B2B decision-makers, respondents told us how much they liked participating in the study compared to other research studies they have done. They said it was “real” and interesting because it was confronting them with questions that reflect the kinds of decisions and trade-offs they make every day in their work.
The technique we used for that study is called Adaptive Conjoint. If you want to know how people in your target audience make decisions—how they weigh the pros and cons of your product or service versus others— adaptive conjoint can be a powerful technique that provides robust and insightful data at the same time it really engages the participants. (more…)
Survey Says: Call Me on My Cell Phone
Thursday, December 30th, 2010The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes. So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population. Does it matter, given that surveys rarely interview everyone anyway? Probably. If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.
