A good chart is the best way to understand the law of diminishing returns when it comes to sample size. So for our June 2011 newsletter we built an interactive graph for choosing sample size. It’s cool, educational, and useful. Moreover, it will show you just how mind boggling the numbers behind sampling can be. It may even give you more sympathy for the majority of people who just don’t “get it” or believe it when it comes to statistical sampling.
Posts Tagged ‘Market Research’
An Interactive Graph for Choosing Sample Size
Thursday, June 9th, 2011The Myth of Too Many Choices
Thursday, April 21st, 2011Ever since the well-publicized “jam” experiments published ten years ago, product managers have been cautious about assuming that more choices generate higher sales. The investigators of the research found that more consumers purchased a jar of jam if the sampling table offered six varieties instead of 24. And there has been a healthy literature and many new experiments since that time exploring what has come to be known as “the paradox of choice.”
How Many Choices Are Too Many?
But there have been a number of studies and a good deal of retail research documenting the opposite as well. Indeed, it makes sense to think that offering more choice will maximize the chance of meeting individual consumers’ needs.
A recent article published in the Journal of Consumer Research suggests that idea of choice overload may be overblown. The authors analyzed results from fifty published and unpublished experiments on the topic. They concluded: (more…)
Reasons to Avoid Grid-Format Questions
Saturday, April 16th, 2011Among the many sources of potential error that can affect surveys are respondents themselves. They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.
This is not to say they are bad or fraudulent respondents. Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions. The problem with respondent error, it turns out, is poor survey design, which may involve biased or ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.
Recent research shows that grid-style questions that look like this:
or this: (more…)
What People Think of Surveys
Wednesday, March 30th, 2011One resource that we give far too little thought in market research are the people who offer their time and thoughts about the stuff we are researching. They are truly the lifeblood of market research.
An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys. The news is mixed. While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great. The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:
Declining Satisfaction with Surveys
In addition to the chart above, key statistics outlined in the article include: (more…)
Overcoming Your Math Curse
Friday, March 18th, 2011
Learning the math behind market research is not easy because there is no programmatic way to master it as a body of learning. It is not like algebra, geometry, calculus, or statistics in high school or college. It is complex and multifaceted and draws upon nearly every area of theoretical mathematics, but it must be continually adapted to the needs and practical problems of measuring and predicting customer behaviors and attitudes.
So it requires both (1) a rigorous foundation in mathematics and (2) years of experience to understand how it gets re-worked and applied to the real-life questions of market research. And even the rigorous foundation needs to be continually reinforced and expanded as the scope of our capabilities expands. Twenty years ago, who would have thought that Bayesian statistics and Monte Carlo simulations would become so central to our work? (more…)
The ABC’s of CBC: Understanding Conjoint for Market Research
Thursday, March 10th, 2011Our March 2011 newsletter focuses on conjoint research, which we consider to be one of the most clever and powerful techniques of survey research. Why? Because it allows us to build working models of decision-making.
Conjoint works by presenting people with scenarios that are more like the real-life trade-offs they always make. For example, instead of just asking a respondent about the importance of price, we ask them to make decisions about price, where price varies based on other attributes that are important to them. (more…)
Can a Focus Group Save Spider-Man?
Wednesday, February 16th, 2011
After spending $65 million and previewing the show over sixty times since the end of last year, the producers of the new Spider-Man musical in New York are turning to focus groups and surveys in hopes that market research can do something…anything…to save the amazing Spider-Man from destruction.
Last week, theater critics roundly panned the show, calling it among the worst Broadway shows ever. The New York Times described is as “so grievously broken in every respect that it is beyond repair.” But days after these negative reviews, a market research firm was brought in, soliciting volunteers to help fix the show: (more…)
The Magic Numbers . . . . Reappear!
Thursday, February 10th, 2011Last quarter we wrote about Magic Numbers in Market Research—those arbitrary rules of thumb and cut-off points we use when quoting things like minimum samples sizes or how many people to include in a focus group. Presto! Like magic, the issue appeared in the New York Times a few weeks ago, this time related to a dispute about the best way to statistically test for the existence of ESP.
The backstory: A respected academic journal in social psychology published an article showing data that suggests ESP exists. Horrified, some researchers argued that psychologists were using old-fashioned inferential statistics when they should be using modern-day Bayesian statistics. Here is a link to the article, if you’re interested. Unfortunately, it does a lousy job explaining what Bayesian statistics is.
But fortunately, in response, the editor in chief of The Annals of Applied Statistics submitted a letter to the New York Times clarifying that all statistics ends up relying on arbitrary magic numbers: (more…)