Suppose you want to measure the importance of 12 attributes for a new product or service.  If you know ahead of time that consumers are going to say that all 12 are extremely important to them, then MaxDiff is an excellent method for differentiating among the attributes so you can focus on the top two or three that matter most.

But what if you don’t know that all 12 attributes are extremely important?  Maybe none of them are.  Maybe they run the gamut from unimportant to extremely important.  The problem with MaxDiff is that it only tells you the importance of attributes relative to each other, but it won’t tell you whether the attributes are important.  The MaxDiff model will assign ratio-level numbers so that you can rank and quantify the importance of each attribute vis-à-vis the others.  But it will not anchor the attributes in a meaningful way.

This week we are designing a study in which we want to differentiate among attributes, but we also want to measure the gap between satisfaction and importance for items that are truly important to our target market.  We cannot do that with data from a typical MaxDiff study.  So we are using an old fashioned importance rating scale instead.

As always, it is critical to think about the story you want to tell with your research data, and then work backwards to the design and the choice of methods.  In many cases MaxDiff is the perfect tool.  In other cases it will leave you with data that is difficult to apply to critical questions.

Feel free to give us a call if you need some help deciding among the best methods for your research, whether it be MaxDiff, other conjoint techniques, or something else entirely.  We’ll help you focus on the story you need to tell and on the research design you need to tell it.

–Joe Hopper, Ph.D.

One way to analyze the data is to use a simple counting technique.  It works pretty well, by the way.  But the more common way these days is to use advanced statistical modeling that allows for stronger individual-level estimates of importance scores.  This affords better opportunities for segmenting the data and analyzing subgroups.

We typically see analysts transforming the importance scores (or utilities) to a 0—100 scale.  It is a compelling scale because all importance scores add up to 100, which mimics the technique of having respondents allocate 100 importance points to each of the features they care about.

But if you do a lot of MaxDiff studies and frequently present them to your management team, you lose the ability to provide a rule of thumb as to what counts as a “high” score vs. a “low” score.  If one MaxDiff exercise uses fifteen attributes, an importance score of 20 is quite high; if another MaxDiff exercise uses just five attributes, and importance score of 20 is just average.

The solution?  Instead, we typically transform scores to a 0—100*N scale, where N=the number of attributes tested.  If you tested fifteen attributes, transform to a 0—1500 scale.  If you tested five attributes, transform to a 0—500 scale.  With this method, a score of 100 is always the average, scores above 100 are always above average, and scores below 100 are always below average.  Sometimes we transform this into a chart showing “percent above/below  average” as shown in the third example below, but this is not always necessary because anchoring the average at 100 makes it simple to calculate those percentages mentally.

Example 1: Versta's Recommended Scaling of MaxDiff Scores

Example 2: The Typical Way of Scaling MaxDiff Scores

Example 3: Another (Sometimes Useful) Way to Scale MaxDiff Scores

Need help with the smartest ways to design, implement, and report MaxDiff studies?  Call us at (312) 348-6089 for assistance and further information.

–Joe Hopper, Ph.D.

Monty Hall in Let's Make A Deal

The hardest part of quantitative market research is not that it involves numbers, math, or even statistics, but that it involves complex problems in probability.

Over the past several years, psychologists have been documenting how difficult it is for us humans to solve even “simple” probability problems.  One fascinating example is a puzzle known as the Monty Hall dilemma based on the 1960’s game show Let’s Make A Deal.  Monty would offer his contestants three doors to choose from, one of which had a valuable prize behind it.  After the contestant chose, Monty would open one of the other two doors, deliberately choosing one that had no prize behind it.  Then he offered the contestant an option of staying with the original choice, or switching to the other unopened door.  Which should the contestant do?

The contestant should always switch.  The odds of winning are two-thirds if she switches, and one-third if she stays.  Most contestants, however, stay with their original choice, believing that the odds of winning are the same whether they stay or switch.  And it turns out that pigeons do a better job solving this puzzle than humans.  In an article published last year in the Journal of Comparative Psychology, researchers showed that if a similar game is played with pigeons, they start to catch on and consistently choose to switch, which maximizes their winnings.  Humans, however, do not.  Not only do we have a hard time grasping the true probabilities conceptually, but even if we play the game over and over, we ignore our experience and learning.

What does this have to do with market research?  Well, behind all the numbers, charts, and percentages that we present to our clients, most of our methods and analyses are based on probabilistic reasoning.  We calculate the probabilities that our sample statistics represent true population values.  We build conjoint or MaxDiff models based on probabilities of certain responses occurring even if we did not measure them directly.  We ask respondents to assess the probabilities of their own behavior (“How likely are you to buy?”) and use those to calculate estimates of market potential. We are dealing with layers upon layers of probabilities.

It is no wonder that market research reports can be so impenetrable and difficult to untangle.  Behind nearly every chart or table is a probability puzzle, and for most of us there is certainly nothing intuitive about probabilities.

And it is no wonder that many research firms do not even try to go beyond giving you charts, data, and tabulations.  But that’s where a firm like Versta Research comes in.  We solve two of the most difficult challenges facing research professionals:  (1) grasping the complex nature of probabilistic reasoning, which may befuddle even the most accomplished mathematicians, and (2) turning mounds (or crumbs) of data and probabilistic reasoning into an effective and compelling story that you can use and that your clients can understand.

When you need help with either or both of these challenges, call us at (312) 348-6089.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/pigeons-beat-people-on-probability-problems/feed/ 0 http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/ http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/#comments Thu, 07 Jul 2011 18:59:16 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1484 Versta Research just hit a magic number: 100.  That’s the number of articles we have written to help our clients and their colleagues keep abreast of important trends in market research.  If your market research supplier is not providing ongoing thought leadership in design, methods, and analytics, then what are the chances they are bringing ongoing and deep insight to your specific research needs?

To celebrate, we’re serving up a sampler of our five best articles.  How did we decide they are the best?  Our clients told us.  These are the articles that they write to us about, forward to their colleagues, and for which they return to our website time and again.  These are also the articles for which we get requests for print-ready PDF versions.  (Just let us know if you want one!)

1. Turning Data into Stories

There are two critical elements to top notch research.  First, it has to be right, which means focusing on the rigors of research design, data collection, and statistical analysis.  Second, it has to be heard, understood, and used, and in our view that means turning data into stories.  In this article we focus on what it means to turn data into stories, and we outline what you gain by doing so.

2. The Art of Asking Questions

Somewhere along the way to research becoming central to how businesses learn about their customers, the art of asking questions was lost. As a result, there is a lot of research for research’s sake, data in search of answers, and findings in search of questions.  The first thing you should do before starting research is figure out what question your research must answer.  Here’s how.

3. Magic Numbers in Market Research

Researchers cling to a handful of “magic numbers” that guide the decisions they make.  There are magic numbers for sample size, the optimal number of points on a scale, thresholds for statistical significance, and how big a focus group should be.  This article demonstrates and explains.

4. The ABCs of CBC: Understanding Conjoint for Market Research

This article focuses on the basic ideas, advantages, and uses of conjoint research. What is conjoint? How and why is it used? What insights can it give you? Furthermore, what are some of the pros and cons of fielding research using a conjoint method vs. other methods you might use?

5. Game Changing Product Innovation

A lot of research supporting new product development is a like machine that ends up creating NON-innovation because of over-benchmarking.  We suggest an alternative.

We write these articles because even though we are in the business of doing rigorous research for our clients, research only matters if it is thoughtfully communicated, understood, and used.  We hope that our efforts help your organization better design and deploy research to make smarter business decisions.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/feed/ 0 http://www.verstaresearch.com/blog/the-abcs-of-cbc-understanding-conjoint-for-market-research/ http://www.verstaresearch.com/blog/the-abcs-of-cbc-understanding-conjoint-for-market-research/#comments Thu, 10 Mar 2011 20:54:06 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1184 Our March 2011 newsletter focuses on conjoint research, which we consider to be one of the most clever and powerful techniques of survey research.  Why?  Because it allows us to build working models of decision-making.

Conjoint works by presenting people with scenarios that are more like the real-life trade-offs they always make.  For example, instead of just asking a respondent about the importance of price, we ask them to make decisions about price, where price varies based on other attributes that are important to them.

There are several types of conjoint research, including traditional full profile conjoint, partial profile conjoint, adaptive conjoint, and choice-based conjoint. There are also choice-based techniques similar to conjoint, such as MaxDiff analysis.  Plus, there are different modes of analysis, including regression and HB (Hierarchical Bayes) estimation.  To make the right choice of method, you need to consider sample size, the need for individual-level vs. aggregate analysis, how many factors must be included in the model, and whether pricing is central to the research.

Yikes, that’s a lot.  So where do you start?  Start here:  The ABC’s of CBC: Understanding Conjoint for Market Research.  It provides a fundamental understanding of what conjoint is, how it works, and the kinds of questions it can answer.

After that, we would be pleased to help you consider your options, including the option of working with us or going it alone.  In fact, here a single conjoint question (no fancy modeling needed!) to help you decide whether we might be of value:

If you find yourself on the right side of the scale, feel free to give us a call.

—Joe Hopper, Ph.D.

The technique we used for that study is called Adaptive Conjoint.  If you want to know how people in your target audience make decisions—how they weigh the pros and cons of your product or service versus others— adaptive conjoint can be a powerful technique that provides robust and insightful data at the same time it really engages the participants.

All conjoint techniques work by presenting people with scenarios that are more like the real-life trade-offs they always make.  For example, instead of just having them rate how important price is, they are asked to make decisions based on a series of scenarios, one piece of which is price.  Based on their answers, the importance of price in their decision-making can be mathematically derived.

The twist with adaptive conjoint versus other types of conjoint is that each scenario learns from the previous answers, so that the decisions confronting respondents get more real and more difficult with each answer they give.  It requires online or computer-based administration (versus paper-based or telephone surveys) because an adaptive algorithm drives the learning based on previous answers.  If it sees that price is not influencing decisions, for example, then the survey stops including price in the scenarios so that the respondent can focus on things that do matter.

Many respondents say they love these surveys because the tasks mimic real-life decisions.  Moreover, as respondents start to realize that the survey is learning and changing based on what matters to them, they become more and more engaged.  And even respondents who do not love the process appreciate how effectively it drill downs to the criteria that really matter to them:

“Interesting to take this survey because it helped me realize what my priorities are as a shopper.”

“My head was hurting near the end, but I expect this approach resulted in more accurate info from me (checks and balances).”

One of the cool things about conjoint analysis is that it builds elegant mathematical models of human decision making, and it works.  The super cool thing about adaptive conjoint is that it interacts with the respondents and brings them into the modeling process itself.

If you would like to see the process at work, or think adaptive conjoint might be right for your research, we would be happy to help you think about an optimal approach.

—Joe Hopper, Ph.D.

A client had a surprising experience this week when a member of our multivariate analysis team showed up online to watch a live in-depth interview with a registered nurse about how prescribing decisions are made.  “Who is that online with us?” the end-client inquired, not recognizing the name.  The qualitative manager answered, “He’s on our multivariate team.”

It must have seemed strange to have a statistician taking a keen interest in the qualitative work.  Strange, because too often the qualitative and quantitative sides of research do not inform each other in the rich ways they can and should.

On the very same day, we read this interview with Professor Jordan Louviere in the American Marketing Association’s Marketing News magazine:

The field [of product choice research] is really in desperate need of theory and a lot of what goes on in choice modeling is nothing more than statistics.  And while there . . . is theory in statistics, it’s not the theory we need.  It’s important, but statistics is just a tool.  We need theory that tells us and informs us much better than in the past how consumers actually do what they do and what are the best ways to approximate that?

So we have, sort of, a mismatch in the field right now with people. . . who are trying to understand how consumers make decisions and choices and other things, and we have choice modelers, who, by and large . . . are statisticians and the two groups don’t speak to each other.

We are proud that at Versta Research the statisticians doing the conjoint choice modeling participate in, and often lead, the qualitative work.  It ensures that the choice models are truly built to reflect and understand the processes at work.  Our models are more than rote applications of attributes and levels, analyzed with fancy tools that do Hierarchical Bayes (HB) estimation.  They are models of the decision processes our clients care about, designed by quantitative experts who know their stats, but who are immersing themselves in the real world.  The end result?  Our clients get deeper insight than is typical, and they get better answers to their most pressing questions.

—Joe Hopper, Ph.D.

Conjoint analysis is one of the techniques we use that comes closest to pure mathematical models of human behavior, and it can lend amazing insight.  With conjoint techniques, we create working models of consumer preferences for different types of products.  It works by presenting people with scenarios that are more like the real-life trade-offs they always make.  For example, instead of just knowing that price is important, the data from a conjoint study lets us assign a value to price relative to other important attributes, and then model how preferences change as we modify each attribute.

Conjoint is a powerful tool because it helps us understand not only which configuration of product features is optimal, but also why. It shows us how each component is being valued to drive overall preference.

Consider using conjoint if you want to understand:

• The tradeoffs that people make when evaluating their options
• How to bundle features to optimize your product or offering
• Needs-based segments among the people you are targeting
• Drivers of behavior that people themselves may be unaware of
• How your new product may affect product choices and market share

While conjoint analysis relies on a complicated set of mathematical algorithms, its beauty is that it ultimately tells a compelling story about humans, how they behave, and why they behave.

–Joe Hopper, Ph.D.