In last year’s cliffhanger episode of “Can a Focus Group Save Spider-Man?” we pondered whether market research was powerful enough to save a Broadway show from doom and destruction.  After crushing reviews from theater critics, the producers hired a market research firm to help them rewrite the show.

Guess what?  It worked.  Since the show re-opened in June, it is regularly among the top five earners among Broadway shows, and The Hollywood Reporter reports that in December it “set a new record for a single-week box office gross, raking in $2,941,794 for the week ending January 1.”

In a year when Steve Jobs’ quip that “it’s not the consumers’ job to know what they want” has been used to bludgeon market research, we see research doing a pretty good job identifying buyers’ needs and helping decision makers address those needs in smarter ways.

It is true that consumers can’t answer questions like, “What show do want us to produce next?” or “What great technological innovation would be most useful to you?”  But they certainly can tell you want they want, and the art of research is to ask just the right questions to give you enough insight about what to do next.  Here are three things you can ask about, and that audiences, customers, and prospects will happily tell you:

1.  What is important to them and what they care about.  Insights about what matters to buyers will help you design your product or service and will provide deep insight about how to market it.

2.  What their frustrations are and what is not working.  Most consumers are eager to critique products and services that fail to meet their needs, which highlights the opportunities for new solutions.

3.  How good or bad your idea is.  There are lots of ways to test concepts, products, and ideas, and survey research is remarkably good at predicting winners versus losers.

The lesson from Spider-Man is that it does not take superhuman powers to fix a flawed strategy nor off-the-charts creative genius to make a good product sell.  A thoughtful effort to ask questions and to listen to your customers is sometimes all it takes to turn harrowing encounters with goblins into success.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/focus-groups-save-spider-man/feed/ 0 http://www.verstaresearch.com/blog/can-a-focus-group-save-spider-man/ http://www.verstaresearch.com/blog/can-a-focus-group-save-spider-man/#comments Wed, 16 Feb 2011 14:09:52 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1140 After spending $65 million and previewing the show over sixty times since the end of last year, the producers of the new Spider-Man musical in New York are turning to focus groups and surveys in hopes that market research can do something…anything…to save the amazing Spider-Man from destruction.

Last week, theater critics roundly panned the show, calling it among the worst Broadway shows ever. The New York Times described is as “so grievously broken in every respect that it is beyond repair.”  But days after these negative reviews, a market research firm was brought in, soliciting volunteers to help fix the show:

If you love Broadway shows, we would like to invite you to be a test audience member and participate in either Act 1 (first part of show) or Act 2 (second part of show) of the all new Broadway musical “Spider-Man: Turn Off the Dark.”

We will be asking you to fill out a survey after the performance, plus stay for a 15 minute discussion in the Spider-Man V.I.P. room. In turn for your feedback and participation you will receive a Spider-Man goodie bag worth over $60.00, on us.

Can market research really rescue the show?  Possibly.  High quality research, both qualitative (such as focus groups) and quantitative (such as surveys) can provide deep insights into how customers and audiences think, perceive, react, and respond to products, services, and opportunities.

If market research can help rescue the show, here are two suggestions we offer that will be critical to keep in mind:

1.  Keep the focus groups focused. Too often clients or managers want their customers to tell them what would make them love their product.  That’s not something customers can reliably tell you.  They can tell you what they care about, what products they purchase and why, what needs and problems they are trying to solve, and what frustrations they face.  Our advice generally: Keep the focus group participants focused on the things that matter to them, then connect the insights gained to the products being offered.

2.  Ask only about things that can be fixed. For all the talk about “actionable” research, there is a great deal of research that offers supposedly actionable ideas that clients or managers simply cannot and will not act upon.  How does the audience feel about Bono’s music for the show?  Unless management is prepared to dump Bono or have him re-write (again), don’t ask.  The key is to determine during the research design which specific decisions can and cannot be made, and then to focus the research on collecting data that lends specific insight to those decisions.

Asking market research to help rescue a $65 million investment at the last minute seems like a tall order, but fortunately great research does not take super-human powers.  At Versta Research we rely on brains, experience, thoughtfulness, and a commitment to working closely with clients to ensure that we deliver insightful stories that are understood and acted upon.

So what will become of Spider-Man?  Will he die under the crushing weight of debt and harsh reviews?  Or will the critics groan in agony as Americans happily pay for the worst show on earth?  Stay tuned for the next exciting episode of “Can a Focus Group Save Spider-Man?”

—Joe Hopper, Ph.D.

The backstory:  A respected academic journal in social psychology published an article showing data that suggests ESP exists.  Horrified, some researchers argued that psychologists were using old-fashioned inferential statistics when they should be using modern-day Bayesian statistics.  Here is a link to the article, if you’re interested.  Unfortunately, it does a lousy job explaining what Bayesian statistics is.

But fortunately, in response, the editor in chief of The Annals of Applied Statistics submitted a letter to the New York Times clarifying that all statistics ends up relying on arbitrary magic numbers:

The heart of the dispute is not about Bayesian versus classical statistical methods; if anything, it is an argument against knee-jerk use of the famous .05 criterion, which generally finds the results of an experiment acceptable if the chances are no greater than 5 percent (that is, 0.05) that they could have occurred randomly.

Physicists, for example, don’t trust .05 and prefer much tougher evidential levels. A claimed result that overturns all ideas of causality might well require something stricter than .05. A Bayesian would have to make the same kind of difficult choice as to what “prior probability” to assign to the existence of ESP.

No general formula will free the scientist, or anyone else, from having to use judgment in interpreting evidence. But general formulas, including .05, are valuable in imposing some order on the Wild West world of claimed results.

Now there’s a statistician after our own heart.  We, including market researchers, all use magic numbers, because in a strange way, magic numbers work.  They help us impose order and make sense of the messy reality we are trying to understand.  But ultimately there is nothing magic about them, and there is no magic solution that can substitute for seasoned judgment and expertise.

That’s why you come to us, right?  It’s easy to push a button that tells whether data is statistically significant.  It’s not so easy to discern the story behind that data, whether that “statistical significance” really matters, and what you should do with it.  That’s the magic of the work we do at Versta Research.

—Joe Hopper, Ph.D.

Editing is key, however.  Who wants to read something in the halting, choppy, in-eloquent speech that most of us use in talking?  A killer quote is one that is short, direct, pithy, and on point.  Achieving this requires a journalistic standard for presenting quotes.  What does that mean?  Here are some steps:

• Transform the long run-on streams of words into sentences
• Remove the um’s, ah’s, like’s, you-know’s and other verbal fillers
• Cut out false starts and small tangents that do not contribute to intent of the sentence
• Do not change the words they use, or the order in which they use them

The goal is to focus on the intent and meaning of what they were saying.  Then clean it up so that it says exactly what they meant, and really did say.  As a check, imagine going back to your respondent a few minutes after they spoke and asking “Here’s what I’ve written down as what you said.  Is this right?”  If they would likely agree, then you’ve got it.

Here is an example from a project we completed this week:

Before: From what I’ve, you know, I’ve understood, that with Acme it’s very easy and very easy to navigate. That’s what the differences are between the companies is how easy it is to get in there and actually find what you’re looking for, and from what I’m . . . my feedback I’ve gotten about Acme, is that it is very easy and people feel comfortable getting in there and going through the process online.

After:  From what I’ve understood, Acme is very easy to navigate.  That’s the difference between companies: How easy it is to get in there and actually find what you are looking for.  That’s the feedback I’ve gotten about Acme.

If you are curious about whether our “journalistic standard” is appropriate, here’s a fascinating radio story that ran several years ago, which NPR re-ran this week in memory of John Solomon, the reporter who wrote it.

Radio interviews, it turns out, are similarly edited.  And according to NPR, they have never had a person interviewed on their show complain about how they were edited.

Need help grabbing your audience and telling a compelling research story?  Give us a call.  We focus on the rigors of research, and then turning data into stories so that your research gets heard and used, and so that it really makes a difference.

—Joe Hopper, Ph.D.

An article in the October 2010 issue of the Journal of Marketing Research offers research to this point.  The authors identify a unique profile of consumers who are measurably better at developing, testing, and reacting to new product ideas.  The personal qualities these consumers possess include (quoting from the article):

• Openness to new experiences and ideas
• An intellective self-focus, or “reflection”
• The ability to apply both experiential and rational processing styles
• The ability to process information both verbally and visually
• High levels of creativity
• Optimism

The authors developed and validated an eight-question scale to measure these qualities, and showed that consumers who score high on the scale are able to give better ideas and feedback compared to those who score low.  The authors provide experimental data showing that products developed with input from this unique group of consumers are more likely to succeed in the marketplace.

So, you could ask a random sample of consumers to participate in your focus groups and surveys as you test and refine new concepts.  But if there is a non-random subset of them who can give you data that is more predictive of your new product’s success, why would you?  Forget about methodological purity, probability samples, and randomness.  You need a purposive sample focused on just this subset of consumers.

As always, good research requires thoughtful attention to strategies, outcomes, and how the research will be used.  Adhering to rigid protocols does not necessarily mean rigorous research.  Thinking about samples in smart and strategic ways is a great example of that.  If you are engaged in new product and innovation research,  give Versta Research a call and we would be happy to think with you about the best approach.  We would also be delighted to share with you (and execute for you) the eight-item scale used to identify the best consumers to test your ideas.

—Joe Hopper, Ph.D.

A client had a surprising experience this week when a member of our multivariate analysis team showed up online to watch a live in-depth interview with a registered nurse about how prescribing decisions are made.  “Who is that online with us?” the end-client inquired, not recognizing the name.  The qualitative manager answered, “He’s on our multivariate team.”

It must have seemed strange to have a statistician taking a keen interest in the qualitative work.  Strange, because too often the qualitative and quantitative sides of research do not inform each other in the rich ways they can and should.

On the very same day, we read this interview with Professor Jordan Louviere in the American Marketing Association’s Marketing News magazine:

The field [of product choice research] is really in desperate need of theory and a lot of what goes on in choice modeling is nothing more than statistics.  And while there . . . is theory in statistics, it’s not the theory we need.  It’s important, but statistics is just a tool.  We need theory that tells us and informs us much better than in the past how consumers actually do what they do and what are the best ways to approximate that?

So we have, sort of, a mismatch in the field right now with people. . . who are trying to understand how consumers make decisions and choices and other things, and we have choice modelers, who, by and large . . . are statisticians and the two groups don’t speak to each other.

We are proud that at Versta Research the statisticians doing the conjoint choice modeling participate in, and often lead, the qualitative work.  It ensures that the choice models are truly built to reflect and understand the processes at work.  Our models are more than rote applications of attributes and levels, analyzed with fancy tools that do Hierarchical Bayes (HB) estimation.  They are models of the decision processes our clients care about, designed by quantitative experts who know their stats, but who are immersing themselves in the real world.  The end result?  Our clients get deeper insight than is typical, and they get better answers to their most pressing questions.

—Joe Hopper, Ph.D.

Here are some great quotes we found to keep in mind as guiding principles:

A prudent question is one half of wisdom—Francis Bacon, 17^th century philosopher and scientist

If you do not ask the right questions, you do not get the right answers—Edward Hodnett, 20^th century poet and writer

The uncreative mind can spot wrong answers, but it takes a very creative mind to spot wrong questions—Sir Antony Jay, contemporary writer

Successful people ask better questions, and as a result, they get better answers—Anthony Robbins, contemporary self-help author and motivational speaker

As noted in the newsletter, one of the best ways to truly add value to the research that you do is to listen carefully to your internal clients and formulate the right questions.  Research is all about answering questions, which is, of course, all about asking questions.  Building fancy statistical models is fun and brainy and definitely worth bragging about at your next party, but few people in your organization care much about models.  They care about what answers those statistics provide to the fundamental business questions that keep them up at night.

Any questions? Give us a call. Struggling to find your questions? Take a look at our March 2010 Newsletter.  We are happy to help you find them.

—Joe Hopper, Ph.D.

You need to find out the who, what, where, when, and why they hate you.  Probe deeply with focus groups, in-depth interviews, social media monitoring, and survey research.  Rigorous research done right (our specialty) will give you valuable insight.  And if it is communicated expertly to management (our specialty) you can probably fix the problems.  Then, the next time you listen to your customers, you won’t need to duck and cover.

-Joe Hopper, Ph.D.

1. Data visualization and graphics will become more important.  The technology for amazing visualization tools is almost there, along with a growing realization that presenting data needs to be far more intuitive.
2. The value-add of insight and interpretation will become essential, not optional.  With new technologies, any monkey can now collect data, and half those monkeys can dump it into a tool that makes a chart.  But what does it all mean?  What’s the story?
3. Focus groups will become a thing of the past, in favor of qualitative ethnography and MROCs (Market Research Online Communities).  Usually focus groups are not the ideal way to answer the questions that clients have, but clients value hearing and seeing their customers.  New technologies have made better methods more accessible.
4. Data integration from multiple sources and areas within an organization will become a new imperative.  There are lots of data in lots of places.  One huge challenge over the next decade will be to bring multiple (and multiplying) sources of data together, connect the dots, and create intelligence.
5. Intellectual expertise to sort through, integrate, and interpret the explosion of data will become key.

Versta Research will be staying ahead of these trends over the next ten years to ensure that you get comprehensive, focused, and deep knowledge of your markets.  We offer help from high level experts who have rigorous training and who have the ability transform your data into stories that you can use.

-Joe Hopper, Ph.D.

The reason?  Many clients have been using focus groups not for deep understanding of consumer needs and preferences, or to generate creative group dynamics among participants.  They were using them to get reactions and “votes” on package designs or product concepts.  Do you like the blue package, or the red package, and why?  Current technologies make it much easier and faster to answer simple questions like this through online surveys.  Since there is not a lot of research design or analysis involved, clients can do this work themselves.

There will always be a role for the expertise that a good focus group moderator or analyst offers, but helping clients pick between the red box and the blue box is not it.  When technologies make our work easier, faster, and cheaper, it makes sense that we bring key elements of this work back in house, while we work with partners and experts to tackle the tougher research challenges we face.

–Joe Hopper, Ph.D.