Earlier this month Arthur C. Nielsen, Jr. died.  He left behind a giant and reputable market research company and a brand name recognized throughout the world.  The A.C. Nielsen company was started by his father and in its early years tracked the sales of goods through grocery and drug stores.  The company then moved into media tracking and became the authoritative source for measuring audience size and demographics.  Nearly every company with an advertising budget continues to rely on Nielsen data to determine where to advertise and how much to spend.

Nielsen’s legacy is that he demonstrated the value of collecting and tracking data, and lots of it.  Every item we purchase is now logged, counted, and tracked.  Every television and radio show is tracked for how many viewers it has and in what markets they live.  And of course everything we do on the Internet is recorded and tracked.  Even our bodily locations are tracked via GPS or cell phone signals.  Most market research firms today generate the bulk of their revenue simply by collecting, tracking, tabulating, and reporting data.

This important legacy has left us with tons of data, growing at an exponential rate,  and a monumental challenge of how to synthesize it and move beyond mere tabulation and reporting.  The question is, how do we meet that challenge and take Nielsen’s legacy to the next frontier?  In our view, it will involve two key efforts:

1.  Understanding data in much deeper ways and analyzing it with data mining tools, new algorithms, and new approaches that go beyond traditional statistics, including Bayesian analysis, neural networks, and machine-learning techniques.

2. Interpreting and communicating data in ways that are more practical, relevant, meaningful, and useful.  In other words, turning data into stories that real people, real managers, and real businesses understand and can use.

To be sure, much of the research industry is still (appropriately) focused on implementing technologies to better manage, tabulate, and report volumes of data.  But automated tables and charts with ever-expanding levels of detail are reaching their limits of utility.  Versta Research is proud to be on the next frontier, where better interpretation and understanding of data is key.

–Joe Hopper, Ph.D.

We were reminded of this upon hearing prominent news reports a few days ago that the average household income in the U.S. has fallen by about 10% in the past decade, most of it happening since the start of the recession four years ago.  But does that mean most Americans’ incomes are falling?  No.  Though it is hard not to think so given how the data are being presented and reported.

The problem is that statistical averages such as means and medians do not tell us what is happening to specific people or groups of people.  They are abstract properties of the whole, but they do not directly describe the parts making up the whole.

Unemployment has grown from 4% to 9%.  As more people lose their income, the average is pulled down even if a large majority remain employed and see no decrease in their income.  There are other factors that pull down the average as well.  Some people who lose their jobs are taking new jobs at lower wages.  Some people have wage or salary increases, but the increases are lower than the rate of inflation.

So overall (at the aggregate level) it is true that income has fallen.  But for most parts of the whole (at the level of individuals), it is probably not true.  We should have been told not only averages, but also the percentage of households that have seen a decline in their individual incomes over the last decade.  Almost certainly it would paint a different picture.

When we at Versta Research interpret and report market research and polling data, we use percentages far more often than averages for exactly this reason.  We are focused on the story that the data tell, which is usually about people not about abstract wholes.  It requires careful attention not only to the stats and the data, but also to the interpretation and communication of those stats so that your audience has a realistic picture of what they mean.

–Joe Hopper, Ph.D.

Monty Hall in Let's Make A Deal

The hardest part of quantitative market research is not that it involves numbers, math, or even statistics, but that it involves complex problems in probability.

Over the past several years, psychologists have been documenting how difficult it is for us humans to solve even “simple” probability problems.  One fascinating example is a puzzle known as the Monty Hall dilemma based on the 1960’s game show Let’s Make A Deal.  Monty would offer his contestants three doors to choose from, one of which had a valuable prize behind it.  After the contestant chose, Monty would open one of the other two doors, deliberately choosing one that had no prize behind it.  Then he offered the contestant an option of staying with the original choice, or switching to the other unopened door.  Which should the contestant do?

The contestant should always switch.  The odds of winning are two-thirds if she switches, and one-third if she stays.  Most contestants, however, stay with their original choice, believing that the odds of winning are the same whether they stay or switch.  And it turns out that pigeons do a better job solving this puzzle than humans.  In an article published last year in the Journal of Comparative Psychology, researchers showed that if a similar game is played with pigeons, they start to catch on and consistently choose to switch, which maximizes their winnings.  Humans, however, do not.  Not only do we have a hard time grasping the true probabilities conceptually, but even if we play the game over and over, we ignore our experience and learning.

What does this have to do with market research?  Well, behind all the numbers, charts, and percentages that we present to our clients, most of our methods and analyses are based on probabilistic reasoning.  We calculate the probabilities that our sample statistics represent true population values.  We build conjoint or MaxDiff models based on probabilities of certain responses occurring even if we did not measure them directly.  We ask respondents to assess the probabilities of their own behavior (“How likely are you to buy?”) and use those to calculate estimates of market potential. We are dealing with layers upon layers of probabilities.

It is no wonder that market research reports can be so impenetrable and difficult to untangle.  Behind nearly every chart or table is a probability puzzle, and for most of us there is certainly nothing intuitive about probabilities.

And it is no wonder that many research firms do not even try to go beyond giving you charts, data, and tabulations.  But that’s where a firm like Versta Research comes in.  We solve two of the most difficult challenges facing research professionals:  (1) grasping the complex nature of probabilistic reasoning, which may befuddle even the most accomplished mathematicians, and (2) turning mounds (or crumbs) of data and probabilistic reasoning into an effective and compelling story that you can use and that your clients can understand.

When you need help with either or both of these challenges, call us at (312) 348-6089.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/pigeons-beat-people-on-probability-problems/feed/ 0 http://www.verstaresearch.com/blog/the-pitfalls-of-auto-coding-text-responses/ http://www.verstaresearch.com/blog/the-pitfalls-of-auto-coding-text-responses/#comments Thu, 28 Jul 2011 13:56:27 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1523 An issue we continually struggle with at Versta Research is how to automate the research process and leverage new technologies without losing the essence of what good research does.  Good research does not report data, build charts, or generate dashboards. It learns, answers new questions, interprets data, and helps users focus on information and findings that are relevant to their needs.

The last couple of weeks we have been working with a group that specializes in coding and tabulating text responses to open-ended questions on surveys.  They have tools and technology that undoubtedly make the process easier and more efficient (we have used those tools, and they are impressive).  They are also have a singular focus and expertise that is supposed to help streamline the process, cut costs, and improve speed and efficiency.

The results have been mediocre at best, even with human coders working the technology and making the critical decisions. They efficiently and accurately coded each response into one or more  buckets.  But they created buckets that give hardly any insight into what the client wants to know.  “What do you love most about this product?”  The coders accurately identified all consumers who mentioned the physical size of the product.  But they lost critical distinctions not only about big versus small, but also about size being a constraint (not enough room for a larger product) versus a preference for how consumers wanted the product to look (an aesthetic choice).  They got the topic right, but did not answer the question in a meaningful way.  So what good was all that coding?

I asked a colleague (highly paid, less efficient) to fix and re-code the data, and I asked her to think not in terms of topics but in terms of answers to questions.  She did, and remarked, “You have an advantage, because you know more about this product and what is relevant to the research than the people (and machines) who did the coding.”

Yes, that’s the point, and that’s the struggle for technology and automation.  Smart researchers who know the right questions and continually think about how best to answer them with data will always have an advantage.  Even when pitted against the best and most efficient technologies, we will always win the insight contest.

There are lots of places where technology is helping us do our work faster, smarter, and at a lower cost.  But no matter what the innovators in tools and technology tell you, it makes a huge difference to have smart people with expertise slogging through the data, deciding how to analyze and present it, and transforming that into a story you can use.  We, at Versta Research, consistently and substantially outperform machines and outsourced labor, which means that you, the client, win as well.

—Joe Hopper, Ph.D.

If your doctor tells you that highly reliable studies have shown that taking a certain pill will cut your risk of getting a serious disease in half, would you take it?

Suppose he adds that the risk is 2 percent for people who do not take the pill, but your risk will be reduced to 1 percent if you do. Would you still take it? And what would you do if he told you that only one of every 100 patients who take the drug will actually benefit from it?

The doctor could have said any of these things, all truthfully, because they are just different ways of describing the same data.

The researchers showed that the data’s persuasiveness and understandability, and the subjects’ views of efficacy varied dramatically for each of these three ways of presenting the data.  Moreover, education or expertise among those being presented with the data made no difference, with both physicians and patients responding in the same ways.

Data have no meaning without an implicit or explicit story to communicate that data.  And indeed the story tells the audience what to do with that data and how to interpret it.  That is what is happening in the research described above, and that is why good research is far more than collecting data, generating tabs, producing charts, and writing phrases that presumably summarize findings.  Good research must turn data into stories, so that your clients and audiences understand the questions, see and grasp the answers, and then act upon the research in smart and effective ways.

Need help?  Turning data into stories is central to our approach for the most complex, esoteric, or even the most mundane studies.  Versta Research can help you get your research heard, understood, and used.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/feed/ 0 http://www.verstaresearch.com/blog/entrepreneurial-advice-rethink-your-research/ http://www.verstaresearch.com/blog/entrepreneurial-advice-rethink-your-research/#comments Thu, 16 Jun 2011 14:42:20 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1437 Executives who lead entrepreneurial firms have dramatically different attitudes about market research from their counterparts at larger established firms, according to a recent study from Saras Sarasvathy, an associate professor of business administration at the University of Virginia.

The study suggests that entrepreneurs are more focused on immediate and practical questions that will help them get their products into the hands of customers, and that traditional market research may not be the best way to get the right data and answers.  That makes sense.

But according to an article in the February issue of Inc. magazine, “when asked what kind of market research they would conduct for [a] hypothetical start-up, most of Sarasvathy’s subjects responded with variations on the following:

“OK, I need to know which of their various groups of students, trainees, and individuals would be most interested so I can target the audience a little bit more. What other information… I’ve never done consumer marketing, so I don’t really know. I think probably…I think mostly I’d just try to…I would…I wouldn’t do all this, actually. I’d just go sell it. I don’t believe in market research. Somebody once told me the only thing you need is a customer. Instead of asking all the questions, I’d try and make some sales. I’d learn a lot, you know: which people, what were the obstacles, what were the questions, which prices work better. Even before I started production. So my market research would actually be hands-on actual selling.”

We heartily agree that sometimes you should get out there and sell rather than conducting more market research (see our article What You May Need Is Marketing, Not Market Research).  But the problem with taking this quote at face value is that a really good market researcher would never say “How would you use market research?”  She would say, “What do you need to know?   What answers to questions would help you achieve your most critical business objectives?” (See our article The Art of Asking Questions.)  Then she would decide whether and how market research can best be used.

At Versta Research we believe that no matter what size company you are, you should be thinking about research more like an entrepreneur.  Great entrepreneurs are using, gathering, analyzing, and interpreting data all the time to help them make decisions.  But their data might be coming from reports on sales calls rather than standard satisfaction or tracking surveys or another new product benchmarking study.  That is a good thing, and a smarter way to approach research even if you are an established firm with a substantial research budget.

So when somebody offers you the shiny new market research tool which is now the “best practice” or the “benchmarked metric,” set it aside.  Instead, outline your questions.  Describe the data and information that would help you achieve your most critical business objectives.  Ask whether research is the most efficient and insightful tool to deliver answers.

If you need help with that process, we are happy to advise.

—Joe Hopper, Ph.D.

Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process.  We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.

But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing.  That’s an unusually high ratio of “suspends” as we call them.  So what was the problem?  Were we just getting lousy respondents who did not want to seriously participate in a survey?  Was the survey was too difficult, tedious, boring, or confusing?  One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.

The story in this data: Something is wrong with your survey

Nearly every respondent who quit got close to finishing and then dropped off at exactly the same point, which was odd because the most difficult questions were earlier in the survey.  In fact, the question where most ended up quitting was an interesting drag-and-drop interactive exercise.  Ah, that was the problem.  The programming for the interactive piece was flawed and respondents were being kicked out.

It wasn’t without a good deal of angst that the programming team tested, re-tested, and confirmed the error.  Everybody involved in this effort resisted: the sampling provider, the programmers, the survey tool developers, the questionnaire designers—they have all done this work hundreds of times, so there can’t be anything wrong with their piece of it, right?

Let the data speak.  It will tell you where the mistakes are. There are lots of places and moments where things may go wrong.  If the top people responsible for the project do not have immediate visibility into the data, they are unable to suggest smart solutions, and bad research will just keep happening.

—Joe Hopper, Ph.D.

Finding Gold in Your Data Mine

The truth, however, is that while technology and tools can multiply our capabilities and help us work better and faster, they cannot discover meaningful patterns or find hidden insights. Only smart people can do that.  The reason is that market research data only become meaningful within a context of questions that need to be answered, or stories that need to be told.  Tools and technology cannot supply that context.

We are working with a client who has been struggling for the last five months to find a story in survey data.  They commissioned the survey to generate data for a whitepaper for presentation to business level clients and prospects.  They’ve been staring at tables and banner tabs, pie charts and bar charts, correlations and gap analyses.  But squeeze the data as they might, the story will not emerge.

Here was our counsel:  Before you can find a story in the data, be more explicit about the context that will bring it to life.  So we asked each person on the team to write three or four dream headlines that they would like to see come out of the research.  Our instructions were as follows:

1.  Focus on headlines that would be most useful to communicate to your audience

2.  Do not look at the survey or the data—forget about what you think it says

3.  Do not go back to any of your previous documents or thinking—do it top of mind, quickly

4.  Do not worry about making your headlines pretty or accurate—make it a brain dump of your dream headlines

Of course we cannot guarantee that the data will support every claim they want to make.  But we can guarantee that with so many data points and ways of linking data in even the smallest of data sets, there are compelling ways to make that data support a story that is on target and relevant.

How do you find the gold in your data mine?  By providing a frame of reference in which the data becomes gold.  Take three steps backward to consider the critical (and specific) questions you need to have answered and outline the relevant stories that would be useful to your audiences.  Then start mining your data for answers to those questions and for data points, contradictions, anomalies, and surprising patterns that relate to your stories.

—Joe Hopper, Ph.D.

Besides the ugliness of taking a verb (to act) and turning it into a noun (action) and then forcing that into an adjective (actionable), “actionable insight” just doesn’t mean much in our industry.  Now we have clients with reports full of “actionable insights” collecting dust on their shelves.

In our view, the problem is that few research professionals make an explicit link in the design phase of their research between the data that will be generated, and the specific decisions that need to be made.  If that link is not specified, then even if the report is rich, detailed, and full of insight, chances are it will not be used.  And if it is not used, it probably was not “actionable” to begin with.

Here is an example.  Suppose you measure customer satisfaction, and then probe deeply into several specific areas.  Many think that including more areas to measure and getting granular-level detail will yield “actionable insights.” You will get specific information to help you formulate actions for improvement, right?  Wrong.  It is only actionable if the business is able to address (and prepared to invest in) each of those areas.  If you already know there is nothing you can do about the speed of delivery to your customers, then do not ask about it.  If you already know you cannot act on an issue, then what the data say will not matter, nor will it matter how much insight you bring to that data.  The decision will be the same, and the research will not be used.

Actionable research is not easy.  It goes against our inclination as researchers to get the fullest possible picture of a problem so that we can offer multiple specific solutions and insights.  It also requires a great deal of thinking, listening, and understanding from senior-level researchers who must be intent on understanding you and your business.  The “actionable insights” boilerplate protocols will not work.

Here is a starting point we recommend:  The Art of Asking Questions.  This newsletter article from Versta Research focuses on asking the right questions of your business partners to ensure that research truly speaks to what they will do with it.  It outlines a process you can use internally, and it is a process that we use with our clients as well.  If you do it right, we guarantee you’ll hear your business partners thanking you for “the actionability of your deliverables” — or better yet, for delivering research they can really use.

—Joe Hopper, Ph.D.

The authors looked at various types of biases that creep into business managers’ decisions when based on data presented to them.  They did this by conducting experiments with business school students and managers who are members of the American Marketing Association.  Some were presented with numeric data in tables, while others were presented with data in charts or graphs.  All tables, charts, and graphs were clear and well-designed.

The results?  When compared to an optimal decision based on a purely rational assessment of the data, decisions were typically biased, and “graphical formats that followed existing recommendation for the appropriate display of data did not reduce . . . biases compared with data presented in tables.”  Moreover, “neither real-world experience nor explicit training reduced these biases.”

In some ways this is surprising, because graphing data can often help us better (and more quickly) grasp its meaning.  On the other hand it is not surprising.  Graphs can be so visually compelling that they might hinder purely rational assessments.

In our view (and the authors’ as well) the research does not invalidate the need for effective data visualization.  A good chart can be a compelling piece of story.  But it is not the story, any more than a table of numbers can be the story.  A good chart is a communication tool.  So are good tables, and so are good sentences that weave together a compelling story.

At Versta Research we use chart and tables in about equal proportions (sometimes tables work better) and we always integrate them into a clear story with appropriate statistical modeling to lend support.  The solution to overcoming bias is not fancier graphics, but rather a clear presentation of information with a compelling synthesis and assessment of what it means.

—Joe Hopper, Ph.D.