The authors looked at various types of biases that creep into business managers’ decisions when based on data presented to them.  They did this by conducting experiments with business school students and managers who are members of the American Marketing Association.  Some were presented with numeric data in tables, while others were presented with data in charts or graphs.  All tables, charts, and graphs were clear and well-designed.

The results?  When compared to an optimal decision based on a purely rational assessment of the data, decisions were typically biased, and “graphical formats that followed existing recommendation for the appropriate display of data did not reduce . . . biases compared with data presented in tables.”  Moreover, “neither real-world experience nor explicit training reduced these biases.”

In some ways this is surprising, because graphing data can often help us better (and more quickly) grasp its meaning.  On the other hand it is not surprising.  Graphs can be so visually compelling that they might hinder purely rational assessments.

In our view (and the authors’ as well) the research does not invalidate the need for effective data visualization.  A good chart can be a compelling piece of story.  But it is not the story, any more than a table of numbers can be the story.  A good chart is a communication tool.  So are good tables, and so are good sentences that weave together a compelling story.

At Versta Research we use chart and tables in about equal proportions (sometimes tables work better) and we always integrate them into a clear story with appropriate statistical modeling to lend support.  The solution to overcoming bias is not fancier graphics, but rather a clear presentation of information with a compelling synthesis and assessment of what it means.

—Joe Hopper, Ph.D.

Algorithms are central to the work that we do in business and market research, and they are top of mind for us at Versta Research because we have been involved in several data-intensive projects that involve either (a) developing new algorithms for clients, or (b) tools that apply sophisticated algorithms to data in new and exciting ways.

What is an algorithm?  It is “a mechanical or recursive computational procedure” (American Heritage Dictionary).  New technologies, data capacities, data collection techniques, and speed have indeed made this the age of algorithms.  Here are some interesting examples, including a few we’ve seen in the news lately, and a few that relate to our work:

• The best example (of course) is Google.  Good search engines are amazingly fast, efficient, and smart algorithms that find relevant materials based on just a few words that you supply.

• Yahoo is using search algorithms not just to find materials, but to make decisions about what to cover in their own news reporting. It used to be that editors made decisions about what is newsworthy; now algorithms do.

• IBM is perfecting a machine that can play (and win) the game of Jeopardy against humans.  It involves algorithms that search for multiple meanings of words and facts, and then matches them against an enormous database of information.  It assigns probabilities to each possible answer (or question, as the case may be) and responds accordingly.

• Versta Research completed a project developing a matching algorithm (matching people with products) based on survey data that had thousands of data points on each person and each potential product.  The process involved mining the data for ways of matching that optimized customer satisfaction.

• We also just completed a project that used convergent cluster and ensemble analysis (CCEA) to explore patient segments using patient chart data.  The technique relies on multiple clustering algorithms, generating hundreds of potential solutions and then selecting the optimal one based on a reproducibility algorithm.

What is interesting is that the “basics” of inferential and descriptive statistics are becoming a smaller and smaller piece of data analysis.  More than ever, we combine them with procedures that involve data mining, predictive analytics, Monte Carlo simulations, and so on.

What are the implications for research firms and the clients who employ us?  The value of having extremely well-trained researchers who are fluent not only in statistics, but in mathematics, modeling, logic, and yes, even algorithms, is essential.  And if, on top of that, you have smart people who know how to use, interpret, and tell a story with that data, the value of your market research will really shine.

—Joe Hopper, Ph.D.

One of the fascinating insights to emerge from our discussion was that social media is not only a new channel of information and data, but that it is fundamentally different from previous channels of data. As such, it opens up new areas of inquiry for our efforts. What is that fundamental difference? It is the networked nature of social media. As we code and tabulate people’s comments on social media as part of our CS&L research, potentially we have access to the networks of each person whose comments we are analyzing. We can know how many people are reading each comment, we can measure how strong and extensive the network of influence is, where it overlaps important segments of customers, and so on.

The implications of this are huge. The impact of loyalty among one’s best customers can now be defined not only in terms of how much they buy and how “sticky” they are, but also in terms of their influence among other customers and prospects. Smart CS&L research will not count everyone’s opinion the same, but will give more weight to those occupying central nodes of critical networks. There are implications for sampling as well. Is true random sampling necessary, or can “networked” sampling provided sufficient entry points that give visibility into the full population of customers?

In my view, this sort of network analysis will bring about a seismic shift in CS&L research, though not all panelists agreed. We were split about 50/50 on whether social media constitutes just one more channel of data to integrate, bringing greater precision to what we’ve always done, or whether it represents a more radical departure. A full report of our panel’s deliberations will be presented in the October 2010 issue of Marketing News, the AMA’s monthly magazine.

The stimulating discussion among panelists also highlighted the importance of remembering that good CS&L research requires ongoing thoughtfulness, intelligence, and curiosity. New tools and technologies are often easy to install, but their value is in offering new opportunities to bring deeper understanding and analysis to research.

Stay tuned – we’ll provide a closer look at some of those opportunities for better research (and reprints upon request) when the AMA publishes its report this fall.

—Joe Hopper, Ph.D.

For those of us at Versta Research, he is well known for his focus on understanding versus research:

“Basically, I’m not interested in doing research and I never have been,” he said. “I’m interested in understanding, which is quite a different thing. And often to understand something you have to work it out yourself because no one else has done it.”  (From an interview cited in the New York Times)

Why are we inspired by this?  Because even though we are interested in doing research, we do it for one reason:  To understand things.  In fact, we named our company from the social science concept (and German word) “verstehen” which means to understand.

Fundamentally, we define our work not by proprietary methods, or the specific tools that we use or develop, or the statistical procedures we implement.  We define it by our interest in and approach to understanding.  And that, of course, means using, learning, developing, and inventing whatever tools, processes, data collection techniques, and analysis that will get good answers.  In short, to achieve understanding, we do research.

Understanding is the outcome.  That’s what we really care about (and what you really care about) and what motivates our work.  Research is the means to that end.

If you have not done so, take a look at the homepage of our website.  It’s all about understanding.  We are committed to helping you understand:

• Your clients
• Your prospects
• The public
• Consumers
• Your competition
• Your customers
• Your data
• Your products
• Your image
• The world

If and when research can help you understand, we are here to help.

—Joe Hopper, Ph.D.

How do you bring the two together?  In our view, you bring them together by investing in people who are trained in both methods and who have deep experience in both.  The newsletter article outlines:

• Why it is best to be fluent in both
• The strengths and promises of each method
• The qualities and training needed to excel at both
• The false promise of technology
• Practical ideas for incorporating the strengths of each into a research effort

Ultimately, research is all about asking questions, finding information and data, thinking about it, probing deeper, assimilating and synthesizing it, and then turning data into stories so that the research gets heard and understood.  Whether you use qualitative or quantitative methods, your data needs to tell a compelling human (business) story, and the best stories draw upon both.

Give us a call, and we would be happy to share with you examples and stories of our work that have successfully bridged the gap.  In the meantime, take a look at our June 2010 Newsletter for a deeper understanding of our approach, and what you can expect from our people.

—Joe Hopper, Ph.D.

The idea seems crazy, but as reported in Science, researchers at Carnegie Mellon University have found that certain kinds of twitter data can give them a good read on public sentiment. The looked at things like positive or negative comments about President Obama and found results that aligned with traditional polls. Comments about finances and savings aligned with consumer confidence polls. What does the polling industry think? Here’s what one colleague says: “I believe that I am now going to run to a dark corner of the house and cower in fear of what may come.”

To be sure, we are not recommending that you rely on social media to accurately measure overall public opinion. No way. But in our view, these findings may lead to new insights about how people, individually and collectively, behave and think, and how research — both academic and practical — can harness these new forms of data to measure markets and the social world.  The findings may also force us to re-think our theories of statistical inference that rely on random sampling. There is much research to be done before we will know, but possibilities for social media someday offering insights that are as statistically valid as our current methods is intriguing and nothing to cower about.

Thinking about measuring social media as part of your research plan? We would be happy to advise you. We can help you explore new options in research while ensuring that your research and findings remain rigorous and defensible.

—Joe Hopper, Ph.D.

Today, an industry colleague (another owner of a market research firm) said to me, “We all lust after those big tracking studies.”  For most market research firms, tracking studies are attractive because they involve big samples, multiple ongoing deliverables, and multi-year commitments, all of which means predictable, ongoing (and usually substantial) revenue.

At Versta Research, we lust after them for a different reason: they are the true test of whether our key personnel can really add value and insight to the work that we do.  How do you take something routine and predictable, and turn it into an effort that delivers an “Aha” every week?  Here’s how we do it:

• We find smart, methodologically-rigorous, and business-savvy associates to run the study instead of “capable” research people who are tracking-study-care-takers.

• We clear off our desks and close our doors and examine the data for new angles and insights with every deliverable.

• We look at the data collection process for ways to streamline so that our time is spent on thinking rather than compiling data and putting numbers into tables and charts.

• We advise you on whether the money spent is worth the insight gained, and if collecting and tabulating data is your only goal, whether there are better ways to automate and/or bring the process in-house.

Too often the smartest people avoid tracking studies because they can be boring.  What you need, however, is smart people who are committed and passionate about adding significant value to every project they do for you.  There is nothing boring about finding the story in your data and helping you communicate it to your managers or internal clients.  That is the challenge of tracking studies that we love.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/of-lust-and-tracking-studies/feed/ 0 http://www.verstaresearch.com/blog/don%e2%80%99t-do-research-in-your-sleep/ http://www.verstaresearch.com/blog/don%e2%80%99t-do-research-in-your-sleep/#comments Mon, 03 May 2010 00:41:22 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=482 A colleague in market research once complained to me that he felt bored and unchallenged by all the client satisfaction and loyalty research he was doing, claiming he had mastered it to the point that he could do satisfaction and loyalty research in his sleep. I was struck because I could not think of any market research that I found boring or unchallenging, and certainly none that I could do in my sleep.  On the contrary, my experience is that doing great research requires intellectual work and waking thoughtfulness no matter how many times it is done and for how many clients.

Contrast this with a client for whom we were generating weekly data reports from a tracking study who said that her directive (and hence our directive) was that “nothing goes to my internal clients without insight.”  And so every week we cleared off our desks, closed the door, and examined the data for new angles and insights.  If we were unable to come up with a compelling, non-boring, challenging story, then surely our client’s client would not find it either.  And if nobody can find a compelling story in the data, why is the client spending money on it, and why are we being paid to do it?

In our view, high quality market research requires smart, thinking people who are:

• Curious about your problems
• Passionate about getting the right data to answer the right questions
• Diligent and detailed in turning that data into stories that give you insight

Nobody can do these things in their sleep.  The next time your research partner starts nodding off mid-sentence, “Why, yes, we have done this kind of work so many times we can do this project in our sleep . . .” take another look at the work they are doing for you and ask yourself whether everything they do delivers thoughtful insights you can use.  If it doesn’t, give us a call for a different perspective on how to do research.

—Joe Hopper, Ph.D.

How do we begin the research design process to ensure incisive and smart research that really helps our clients answer their critical questions?  Here are five key elements of the process for us, which we offer to you as tips for your own success when launching an internal research effort:

1.  Focus on the questions that need to be answered, not on research methods.

It is tempting to start the research process by saying “I need a survey about toothpaste preferences” or “We need focus groups about our advertising message.”  A critical first step in research design is to separate the business questions from the methods that will be used to answer them.  (Need help?  See our recent article on this topic, The Art of Asking Questions.)

2.  Consider how the research will be used.

Sometimes research findings are for internal strategic insight, while other times they are needed for communications and public release.  Whom you include in your sample, the kinds of questions you ask, and the answer scales you choose all depend on who needs to see the results, and how they will use them.

3.  Know what is known.

By this we mean that you must do your homework, because many aspects of a problem are likely known from other research, or could be answered easily with existing data.  Good research will build upon what is known, rather than replicating it.

4.  Play with opposites.

If your first impulse is to conduct a survey, consider what would happen if you did qualitative research instead.  And vice versa.  We recently conducted a powerful conjoint study that started out as an idea for focus groups, until the client realized how compelling it would be to have a mathematical model of preference.  The findings from this research informed a crucial and profitable decision for the company, and was deliberated at the highest executive levels in the company.

5.  Plan all steps.

Write a full plan for all steps of the research process, including sampling, questionnaire design, fieldwork, data collection, timing, budget, analysis, and reporting.  Believe it or not, every one of these steps will affect the questions you will want to ask in your research, and how you will ask them.

We’re serious when we say we love writing research proposals.  It’s because we love designing research, which is where brainpower and high level expertise add significant value.  Need help?  We’ll write one for you.  Or, if your current project is strictly do-it-yourself, call us anyway and we would be happy to offer ideas and tips.

—Joe Hopper, Ph.D.

“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”

Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters.  Why?  Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works.

Their conclusion is particularly surprising because although statistical inference based on probability sampling has a solid theoretical underpinning, in practice pure probability samples are almost never achieved.  Nearly always we are faced with low response rates and non-response biases.  We work around this in practical ways (not always supported by extensive theory) to weight the data, adjust for biases, understand the sources of biases, add caveats to our findings, and so on.  And sure enough, our efforts tend to work, so we keep doing it.

There is substantial data to show that carefully managed non-probability online panels can be used to estimate certain population parameters depending on the nature of the study and how exact those estimates need to be.  Someday our academics and social theorists will help us understand why.  Here’s our theory: Social bodies are not composed of individual units (the assumption underlying inferential statistical theory)  but nodes of networks such that non-random entry points to measure social forces can provide as much information as a random selection of individuals.  We’ll leave it to the future theorists to tell us if we’re right.

In the meantime, there is a lot of research for which using non-probability online panels makes good sense (which the AAPOR report acknowledges), so we’ll keep doing it and we’ll keep extending it into new areas even if the theory can’t keep up.  And we’ll keep refining our techniques based on experience and practice to make it work better, and we’ll keep thinking about why it works and lending that insight to the work we do for you.

If you need help or expertise designing and executing your research based on probability or nonprobability samples, online panels or phone, let us know.  We know our statistical theory.  More importantly, we know our practical statistics.  We can help you sort through it all to ensure rigorous research and practical results.

—Joe Hopper, Ph.D.