One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

Earlier this month Arthur C. Nielsen, Jr. died.  He left behind a giant and reputable market research company and a brand name recognized throughout the world.  The A.C. Nielsen company was started by his father and in its early years tracked the sales of goods through grocery and drug stores.  The company then moved into media tracking and became the authoritative source for measuring audience size and demographics.  Nearly every company with an advertising budget continues to rely on Nielsen data to determine where to advertise and how much to spend.

Nielsen’s legacy is that he demonstrated the value of collecting and tracking data, and lots of it.  Every item we purchase is now logged, counted, and tracked.  Every television and radio show is tracked for how many viewers it has and in what markets they live.  And of course everything we do on the Internet is recorded and tracked.  Even our bodily locations are tracked via GPS or cell phone signals.  Most market research firms today generate the bulk of their revenue simply by collecting, tracking, tabulating, and reporting data.

This important legacy has left us with tons of data, growing at an exponential rate,  and a monumental challenge of how to synthesize it and move beyond mere tabulation and reporting.  The question is, how do we meet that challenge and take Nielsen’s legacy to the next frontier?  In our view, it will involve two key efforts:

1.  Understanding data in much deeper ways and analyzing it with data mining tools, new algorithms, and new approaches that go beyond traditional statistics, including Bayesian analysis, neural networks, and machine-learning techniques.

2. Interpreting and communicating data in ways that are more practical, relevant, meaningful, and useful.  In other words, turning data into stories that real people, real managers, and real businesses understand and can use.

To be sure, much of the research industry is still (appropriately) focused on implementing technologies to better manage, tabulate, and report volumes of data.  But automated tables and charts with ever-expanding levels of detail are reaching their limits of utility.  Versta Research is proud to be on the next frontier, where better interpretation and understanding of data is key.

–Joe Hopper, Ph.D.

Two researchers at the U.S. Census Bureau recently outlined an emerging innovation in survey research that could reverse the trend towards passive, boring, self-administered surveys that characterizes much online research.  The idea is to use internet avatars in real-time interviewing with survey respondents.

Beyond just the heightened interest of having an animated survey, the avatars would be programmed to register and interpret respondents’ verbal answers, facial expressions, and body language through webcams.

Suppose, for example, that a respondent answers a question with detailed information that answers a follow-up question as well.  The avatar would use natural language processing to insert that data into the subsequent question, and then avoid asking the follow-up.  Or if the respondent looks away from the screen and pauses for time longer than is typical, the avatar can offer a rephrased question or a reassuring comment to re-engage the participant and to put him or her at ease.  This type of innovation  could bring many of the advantages of live interviewing back into the realm of internet surveys, which are far more efficient in terms of time and cost.

The use of effective avatar interviewers is at least several years away, however, because it involves not only evolving internet technologies, but also advanced linguistic processing, facial and voice recognition technologies, and so on.  In fact, the sheer technological difficulty of truly replacing human interviewers reminds us of how absurd it is for research companies to make claims about technology replacing higher-order activities in the research process, such as providing analysis and insight.

At least for now, software and services with “actionable insight” buttons generate yet more mountains of data in need of human synthesis and interpretation.  If anything, the role for smart and experienced researchers who can turn all that data into a story is growing.  It is growing for researchers who work on the client side and who have direct accountability to the executives who need data-driven insights.  And it is growing for firms like Versta Research where the highest levels of intellectual and human capital are central to our work.

–Joe Hopper, Ph.D.

It's pretty, but it's chartjunk

Turning data into stories involves not just words, but pictures as well.  In the world of quantitative market research, that usually means charts, graphs, and tables.  Moreover, just like poorly written sentences that often complicate rather than clarify data, charts and graphs in market research too often suffer from “chartjunk,” as Edward Tufte calls it.  Any superfluous details, design elements, or decorations that do not tell the viewer something new about the data are chartjunk.

At Versta Research we write a lot of reports.  We also revise others’ reports to help our clients find and more clearly present research stories to their management teams.  Here are three of the more common chart design mistakes we see and help our clients avoid:

1.  3-D Charts.  Few of us in market research work in multidimensional spaces, so 3-D charts have no purpose other than to “Bring more creativity to your presentations!” or “Lift your charts above the ordinary!”  In fact, 3-D charts nearly always distort proportions and make it more difficult to compare and contrast relevant data.  For the most part, we keep our charts in flatland.

2.  Grid Lines.  For some reason PowerPoint includes gridlines by default.  But gridlines are rarely needed, and usually they are distracting.  Typically we label all data points, so gridlines that pull your eyes to the axes are superfluous.  That said, when gridlines are useful, we make them light gray so that the data stands out and the grid recedes to the background.

3. Irrelevant Data. The best charts pack amazingly large amounts of data, but in elegant ways that never overwhelm with irrelevant information.  The problem with research data is that we always have more data we could put into a chart, so the key is to figure out which data helps tell the story.  For example, if just 4% of customers express dissatisfaction, there is no reason to show details down to the level of “somewhat dissatisfied” versus “very dissatisfied” versus “extremely dissatisfied.”

Data charts are easy to generate nowadays, perhaps too easy.  Too many charts (and dashboards, and “actionable” report generators) are now data dumps that fail to tell a story any more than the raw data that was dumped into them.

To get your research understood, used, and promoted by your management team, it needs to tell a story.  That requires a thoughtful, deliberate approach whether by words or by pictures.

–Joe Hopper, Ph.D.

Monty Hall in Let's Make A Deal

The hardest part of quantitative market research is not that it involves numbers, math, or even statistics, but that it involves complex problems in probability.

Over the past several years, psychologists have been documenting how difficult it is for us humans to solve even “simple” probability problems.  One fascinating example is a puzzle known as the Monty Hall dilemma based on the 1960’s game show Let’s Make A Deal.  Monty would offer his contestants three doors to choose from, one of which had a valuable prize behind it.  After the contestant chose, Monty would open one of the other two doors, deliberately choosing one that had no prize behind it.  Then he offered the contestant an option of staying with the original choice, or switching to the other unopened door.  Which should the contestant do?

The contestant should always switch.  The odds of winning are two-thirds if she switches, and one-third if she stays.  Most contestants, however, stay with their original choice, believing that the odds of winning are the same whether they stay or switch.  And it turns out that pigeons do a better job solving this puzzle than humans.  In an article published last year in the Journal of Comparative Psychology, researchers showed that if a similar game is played with pigeons, they start to catch on and consistently choose to switch, which maximizes their winnings.  Humans, however, do not.  Not only do we have a hard time grasping the true probabilities conceptually, but even if we play the game over and over, we ignore our experience and learning.

What does this have to do with market research?  Well, behind all the numbers, charts, and percentages that we present to our clients, most of our methods and analyses are based on probabilistic reasoning.  We calculate the probabilities that our sample statistics represent true population values.  We build conjoint or MaxDiff models based on probabilities of certain responses occurring even if we did not measure them directly.  We ask respondents to assess the probabilities of their own behavior (“How likely are you to buy?”) and use those to calculate estimates of market potential. We are dealing with layers upon layers of probabilities.

It is no wonder that market research reports can be so impenetrable and difficult to untangle.  Behind nearly every chart or table is a probability puzzle, and for most of us there is certainly nothing intuitive about probabilities.

And it is no wonder that many research firms do not even try to go beyond giving you charts, data, and tabulations.  But that’s where a firm like Versta Research comes in.  We solve two of the most difficult challenges facing research professionals:  (1) grasping the complex nature of probabilistic reasoning, which may befuddle even the most accomplished mathematicians, and (2) turning mounds (or crumbs) of data and probabilistic reasoning into an effective and compelling story that you can use and that your clients can understand.

When you need help with either or both of these challenges, call us at (312) 348-6089.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/pigeons-beat-people-on-probability-problems/feed/ 0 http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/ http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/#comments Thu, 14 Jul 2011 17:42:55 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1500 How statistics are calculated and presented has a huge effect on how audiences interpret information and make decisions.  A recent study about medical decisions based on drug efficacy data highlights the critical importance of how you turn your data into stories, no matter what industry.  The research shows that different stories, all of them true and all of them based on the same data will lead to sharply different assessments and decisions.  An article in the New York Times summarized one scenario tested by the researchers:

If your doctor tells you that highly reliable studies have shown that taking a certain pill will cut your risk of getting a serious disease in half, would you take it?

Suppose he adds that the risk is 2 percent for people who do not take the pill, but your risk will be reduced to 1 percent if you do. Would you still take it? And what would you do if he told you that only one of every 100 patients who take the drug will actually benefit from it?

The doctor could have said any of these things, all truthfully, because they are just different ways of describing the same data.

The researchers showed that the data’s persuasiveness and understandability, and the subjects’ views of efficacy varied dramatically for each of these three ways of presenting the data.  Moreover, education or expertise among those being presented with the data made no difference, with both physicians and patients responding in the same ways.

Data have no meaning without an implicit or explicit story to communicate that data.  And indeed the story tells the audience what to do with that data and how to interpret it.  That is what is happening in the research described above, and that is why good research is far more than collecting data, generating tabs, producing charts, and writing phrases that presumably summarize findings.  Good research must turn data into stories, so that your clients and audiences understand the questions, see and grasp the answers, and then act upon the research in smart and effective ways.

Need help?  Turning data into stories is central to our approach for the most complex, esoteric, or even the most mundane studies.  Versta Research can help you get your research heard, understood, and used.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/feed/ 0 http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/ http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/#comments Thu, 07 Jul 2011 18:59:16 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1484 Versta Research just hit a magic number: 100.  That’s the number of articles we have written to help our clients and their colleagues keep abreast of important trends in market research.  If your market research supplier is not providing ongoing thought leadership in design, methods, and analytics, then what are the chances they are bringing ongoing and deep insight to your specific research needs?

To celebrate, we’re serving up a sampler of our five best articles.  How did we decide they are the best?  Our clients told us.  These are the articles that they write to us about, forward to their colleagues, and for which they return to our website time and again.  These are also the articles for which we get requests for print-ready PDF versions.  (Just let us know if you want one!)

1. Turning Data into Stories

There are two critical elements to top notch research.  First, it has to be right, which means focusing on the rigors of research design, data collection, and statistical analysis.  Second, it has to be heard, understood, and used, and in our view that means turning data into stories.  In this article we focus on what it means to turn data into stories, and we outline what you gain by doing so.

2. The Art of Asking Questions

Somewhere along the way to research becoming central to how businesses learn about their customers, the art of asking questions was lost. As a result, there is a lot of research for research’s sake, data in search of answers, and findings in search of questions.  The first thing you should do before starting research is figure out what question your research must answer.  Here’s how.

3. Magic Numbers in Market Research

Researchers cling to a handful of “magic numbers” that guide the decisions they make.  There are magic numbers for sample size, the optimal number of points on a scale, thresholds for statistical significance, and how big a focus group should be.  This article demonstrates and explains.

4. The ABCs of CBC: Understanding Conjoint for Market Research

This article focuses on the basic ideas, advantages, and uses of conjoint research. What is conjoint? How and why is it used? What insights can it give you? Furthermore, what are some of the pros and cons of fielding research using a conjoint method vs. other methods you might use?

5. Game Changing Product Innovation

A lot of research supporting new product development is a like machine that ends up creating NON-innovation because of over-benchmarking.  We suggest an alternative.

We write these articles because even though we are in the business of doing rigorous research for our clients, research only matters if it is thoughtfully communicated, understood, and used.  We hope that our efforts help your organization better design and deploy research to make smarter business decisions.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/feed/ 0 http://www.verstaresearch.com/blog/how-data-can-highlight-mistakes/ http://www.verstaresearch.com/blog/how-data-can-highlight-mistakes/#comments Thu, 12 May 2011 15:11:00 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1346 We are often surprised by the number of senior researchers in the market research industry who never touch raw data.  Often they don’t even have the tools, since “data processing” is outsourced to lower levels or other countries.  It is surprising because we almost always engage in work where getting into the data and puzzling over anomalies or hypotheses yields much deeper insight.

Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process.  We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.

But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing.  That’s an unusually high ratio of “suspends” as we call them.  So what was the problem?  Were we just getting lousy respondents who did not want to seriously participate in a survey?  Was the survey was too difficult, tedious, boring, or confusing?  One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.

The story in this data: Something is wrong with your survey

Nearly every respondent who quit got close to finishing and then dropped off at exactly the same point, which was odd because the most difficult questions were earlier in the survey.  In fact, the question where most ended up quitting was an interesting drag-and-drop interactive exercise.  Ah, that was the problem.  The programming for the interactive piece was flawed and respondents were being kicked out.

It wasn’t without a good deal of angst that the programming team tested, re-tested, and confirmed the error.  Everybody involved in this effort resisted: the sampling provider, the programmers, the survey tool developers, the questionnaire designers—they have all done this work hundreds of times, so there can’t be anything wrong with their piece of it, right?

Let the data speak.  It will tell you where the mistakes are. There are lots of places and moments where things may go wrong.  If the top people responsible for the project do not have immediate visibility into the data, they are unable to suggest smart solutions, and bad research will just keep happening.

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

Finding Gold in Your Data Mine

The truth, however, is that while technology and tools can multiply our capabilities and help us work better and faster, they cannot discover meaningful patterns or find hidden insights. Only smart people can do that.  The reason is that market research data only become meaningful within a context of questions that need to be answered, or stories that need to be told.  Tools and technology cannot supply that context.

We are working with a client who has been struggling for the last five months to find a story in survey data.  They commissioned the survey to generate data for a whitepaper for presentation to business level clients and prospects.  They’ve been staring at tables and banner tabs, pie charts and bar charts, correlations and gap analyses.  But squeeze the data as they might, the story will not emerge.

Here was our counsel:  Before you can find a story in the data, be more explicit about the context that will bring it to life.  So we asked each person on the team to write three or four dream headlines that they would like to see come out of the research.  Our instructions were as follows:

1.  Focus on headlines that would be most useful to communicate to your audience

2.  Do not look at the survey or the data—forget about what you think it says

3.  Do not go back to any of your previous documents or thinking—do it top of mind, quickly

4.  Do not worry about making your headlines pretty or accurate—make it a brain dump of your dream headlines

Of course we cannot guarantee that the data will support every claim they want to make.  But we can guarantee that with so many data points and ways of linking data in even the smallest of data sets, there are compelling ways to make that data support a story that is on target and relevant.

How do you find the gold in your data mine?  By providing a frame of reference in which the data becomes gold.  Take three steps backward to consider the critical (and specific) questions you need to have answered and outline the relevant stories that would be useful to your audiences.  Then start mining your data for answers to those questions and for data points, contradictions, anomalies, and surprising patterns that relate to your stories.

—Joe Hopper, Ph.D.