One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

Earlier this month Arthur C. Nielsen, Jr. died.  He left behind a giant and reputable market research company and a brand name recognized throughout the world.  The A.C. Nielsen company was started by his father and in its early years tracked the sales of goods through grocery and drug stores.  The company then moved into media tracking and became the authoritative source for measuring audience size and demographics.  Nearly every company with an advertising budget continues to rely on Nielsen data to determine where to advertise and how much to spend.

Nielsen’s legacy is that he demonstrated the value of collecting and tracking data, and lots of it.  Every item we purchase is now logged, counted, and tracked.  Every television and radio show is tracked for how many viewers it has and in what markets they live.  And of course everything we do on the Internet is recorded and tracked.  Even our bodily locations are tracked via GPS or cell phone signals.  Most market research firms today generate the bulk of their revenue simply by collecting, tracking, tabulating, and reporting data.

This important legacy has left us with tons of data, growing at an exponential rate,  and a monumental challenge of how to synthesize it and move beyond mere tabulation and reporting.  The question is, how do we meet that challenge and take Nielsen’s legacy to the next frontier?  In our view, it will involve two key efforts:

1.  Understanding data in much deeper ways and analyzing it with data mining tools, new algorithms, and new approaches that go beyond traditional statistics, including Bayesian analysis, neural networks, and machine-learning techniques.

2. Interpreting and communicating data in ways that are more practical, relevant, meaningful, and useful.  In other words, turning data into stories that real people, real managers, and real businesses understand and can use.

To be sure, much of the research industry is still (appropriately) focused on implementing technologies to better manage, tabulate, and report volumes of data.  But automated tables and charts with ever-expanding levels of detail are reaching their limits of utility.  Versta Research is proud to be on the next frontier, where better interpretation and understanding of data is key.

–Joe Hopper, Ph.D.

We were reminded of this upon hearing prominent news reports a few days ago that the average household income in the U.S. has fallen by about 10% in the past decade, most of it happening since the start of the recession four years ago.  But does that mean most Americans’ incomes are falling?  No.  Though it is hard not to think so given how the data are being presented and reported.

The problem is that statistical averages such as means and medians do not tell us what is happening to specific people or groups of people.  They are abstract properties of the whole, but they do not directly describe the parts making up the whole.

Unemployment has grown from 4% to 9%.  As more people lose their income, the average is pulled down even if a large majority remain employed and see no decrease in their income.  There are other factors that pull down the average as well.  Some people who lose their jobs are taking new jobs at lower wages.  Some people have wage or salary increases, but the increases are lower than the rate of inflation.

So overall (at the aggregate level) it is true that income has fallen.  But for most parts of the whole (at the level of individuals), it is probably not true.  We should have been told not only averages, but also the percentage of households that have seen a decline in their individual incomes over the last decade.  Almost certainly it would paint a different picture.

When we at Versta Research interpret and report market research and polling data, we use percentages far more often than averages for exactly this reason.  We are focused on the story that the data tell, which is usually about people not about abstract wholes.  It requires careful attention not only to the stats and the data, but also to the interpretation and communication of those stats so that your audience has a realistic picture of what they mean.

–Joe Hopper, Ph.D.

It's pretty, but it's chartjunk

Turning data into stories involves not just words, but pictures as well.  In the world of quantitative market research, that usually means charts, graphs, and tables.  Moreover, just like poorly written sentences that often complicate rather than clarify data, charts and graphs in market research too often suffer from “chartjunk,” as Edward Tufte calls it.  Any superfluous details, design elements, or decorations that do not tell the viewer something new about the data are chartjunk.

At Versta Research we write a lot of reports.  We also revise others’ reports to help our clients find and more clearly present research stories to their management teams.  Here are three of the more common chart design mistakes we see and help our clients avoid:

1.  3-D Charts.  Few of us in market research work in multidimensional spaces, so 3-D charts have no purpose other than to “Bring more creativity to your presentations!” or “Lift your charts above the ordinary!”  In fact, 3-D charts nearly always distort proportions and make it more difficult to compare and contrast relevant data.  For the most part, we keep our charts in flatland.

2.  Grid Lines.  For some reason PowerPoint includes gridlines by default.  But gridlines are rarely needed, and usually they are distracting.  Typically we label all data points, so gridlines that pull your eyes to the axes are superfluous.  That said, when gridlines are useful, we make them light gray so that the data stands out and the grid recedes to the background.

3. Irrelevant Data. The best charts pack amazingly large amounts of data, but in elegant ways that never overwhelm with irrelevant information.  The problem with research data is that we always have more data we could put into a chart, so the key is to figure out which data helps tell the story.  For example, if just 4% of customers express dissatisfaction, there is no reason to show details down to the level of “somewhat dissatisfied” versus “very dissatisfied” versus “extremely dissatisfied.”

Data charts are easy to generate nowadays, perhaps too easy.  Too many charts (and dashboards, and “actionable” report generators) are now data dumps that fail to tell a story any more than the raw data that was dumped into them.

To get your research understood, used, and promoted by your management team, it needs to tell a story.  That requires a thoughtful, deliberate approach whether by words or by pictures.

–Joe Hopper, Ph.D.

Monty Hall in Let's Make A Deal

The hardest part of quantitative market research is not that it involves numbers, math, or even statistics, but that it involves complex problems in probability.

Over the past several years, psychologists have been documenting how difficult it is for us humans to solve even “simple” probability problems.  One fascinating example is a puzzle known as the Monty Hall dilemma based on the 1960’s game show Let’s Make A Deal.  Monty would offer his contestants three doors to choose from, one of which had a valuable prize behind it.  After the contestant chose, Monty would open one of the other two doors, deliberately choosing one that had no prize behind it.  Then he offered the contestant an option of staying with the original choice, or switching to the other unopened door.  Which should the contestant do?

The contestant should always switch.  The odds of winning are two-thirds if she switches, and one-third if she stays.  Most contestants, however, stay with their original choice, believing that the odds of winning are the same whether they stay or switch.  And it turns out that pigeons do a better job solving this puzzle than humans.  In an article published last year in the Journal of Comparative Psychology, researchers showed that if a similar game is played with pigeons, they start to catch on and consistently choose to switch, which maximizes their winnings.  Humans, however, do not.  Not only do we have a hard time grasping the true probabilities conceptually, but even if we play the game over and over, we ignore our experience and learning.

What does this have to do with market research?  Well, behind all the numbers, charts, and percentages that we present to our clients, most of our methods and analyses are based on probabilistic reasoning.  We calculate the probabilities that our sample statistics represent true population values.  We build conjoint or MaxDiff models based on probabilities of certain responses occurring even if we did not measure them directly.  We ask respondents to assess the probabilities of their own behavior (“How likely are you to buy?”) and use those to calculate estimates of market potential. We are dealing with layers upon layers of probabilities.

It is no wonder that market research reports can be so impenetrable and difficult to untangle.  Behind nearly every chart or table is a probability puzzle, and for most of us there is certainly nothing intuitive about probabilities.

And it is no wonder that many research firms do not even try to go beyond giving you charts, data, and tabulations.  But that’s where a firm like Versta Research comes in.  We solve two of the most difficult challenges facing research professionals:  (1) grasping the complex nature of probabilistic reasoning, which may befuddle even the most accomplished mathematicians, and (2) turning mounds (or crumbs) of data and probabilistic reasoning into an effective and compelling story that you can use and that your clients can understand.

When you need help with either or both of these challenges, call us at (312) 348-6089.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/pigeons-beat-people-on-probability-problems/feed/ 0 http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/ http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/#comments Thu, 14 Jul 2011 17:42:55 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1500 How statistics are calculated and presented has a huge effect on how audiences interpret information and make decisions.  A recent study about medical decisions based on drug efficacy data highlights the critical importance of how you turn your data into stories, no matter what industry.  The research shows that different stories, all of them true and all of them based on the same data will lead to sharply different assessments and decisions.  An article in the New York Times summarized one scenario tested by the researchers:

If your doctor tells you that highly reliable studies have shown that taking a certain pill will cut your risk of getting a serious disease in half, would you take it?

Suppose he adds that the risk is 2 percent for people who do not take the pill, but your risk will be reduced to 1 percent if you do. Would you still take it? And what would you do if he told you that only one of every 100 patients who take the drug will actually benefit from it?

The doctor could have said any of these things, all truthfully, because they are just different ways of describing the same data.

The researchers showed that the data’s persuasiveness and understandability, and the subjects’ views of efficacy varied dramatically for each of these three ways of presenting the data.  Moreover, education or expertise among those being presented with the data made no difference, with both physicians and patients responding in the same ways.

Data have no meaning without an implicit or explicit story to communicate that data.  And indeed the story tells the audience what to do with that data and how to interpret it.  That is what is happening in the research described above, and that is why good research is far more than collecting data, generating tabs, producing charts, and writing phrases that presumably summarize findings.  Good research must turn data into stories, so that your clients and audiences understand the questions, see and grasp the answers, and then act upon the research in smart and effective ways.

Need help?  Turning data into stories is central to our approach for the most complex, esoteric, or even the most mundane studies.  Versta Research can help you get your research heard, understood, and used.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/feed/ 0 http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/ http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/#comments Thu, 07 Jul 2011 18:59:16 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1484 Versta Research just hit a magic number: 100.  That’s the number of articles we have written to help our clients and their colleagues keep abreast of important trends in market research.  If your market research supplier is not providing ongoing thought leadership in design, methods, and analytics, then what are the chances they are bringing ongoing and deep insight to your specific research needs?

To celebrate, we’re serving up a sampler of our five best articles.  How did we decide they are the best?  Our clients told us.  These are the articles that they write to us about, forward to their colleagues, and for which they return to our website time and again.  These are also the articles for which we get requests for print-ready PDF versions.  (Just let us know if you want one!)

1. Turning Data into Stories

There are two critical elements to top notch research.  First, it has to be right, which means focusing on the rigors of research design, data collection, and statistical analysis.  Second, it has to be heard, understood, and used, and in our view that means turning data into stories.  In this article we focus on what it means to turn data into stories, and we outline what you gain by doing so.

2. The Art of Asking Questions

Somewhere along the way to research becoming central to how businesses learn about their customers, the art of asking questions was lost. As a result, there is a lot of research for research’s sake, data in search of answers, and findings in search of questions.  The first thing you should do before starting research is figure out what question your research must answer.  Here’s how.

3. Magic Numbers in Market Research

Researchers cling to a handful of “magic numbers” that guide the decisions they make.  There are magic numbers for sample size, the optimal number of points on a scale, thresholds for statistical significance, and how big a focus group should be.  This article demonstrates and explains.

4. The ABCs of CBC: Understanding Conjoint for Market Research

This article focuses on the basic ideas, advantages, and uses of conjoint research. What is conjoint? How and why is it used? What insights can it give you? Furthermore, what are some of the pros and cons of fielding research using a conjoint method vs. other methods you might use?

5. Game Changing Product Innovation

A lot of research supporting new product development is a like machine that ends up creating NON-innovation because of over-benchmarking.  We suggest an alternative.

We write these articles because even though we are in the business of doing rigorous research for our clients, research only matters if it is thoughtfully communicated, understood, and used.  We hope that our efforts help your organization better design and deploy research to make smarter business decisions.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/top-5-picks-best-articles-on-market-research/feed/ 0 http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/ http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/#comments Thu, 28 Apr 2011 13:32:38 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1306 As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys.  A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion.  The insanely popular survey question used to calculate Net Promoter Scores is a good example:

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

It’s an oddly common situation we find ourselves in these days: advising our client against more research.  For advocates of information and fact-based strategies (including us), the increasingly central role for market research is gratifying.  But too many surveys and research-for-research’s-sake can’t sustain itself, nor should it.  Market research only matters if it is acted upon and used in smart and strategic ways.

How do you know if, instead of research, you should focus on a full-force marketing effort or at least a better strategic plan before launching research?  Here are three situations we typically see:

1.  When you are unable to formulate specific questions that need to be answered.  There is a good deal of research done in the name of “best practices” or because the latest Harvard Business Review suggested (foolishly) that everybody ought to be measuring their Net Promoter Score.  Before you do research, you must have specific unanswered questions and a clear understanding of what you would do with those answers.  (See our whitepaper, The Art of Asking Questions.)

2.  When you have a stack of old research reports and are not sure what to do with them. While there is some chance that the research was inherently not “actionable,” there is an even greater chance that it was done without a focused need.  There is also a good chance that if you go back to step one—formulating the questions that need to be answered—you may find some answers in those old reports.  The story and the insights in research data never speak for themselves.  They need the context of questions that need to be answered and that can be acted upon.

3.  When your organization is struggling to organize and implement marketing campaigns.  There is often a lot of low-hanging fruit in marketing, and if you’re not grabbing it, then your resources may be better spent fixing your marketing operation than collecting new data.  We recently advised a company that was struggling to implement any consistent marketing campaign beyond an e-mail blast to current customers.  What would they do with research?

Market research can never substitute for doing the work of marketing.  Nor can another round of data and “actionable insights” replace the crucial role of marketing leadership, know-how, and initiative.

Give us a call and we would be happy to help you think more thoroughly about your research needs.  We love solving problems, finding information, and turning data into stories that answer your critical questions.  We also love the point at which we tell our clients, “Enough with the research already—it is time for you to get out there and sell!”

—Joe Hopper, Ph.D.

The backstory:  A respected academic journal in social psychology published an article showing data that suggests ESP exists.  Horrified, some researchers argued that psychologists were using old-fashioned inferential statistics when they should be using modern-day Bayesian statistics.  Here is a link to the article, if you’re interested.  Unfortunately, it does a lousy job explaining what Bayesian statistics is.

But fortunately, in response, the editor in chief of The Annals of Applied Statistics submitted a letter to the New York Times clarifying that all statistics ends up relying on arbitrary magic numbers:

The heart of the dispute is not about Bayesian versus classical statistical methods; if anything, it is an argument against knee-jerk use of the famous .05 criterion, which generally finds the results of an experiment acceptable if the chances are no greater than 5 percent (that is, 0.05) that they could have occurred randomly.

Physicists, for example, don’t trust .05 and prefer much tougher evidential levels. A claimed result that overturns all ideas of causality might well require something stricter than .05. A Bayesian would have to make the same kind of difficult choice as to what “prior probability” to assign to the existence of ESP.

No general formula will free the scientist, or anyone else, from having to use judgment in interpreting evidence. But general formulas, including .05, are valuable in imposing some order on the Wild West world of claimed results.

Now there’s a statistician after our own heart.  We, including market researchers, all use magic numbers, because in a strange way, magic numbers work.  They help us impose order and make sense of the messy reality we are trying to understand.  But ultimately there is nothing magic about them, and there is no magic solution that can substitute for seasoned judgment and expertise.

That’s why you come to us, right?  It’s easy to push a button that tells whether data is statistically significant.  It’s not so easy to discern the story behind that data, whether that “statistical significance” really matters, and what you should do with it.  That’s the magic of the work we do at Versta Research.

—Joe Hopper, Ph.D.