It does not take super fancy (and expensive) mapping software or specialized firms to create accurate, useful, and compelling maps from market research data.  We recently created maps for a client showing where in a three-county region their best customers lived.  Everything we used to make these maps was free and publicly available for download on the Internet.  Here are the steps we used:

1.  Download shapefiles from the U.S. Census Bureau.  These files contain data to demarcate all legal and statistical geographical areas in the U.S. including states, counties, county subdivisions, census tracts, blocks, and so on.

2.  Edit the shapefiles with a program like QGIS.  There are several high quality, free, open-source software packages that you can use to read and manipulate census shapefiles.  We used QGIS, which is a program created and continually developed by the Open Source Geospatial Foundation.

3.  Link customer data to shapefile data in a spreadsheet program.  We looked at the number of customers in every zip code, then linked that data to county subdivisions in the shapefile by using a minimum distance function based on latitude and longitude coordinates.

4.  Plot the data and create the map using R.  R is quickly becoming the statistics package of choice in the academic world.  It is a free “integrated suite of software…for statistical computing and graphics” and can easily turn shapefiles and data linked to those shapefiles into visual displays.

Ultimately we created a heat map that displays customer location data for the three counties, which are divided into more than 50 townships, as shown in the map above, with darker colors signifying more customers than lighter colors.

As always, the ongoing challenge for researchers working with a burgeoning volume of data is how to interpret all that data, synthesize it, and simplify it into a story that is useful to decision makers.  Maps have always been a useful and compelling way to visually present data.  Finding the path to producing them from your data is now easier than ever.

–Joe Hopper, Ph.D.

One powerful way to gain visibility and credibility in your marketplace is by sponsoring survey research that documents problems and solutions in areas where you have expertise.  To be successful, it requires (1) rigorous research carefully designed to uncover the right topics, and (2) savvy PR work that uses data to tell a credible and compelling story.

The Public Relations Society of America (PRSA) and the American Statistical Association have just published a handy guide for PR professionals that outlines best practices for using, interpreting and reporting statistics in press releases and other PR materials.  Some of those best practices include the following:

• Disclose who paid for the work, and who did the research
• Clearly describe how the research was conducted
• Describe the sample of the survey, and whether it was truly random
• Remember that almost any survey can suffer from bias
• Rely on descriptive statistics to report the data
• Remember that all statistical research includes some level of uncertainty
• Clearly describe trends and effects
• Avoid making claims about the future based on recent history
• Use causal statements cautiously, as they are very hard to prove
• Run your insights by the person who did the research to be sure the data support it

Turning data into stories is never easy, but as this partnership between PRSA and the ASA makes clear, it is becoming more and more important for PR professionals to have a working knowledge of statistics.  And if Shonali Burke’s report from the 2011 PRSA conference is any indication, PR professionals are embracing the challenge.

Versta Research can help you with this challenge.  We are experts in research.  We know how to design surveys and report statistics that get your message heard.  Give us a call at (312) 348-6089 when you need a compelling custom survey or omnibus survey to help you tell your story.

–Joe Hopper, Ph.D.

We were reminded of this upon hearing prominent news reports a few days ago that the average household income in the U.S. has fallen by about 10% in the past decade, most of it happening since the start of the recession four years ago.  But does that mean most Americans’ incomes are falling?  No.  Though it is hard not to think so given how the data are being presented and reported.

The problem is that statistical averages such as means and medians do not tell us what is happening to specific people or groups of people.  They are abstract properties of the whole, but they do not directly describe the parts making up the whole.

Unemployment has grown from 4% to 9%.  As more people lose their income, the average is pulled down even if a large majority remain employed and see no decrease in their income.  There are other factors that pull down the average as well.  Some people who lose their jobs are taking new jobs at lower wages.  Some people have wage or salary increases, but the increases are lower than the rate of inflation.

So overall (at the aggregate level) it is true that income has fallen.  But for most parts of the whole (at the level of individuals), it is probably not true.  We should have been told not only averages, but also the percentage of households that have seen a decline in their individual incomes over the last decade.  Almost certainly it would paint a different picture.

When we at Versta Research interpret and report market research and polling data, we use percentages far more often than averages for exactly this reason.  We are focused on the story that the data tell, which is usually about people not about abstract wholes.  It requires careful attention not only to the stats and the data, but also to the interpretation and communication of those stats so that your audience has a realistic picture of what they mean.

–Joe Hopper, Ph.D.

One way to analyze the data is to use a simple counting technique.  It works pretty well, by the way.  But the more common way these days is to use advanced statistical modeling that allows for stronger individual-level estimates of importance scores.  This affords better opportunities for segmenting the data and analyzing subgroups.

We typically see analysts transforming the importance scores (or utilities) to a 0—100 scale.  It is a compelling scale because all importance scores add up to 100, which mimics the technique of having respondents allocate 100 importance points to each of the features they care about.

But if you do a lot of MaxDiff studies and frequently present them to your management team, you lose the ability to provide a rule of thumb as to what counts as a “high” score vs. a “low” score.  If one MaxDiff exercise uses fifteen attributes, an importance score of 20 is quite high; if another MaxDiff exercise uses just five attributes, and importance score of 20 is just average.

The solution?  Instead, we typically transform scores to a 0—100*N scale, where N=the number of attributes tested.  If you tested fifteen attributes, transform to a 0—1500 scale.  If you tested five attributes, transform to a 0—500 scale.  With this method, a score of 100 is always the average, scores above 100 are always above average, and scores below 100 are always below average.  Sometimes we transform this into a chart showing “percent above/below  average” as shown in the third example below, but this is not always necessary because anchoring the average at 100 makes it simple to calculate those percentages mentally.

Example 1: Versta's Recommended Scaling of MaxDiff Scores

Example 2: The Typical Way of Scaling MaxDiff Scores

Example 3: Another (Sometimes Useful) Way to Scale MaxDiff Scores

Need help with the smartest ways to design, implement, and report MaxDiff studies?  Call us at (312) 348-6089 for assistance and further information.

–Joe Hopper, Ph.D.

It's pretty, but it's chartjunk

Turning data into stories involves not just words, but pictures as well.  In the world of quantitative market research, that usually means charts, graphs, and tables.  Moreover, just like poorly written sentences that often complicate rather than clarify data, charts and graphs in market research too often suffer from “chartjunk,” as Edward Tufte calls it.  Any superfluous details, design elements, or decorations that do not tell the viewer something new about the data are chartjunk.

At Versta Research we write a lot of reports.  We also revise others’ reports to help our clients find and more clearly present research stories to their management teams.  Here are three of the more common chart design mistakes we see and help our clients avoid:

1.  3-D Charts.  Few of us in market research work in multidimensional spaces, so 3-D charts have no purpose other than to “Bring more creativity to your presentations!” or “Lift your charts above the ordinary!”  In fact, 3-D charts nearly always distort proportions and make it more difficult to compare and contrast relevant data.  For the most part, we keep our charts in flatland.

2.  Grid Lines.  For some reason PowerPoint includes gridlines by default.  But gridlines are rarely needed, and usually they are distracting.  Typically we label all data points, so gridlines that pull your eyes to the axes are superfluous.  That said, when gridlines are useful, we make them light gray so that the data stands out and the grid recedes to the background.

3. Irrelevant Data. The best charts pack amazingly large amounts of data, but in elegant ways that never overwhelm with irrelevant information.  The problem with research data is that we always have more data we could put into a chart, so the key is to figure out which data helps tell the story.  For example, if just 4% of customers express dissatisfaction, there is no reason to show details down to the level of “somewhat dissatisfied” versus “very dissatisfied” versus “extremely dissatisfied.”

Data charts are easy to generate nowadays, perhaps too easy.  Too many charts (and dashboards, and “actionable” report generators) are now data dumps that fail to tell a story any more than the raw data that was dumped into them.

To get your research understood, used, and promoted by your management team, it needs to tell a story.  That requires a thoughtful, deliberate approach whether by words or by pictures.

–Joe Hopper, Ph.D.

Monty Hall in Let's Make A Deal

The hardest part of quantitative market research is not that it involves numbers, math, or even statistics, but that it involves complex problems in probability.

Over the past several years, psychologists have been documenting how difficult it is for us humans to solve even “simple” probability problems.  One fascinating example is a puzzle known as the Monty Hall dilemma based on the 1960’s game show Let’s Make A Deal.  Monty would offer his contestants three doors to choose from, one of which had a valuable prize behind it.  After the contestant chose, Monty would open one of the other two doors, deliberately choosing one that had no prize behind it.  Then he offered the contestant an option of staying with the original choice, or switching to the other unopened door.  Which should the contestant do?

The contestant should always switch.  The odds of winning are two-thirds if she switches, and one-third if she stays.  Most contestants, however, stay with their original choice, believing that the odds of winning are the same whether they stay or switch.  And it turns out that pigeons do a better job solving this puzzle than humans.  In an article published last year in the Journal of Comparative Psychology, researchers showed that if a similar game is played with pigeons, they start to catch on and consistently choose to switch, which maximizes their winnings.  Humans, however, do not.  Not only do we have a hard time grasping the true probabilities conceptually, but even if we play the game over and over, we ignore our experience and learning.

What does this have to do with market research?  Well, behind all the numbers, charts, and percentages that we present to our clients, most of our methods and analyses are based on probabilistic reasoning.  We calculate the probabilities that our sample statistics represent true population values.  We build conjoint or MaxDiff models based on probabilities of certain responses occurring even if we did not measure them directly.  We ask respondents to assess the probabilities of their own behavior (“How likely are you to buy?”) and use those to calculate estimates of market potential. We are dealing with layers upon layers of probabilities.

It is no wonder that market research reports can be so impenetrable and difficult to untangle.  Behind nearly every chart or table is a probability puzzle, and for most of us there is certainly nothing intuitive about probabilities.

And it is no wonder that many research firms do not even try to go beyond giving you charts, data, and tabulations.  But that’s where a firm like Versta Research comes in.  We solve two of the most difficult challenges facing research professionals:  (1) grasping the complex nature of probabilistic reasoning, which may befuddle even the most accomplished mathematicians, and (2) turning mounds (or crumbs) of data and probabilistic reasoning into an effective and compelling story that you can use and that your clients can understand.

When you need help with either or both of these challenges, call us at (312) 348-6089.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/pigeons-beat-people-on-probability-problems/feed/ 0 http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/ http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/#comments Thu, 14 Jul 2011 17:42:55 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1500 How statistics are calculated and presented has a huge effect on how audiences interpret information and make decisions.  A recent study about medical decisions based on drug efficacy data highlights the critical importance of how you turn your data into stories, no matter what industry.  The research shows that different stories, all of them true and all of them based on the same data will lead to sharply different assessments and decisions.  An article in the New York Times summarized one scenario tested by the researchers:

If your doctor tells you that highly reliable studies have shown that taking a certain pill will cut your risk of getting a serious disease in half, would you take it?

Suppose he adds that the risk is 2 percent for people who do not take the pill, but your risk will be reduced to 1 percent if you do. Would you still take it? And what would you do if he told you that only one of every 100 patients who take the drug will actually benefit from it?

The doctor could have said any of these things, all truthfully, because they are just different ways of describing the same data.

The researchers showed that the data’s persuasiveness and understandability, and the subjects’ views of efficacy varied dramatically for each of these three ways of presenting the data.  Moreover, education or expertise among those being presented with the data made no difference, with both physicians and patients responding in the same ways.

Data have no meaning without an implicit or explicit story to communicate that data.  And indeed the story tells the audience what to do with that data and how to interpret it.  That is what is happening in the research described above, and that is why good research is far more than collecting data, generating tabs, producing charts, and writing phrases that presumably summarize findings.  Good research must turn data into stories, so that your clients and audiences understand the questions, see and grasp the answers, and then act upon the research in smart and effective ways.

Need help?  Turning data into stories is central to our approach for the most complex, esoteric, or even the most mundane studies.  Versta Research can help you get your research heard, understood, and used.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/the-most-persuasive-way-to-present-data/feed/ 0 http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/ http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/#comments Thu, 28 Apr 2011 13:32:38 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1306 As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys.  A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion.  The insanely popular survey question used to calculate Net Promoter Scores is a good example:

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

It is important to distinguish the two because unfortunately there is plenty of research not designed to answer important questions.  Research is frequently done because someone has a nagging feeling that they need more information to make better decisions or because doing research is considered a “best practice.”  Such research generates lots of data that marketing managers wonder what to do with.  Not surprisingly, they and their bosses start to question the value of research.

Our advice is to do market research only after formulating specific questions and information needs and only after you have a clear idea (in writing) of what you will do with the answers to those questions.  We have produced a whitepaper entitled The Art of Asking Questions (you can download it by clicking on the image) that outlines a useful process to help you formulate those questions.

The Art of Asking Questions: A White Paper from Versta Research

The key is to articulate five types of questions, including:

• The mission-critical questions
• The nice-to-know questions
• The red-herring questions
• The already-answered questions
• The look-elsewhere questions

Next, you outline at least two likely or possible answers to those questions.  And finally you describe what action you might take based on each possible answer.

Articulating specific questions and outcomes will not only vastly improve the quality of your research, it will also make it easier to bring a full executive team on board with the research.  Why?  Because it moves internal discussions away from the value of research per se, towards the importance of the questions that have been formulated and the business value of having answers.

Versta Research would be happy to help you formulate these questions and/or help you decide that research is premature.  Feel free to call us at 312-348-6089.

—Joe Hopper, Ph.D.

There is also, of course, the magic of story-telling, which market research can embrace, but rarely does.  We were inspired by a recent New York Times interview with Aaron Levie, CEO of Box.net.  What could be more boring than online file storage?  But as Levie said:

In middle school, I did magic shows.  It actually applies to what I’m doing now [leading a company] because it’s all about getting in front of people and telling a story. . . It’s all about capturing people’s imaginations and getting them excited about what’s possible.

Market research, too, can be exciting and interesting if it is communicated effectively.  There is nothing inherently boring about numbers (in fact, they are fascinating).  Nor are numbers inherently complicated (in fact, they always boil down to the simplicity of counting).

At Versta Research we are here to help you with both the wizardry of numbers and the magic of stories.  Plus we can help you see through the smoke and mirrors of black box “methodologies” and “magic numbers” that are all too common in market research.  Give us a call, and we would be happy to share with you our secrets.  In the meantime, take a look at the December 2010 Newsletter for our take what’s magic and what’s not magic in market research.

—Joe Hopper, Ph.D.