It's pretty, but it's chartjunk

Turning data into stories involves not just words, but pictures as well.  In the world of quantitative market research, that usually means charts, graphs, and tables.  Moreover, just like poorly written sentences that often complicate rather than clarify data, charts and graphs in market research too often suffer from “chartjunk,” as Edward Tufte calls it.  Any superfluous details, design elements, or decorations that do not tell the viewer something new about the data are chartjunk.

At Versta Research we write a lot of reports.  We also revise others’ reports to help our clients find and more clearly present research stories to their management teams.  Here are three of the more common chart design mistakes we see and help our clients avoid:

1.  3-D Charts.  Few of us in market research work in multidimensional spaces, so 3-D charts have no purpose other than to “Bring more creativity to your presentations!” or “Lift your charts above the ordinary!”  In fact, 3-D charts nearly always distort proportions and make it more difficult to compare and contrast relevant data.  For the most part, we keep our charts in flatland.

2.  Grid Lines.  For some reason PowerPoint includes gridlines by default.  But gridlines are rarely needed, and usually they are distracting.  Typically we label all data points, so gridlines that pull your eyes to the axes are superfluous.  That said, when gridlines are useful, we make them light gray so that the data stands out and the grid recedes to the background.

3. Irrelevant Data. The best charts pack amazingly large amounts of data, but in elegant ways that never overwhelm with irrelevant information.  The problem with research data is that we always have more data we could put into a chart, so the key is to figure out which data helps tell the story.  For example, if just 4% of customers express dissatisfaction, there is no reason to show details down to the level of “somewhat dissatisfied” versus “very dissatisfied” versus “extremely dissatisfied.”

Data charts are easy to generate nowadays, perhaps too easy.  Too many charts (and dashboards, and “actionable” report generators) are now data dumps that fail to tell a story any more than the raw data that was dumped into them.

To get your research understood, used, and promoted by your management team, it needs to tell a story.  That requires a thoughtful, deliberate approach whether by words or by pictures.

–Joe Hopper, Ph.D.

What does Versta Research recommend when it comes to sample size?  Well, the answer always depends on the type of study, the objectives of the study, the audience to whom it will be presented, and of course budget. We can make legitimate statistical calculations based on any sample size, but there are dramatic trade-offs in precision and cost no matter what sample size is chosen.

Here, however, are some general guidelines:

If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision.  The benefit of doing this will almost never exceed the cost of including that many more people in your sample.

The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.

With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point.  But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points.  Another twenty five, fifty, or one hundred respondents can make an important difference when you are looking at small samples.

Most sample sizes will range from about 100 and 1,200 and within this range, for each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates.  So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people.  Some of the healthcare research we do relies on samples sizes of fifty to one-hundred.  Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200.

Unfortunately there is no magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand.  But there is a magic phone number worth trying:  (312) 348-6089.  Versta Research has a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget.  As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/an-interactive-graph-for-choosing-sample-size/feed/ 0 http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/ http://www.verstaresearch.com/blog/advice-for-pr-surveys-avoid-numeric-scales/#comments Thu, 28 Apr 2011 13:32:38 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1306 As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys.  A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion.  The insanely popular survey question used to calculate Net Promoter Scores is a good example:

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

The authors looked at various types of biases that creep into business managers’ decisions when based on data presented to them.  They did this by conducting experiments with business school students and managers who are members of the American Marketing Association.  Some were presented with numeric data in tables, while others were presented with data in charts or graphs.  All tables, charts, and graphs were clear and well-designed.

The results?  When compared to an optimal decision based on a purely rational assessment of the data, decisions were typically biased, and “graphical formats that followed existing recommendation for the appropriate display of data did not reduce . . . biases compared with data presented in tables.”  Moreover, “neither real-world experience nor explicit training reduced these biases.”

In some ways this is surprising, because graphing data can often help us better (and more quickly) grasp its meaning.  On the other hand it is not surprising.  Graphs can be so visually compelling that they might hinder purely rational assessments.

In our view (and the authors’ as well) the research does not invalidate the need for effective data visualization.  A good chart can be a compelling piece of story.  But it is not the story, any more than a table of numbers can be the story.  A good chart is a communication tool.  So are good tables, and so are good sentences that weave together a compelling story.

At Versta Research we use chart and tables in about equal proportions (sometimes tables work better) and we always integrate them into a clear story with appropriate statistical modeling to lend support.  The solution to overcoming bias is not fancier graphics, but rather a clear presentation of information with a compelling synthesis and assessment of what it means.

—Joe Hopper, Ph.D.

We highly recommend learning about theories of presentation, perception, and data visualization, and we are big fans of Edward Tufte’s approach to visual explanations.  At the same time, we recommend learning the basics of using simple tools, like pie charts, bar charts, line graphs, and so on.  One good source for developing a mastery of the basics is a book called Graphing Statistics & Data: Creating Better Charts, from which we have learned a few tips about using bar charts:

• Use bar charts to show variables with distinct (non-continuous) values
• Bar charts are good at showing both proportion and quantity (unlike pie charts, which are good at showing proportions only)
• Adjust the chart settings so that the bars are wider than the gaps between the bars.  They need to be wide enough to invite visual comparisons among them, but not so wide that they resemble a histogram (see the examples below)
• If your variable has many values you want to show, or if labeling the values requires a lot of text, use a horizontal bar chart
• For horizontal bar charts, rank order the bars so that long bars (high values) are at the top of the chart, and short bars (low values) are at the bottom

This bar chart is flawed because the bars are too skinny relative to the spaces between them. They do not invite visual comparison.

The bar chart is flawed because the spaces between the bars are too narrow. It makes the chart look like a histogram, which is appropriate only for continuous variables.

This bar chart is just right.

As most of us know from having tried to disentangle one too many indecipherable charts in murky research reports, these basics seem easy, but they rarely are.  They require skill, art, and expertise developed through years of day-to-day efforts to understand, synthesize, and communicate data.  When done well, good charts tell the story of the data.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/visualizing-data-five-tips-to-using-a-bar-chart/feed/ 0 http://www.verstaresearch.com/blog/two-keys-to-writing-great-research-reports/ http://www.verstaresearch.com/blog/two-keys-to-writing-great-research-reports/#comments Thu, 25 Mar 2010 13:36:29 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=437 A truly effective research report is both parsimonious and richly nuanced.  In other words, (1) it is short and to the point, and (2) it captures the complexity of reality.  But how do you do both?

The importance of the first was highlighted in Sunday’s “Corner Office” interview in the New York Times business section.  Guy Kawasaki, co-founder of the Alltop news aggregation site, noted the importance of brief and pithy reports:

Q. What should business schools teach more of, or less of?

A. They should teach students how to communicate in five-sentence e-mails and with 10-slide PowerPoint presentations. If they just taught every student that, American business would be much better off.

Q. Why?

A. Because no one wants to read “War and Peace” e-mails. Who has the time? Ditto with 60 PowerPoint slides for a one-hour meeting.  What you learn in school is the opposite of what happens in the real world. In school, you’re always worried about minimums. You have to reach 20 pages or you have to have so many slides or whatever. Then you get out in the real world and you think, “I have to have a minimum of 20 pages and 50 slides.”

However, the importance of complexity was highlighted the next day in an article about Edward Tufte, a contemporary champion of presenting rich data via simple and compelling graphics.  Over-simplification can lead to bad decisions:

Mr. Tufte devotes a section of one of his books to explaining how clearer graphics could have persuaded NASA officials to postpone the takeoff [that resulted in the space shuttle Challenger disaster] because of cold weather. One of his conclusions is that presentations before the explosion, and even after, were too simplified. For simplicity, information was left out about the many missions during warmer weather that were uneventful. But the absence of that information meant that it was easy to overlook the larger pattern, that cold weather was dangerous to the O-ring.

In an interview, Mr. Tufte emphasized the need to enlist “the clarity of intense information.”

That’s the thing about transparency: you know it when you don’t see it. It some cases, it can mean more information. But other times, the reader can be overwhelmed by too much irrelevant information or, in one of Mr. Tufte’s favorite terms, “chartjunk.”

The goal in writing and presenting research should be to keep it brief and concise, but at the same time, communicate “intense” and relevant information.  A clear story will do this for you because good research stories are laden with meaning, complexity, and nuance, but they can be communicated succinctly, clearly, and with unambiguous implications.  Need help?  Versta Research specializes in rigorous research and communicating intense information (without all the chartjunk) to internal and external audiences.  We help you turn data into stories, and would be pleased to assist on your next research project.

—Joe Hopper, Ph.D.

As fancy charts and graphics become more accessible to everyday users, we think it is critical to revisit the basics of charts.  Many charts, even easy ones, are poorly conceptualized and poorly executed, which is even worse than showing your manager no chart at all.  It is critical to understand what kinds of charts best display different types of data and highlight specific kinds of relationships you are trying to show.

In this spirit, we offer you this handy chart, designed by Andrew Abela. It provides a starting point for choosing which type of chart will most effectively display your data based on the story you are trying to tell.

This diagram helps you select the right chart for your needs, including pie charts, bar charts, column graphs, bubble charts, histograms, and so on. You can click on the image for a full-size jpeg of the chart.

Of course this is just a starting point, but it highlights the idea that all charts are not the same.  Pie charts, bar charts, column charts, and bubble charts are not interchangeable.  If you would like some help thinking about your data and how best to present it powerfully with a chart or image, feel free to reach out to us.  We would be happy to help.

-Joe Hopper, Ph.D.

1. Data visualization and graphics will become more important.  The technology for amazing visualization tools is almost there, along with a growing realization that presenting data needs to be far more intuitive.
2. The value-add of insight and interpretation will become essential, not optional.  With new technologies, any monkey can now collect data, and half those monkeys can dump it into a tool that makes a chart.  But what does it all mean?  What’s the story?
3. Focus groups will become a thing of the past, in favor of qualitative ethnography and MROCs (Market Research Online Communities).  Usually focus groups are not the ideal way to answer the questions that clients have, but clients value hearing and seeing their customers.  New technologies have made better methods more accessible.
4. Data integration from multiple sources and areas within an organization will become a new imperative.  There are lots of data in lots of places.  One huge challenge over the next decade will be to bring multiple (and multiplying) sources of data together, connect the dots, and create intelligence.
5. Intellectual expertise to sort through, integrate, and interpret the explosion of data will become key.

Versta Research will be staying ahead of these trends over the next ten years to ensure that you get comprehensive, focused, and deep knowledge of your markets.  We offer help from high level experts who have rigorous training and who have the ability transform your data into stories that you can use.

-Joe Hopper, Ph.D.

Pie charts can be challenging because they are so common and so commonly misused.  When done poorly, they force us to think hard about what we are seeing and why.  When done well, they tell the story of our data.

Here are six tips on effectively designing and using pie charts:

• Pie charts are good for showing proportions, not quantities
• Use pie charts for category variables (like gender, or region) but not for variables that measure levels of things (like satisfaction)
• Pie charts should begin at 12 o’clock, and are read clockwise
• The maximum number of slices to the pie should be five or six
• Always show the numerical values on the chart
• Avoid three-dimensional “tilting” of the chart because it distorts true proportions

Programs that create charts automatically can be terrific tools, but they will never replace the need for you to think about your story and how to portray it in a chart.

–Joe Hopper, Ph.D.