I recently saw a press release about a study showing that only 19.5% of news release headlines are optimized for SEO.  It brought to mind all kinds of issues about how best to report numbers in press releases.  In particular it highlighted the important issue of whether specific numbers are meaningful  and whether they communicate a misleading sense of precision.

For example, when a survey reports a margin of error to any decimal place, it suggests a level of precision that is misleading.  Do a quick search, and you’ll find press releases reporting margins of sampling error such as +/- 4.8%, +/- 10.5%, or +/- 1.85%.  These numbers are based on sample size formulas that assume perfect random sampling and one hundred percent response rates, which are almost never achieved.

Moreover, these overly-precise numbers convey a misleading impression about how good a study’s findings are overall.  How many readers understand the difference between a margin of sampling error vs. the “overall” margin of error which is what most readers actually care about?  Not many.  As the American Association of Public Opinion Polling (AAPOR) correctly notes:

Poll results are subject to lots of sources of errors ranging from how well the questions were designed and asked to how well the interview was conducted to how well the sample design was implemented.  Good pollsters and researchers do everything in their power to minimize these other possible sources of errors, but they are non-measurable in any case, and one can never know the precise amount of error associated with any poll finding.

Indeed, there are so many conceptual and practical difficulties with reporting margins of error, that we, along with other major polling firms, typically recommend not reporting them.

But what if we could accurately report numbers, margins of error, and findings down the tenths or hundredths of a percent?  (And maybe they can in the SEO and headlines study—hard to say without knowing more about their methods.)  Is a tenth of a percent a meaningful level of precision worth reporting?  Does it matter that it 19.5% of headlines are optimized versus just 18.8%?  Probably not.  Better to just say 19%, which is likely to be a more credible statistic because it does not pretend to be more meaningful or precise than it really is.

–Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

Conventional wisdom is that your current customers are your best customers.  Indeed, when marketers focus on cross-selling as a strategy, they typically see substantial increases in revenue and profit.  Not only that, but when you look only at customers who cross-buy a lot, the profit from these customers is huge compared to customers who do not cross-buy or who cross-buy just a little.

But it turns out that hidden within these cross-buying customers are segments that are actually a significant drain on profits.  A study just published in the Journal of Marketing found that up to one-third of cross-buying customers are not profitable at all, and in fact, may account for the bulk of a company’s losses attributable to customers.

Who are these customers?  They are the ones who:

• Consistently ask for and use customer services
• Return products, or default on loans
• Change their “old” spending so there is little net gain in the cross-sell
• Consistently focus on discounts and promotions

The study is based on data from both B2B and B2C firms, and from a range of industries, including financial services, IT, and retail.

So instead of just combing your data and developing algorithms for identifying which customers are most likely to buy which products and at what time, the smart marketer first identifies two segments of customers based on key behavioral traits.   As the authors summarize:

“It is not prudent to cross-sell a product to every customer who is likely to buy an additional product.  This is a significant shift from conventional marketing practices that emphasize cross-selling to all customers.”

Just a little bit of segmentation and research know-how can significantly boost the effectiveness and profitability of your cross-sell marketing strategy.  If you need help thinking about how to apply these findings to your own efforts, feel free to give us a call at (312) 780-0245.  We are happy to advise.

–Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

How do you make a beautiful, elegant, intuitive, and useful chart showing changes in market share over time?  In R.  That always seems to be the answer these days when it comes to data visualization as well as data analysis.  It is the reason that we at Versta Research are in the midst of an intensive course of training and retraining in R.  Here is just one beautiful example of what R can do.

This first chart we created the old-fashioned way, using PowerPoint.  It shows membership in interest groups over a ten year period within the Chicago AMA (American Marketing Association).  Of particular interest to us was the sharp decline in Market Research.  The other interest groups get tangled up and are hard to read.

This market share chart was created in PowerPoint

In this second chart, we took the exact same matrix of data, and created a chart using the ggplot function in the ggplot2 package for R.

This market share chart was created in R

It is a far better visualization of market share changes over time.  Guaranteed, nobody in your boardroom will be squinting at data points trying to untangle what this means.

By the way, R is open source and free.  It is quickly becoming the statistical computing software of choice across a range of disciplines, including business, finance, healthcare, and so on.  The learning curve is steep, but also a reminder of why it is critical for market research firms to continually invest in learning and development.  Clients depend on us to help them stay at the forefront of market research methods and knowledge.

–Joe Hopper, Ph.D.

Most people who take surveys want to share their opinions, which is important for researchers hoping to get a few nuggets of data from willing respondents.  The trouble is, if a survey it not written carefully, a respondent’s urgent desire to share their feelings may bias their answers to other questions.

Two marketing professors at Northwestern University’s Kellogg School of Management recently published a paper in the AMA’s Journal of Marketing Research identifying and documenting this unique kind of bias.  They call it “response substitution.”

Suppose you eat at a fancy restaurant, pay a lot of money, have delicious food, but suffer horrid and neglectful service.  In your eagerness to rant about your bad experience, you are likely to give poor ratings to the quality of food.  Poor chef!  He thinks his food is terrible.  The waiter did a lousy job, so everyone suffers.  But wait.  It turns out that if you are told before taking the customer satisfaction survey that you will have an opportunity to share all comments and opinions at the end of the survey, you will rate the food higher, knowing you can vent about the waiter later on.

“Response substitution” is an important source of bias, with an easy solution.  The lessons for researchers are this:

1.  Know when you are likely to encounter response substitution bias. Situations where people are likely to have strong opinions they want to share (satisfaction with personal experiences, politically polarizing issues, etc.) are where you should take measures to reduce response substitution bias.

2.  Protect against it with a short introduction to respondents. Tell them up front that their opinions are valued and they will be able to share all opinions in the survey.  Remind them to focus on the specific issues being asked about.  Be sure that you do give them an opportunity to answer unasked questions in open-ended format somewhere in the survey.

–Joe Hopper, Ph.D.

Every ten years the Census Bureau provides a count of all people living in the United States.  More importantly, the Census Bureau and other agencies conduct ongoing surveys of the population to document crucial facts and figures about who we are.

And every year, it seems, these agencies come under attack from politicians who would like to do away with all public goods.  This year it is the American Community Survey that is under attack, as described in this article from The Wall Street Journal.  Prominent professional and business groups (including AAPOR, The American Association of Public Opinion Research, of which we are members) of all political and non-political persuasions are working to fend off the attack.

Here are two reasons why you and your company should aggressively support our efforts to protect the Bureau’s data collection efforts:

1.  The data are amazingly useful and important. We regularly consult and work with this data for almost every client who hires us.  It tells us how many people, of what ages and incomes, live in what counties and zip codes.  It gives us information about their family composition, so if we need to help you understand a market of working moms with kids, we know how to do this efficiently.  It gives us solid information about peoples’ health and financial well-being, which all of our customers care about for all kinds of reasons.  It provides critical statistical benchmarks for understanding your data and analyzing it correctly.

2.  The Census Bureau leads important innovations. Few market research firms (and few academics) do research as methodologically rigorous and as statistically robust as the Census Bureau.  It is difficult, costly, and time-consuming.  Research of this caliber is not usually an investment our customers want (or should) make.  But we and they benefit enormously from the development of methods that become the gold standard for social and market research.  Currently one of the most important innovations underway is the study and development of statistical techniques to transform large datasets that preserve data structure while stripping away the ability to identify individual cases.  As these innovations make their way into the private sector, every business will benefit from techniques that help them protect the confidentiality of their data and that of their customers.

Census and other government agency data are one of the most outstanding and useful public goods available for businesses. They allow you to function and innovate in informed ways, and to deliver goods and services to customers who need them.  They are as important as roads and bridges.

Do what you can to protect this public good, and to ensure a continuing public investment in a resource that benefits you.

–Joe Hopper, Ph.D.

How big of a sample size do you really need? A recent article in the New York Times cited the following statistics:

• A small Voice of the Customer (VoC) research company called Mindshare Technologies collects satisfaction data from 175,000 respondents every day. That’s 60 million in a year.
• ForeSee, a small customer experience analytics firm fielded 15 million surveys in 2011.

These numbers are believable. I get a pop-up survey from ForeSee at least two or three times a week.

And it is absurd. Granted, these companies (and hundreds of other similar firms) are collecting surveys for multiple clients. But almost certainly, nobody needs to collect that much survey data from that many survey respondents. Why not?

1. Huge sample sizes do not improve data precision. While there are significant gains in margin of error by increasing sample sizes at the low end of the spectrum, once you get beyond a few thousand, you gain almost nothing.

2. The cost of analyzing so much data is huge. No matter what technology can do, it can’t make crucial decisions about how to assess whether data are valid and how to fix biases in data, nor can it report on them in meaningful ways that go beyond the simplest of queries. Likewise, big computers can mine lots of data, but all that mining requires smart (and expensive) people programming, analyzing, and reporting on it. It’s the reason that firms have so much data with no idea what it might be telling them.

3. It poisons the well for future surveys. People want to share their opinions and are generous in giving their time for meaningful surveys, but every annoying pop-up survey and each invitation on every cash register receipt teaches another potentially generous respondent to say “No, so please stop asking me.” And sure enough, response rates for surveys have plummeted in the last 10 to 15 years.

There is a smart way to field surveys and to collect data and analyze it. Take only what you need. Think hard and be careful about getting exactly what you need. Make continual adjustments along the way. Be happy (delighted, in fact) with your sample size of 800 because chances are it means somebody thoughtfully designed and fielded your survey, and gave you far deeper insight than a blast of 60 million surveys will ever deliver.

–Joe Hopper, Ph.D.

Here are our top picks for brief summer courses that will keep you and your colleagues at the cutting edge of research methods:

Exploratory Data Mining, offered at the University of California in Davis by the American Psychological Association — a five-day course that will cover the conceptual bases and strategies of exploratory data mining, and will review leading current techniques and software.

Social Network Analysis, offered by the University of Buffalo’s Department of Sociology — a three-day class that will teach theoretical and methodological skills to conduct studies using social network analysis. The course will focus on different network theories, their corresponding measures, and how to get the data to conduct a study.

Data Analysis with R, offered by the University of Kansas’ Center for Research Methods and Data Analysis — a five-day course on data management, statistical computing, modeling and graphics in R, which is destined to become the dominant statistical software package within the next decade.

Introduction to Bayesian Statistical Modeling, a three-day course offered at the University of Maryland — intended as both a theoretical and practical introduction to Bayesian statistics.

Mixed Methods: Approaches for Combining Qualitative and Quantitative Research Strategies, offered in Chapel Hill by the Inter-University Consortium for Political and Social Research (ICPSR) — a three day workshop that will highlight the main elements of a research approach that collects, analyzes, and integrates both qualitative and quantitative data within a single study.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/five-classes-to-make-you-smarter/feed/ 0 http://www.verstaresearch.com/blog/what-statisticians-really-do/ http://www.verstaresearch.com/blog/what-statisticians-really-do/#comments Wed, 28 Mar 2012 20:07:27 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=2007 We came across these images in a series of humorous montages that professionals had created about what they do.  This one was created by Jason Sullivan:

Each of these rings true not only because it represents how quantitative researchers are seen by different audiences, but also because there is a grain of truth in each as well.  Statistics involves complicated math, a bit of art mixed in with the science, lots of computing, and discerning reality behind the numbers.  Regrettably, too, some market research may stretch the truth.

But most unfortunate of all is the BIG grain of truth in the final image.  It is a boring bar chart, which truly reflects what most market research produces nowadays.

It doesn’t have to be that way.  Behind every bar chart or every data table is a story.  The challenge for researchers and statisticians is to tell that story.  All it takes is bit of experience, brainpower, and thoughtful effort to link the data to questions that need to be answered.  How?  Here are two articles we highly recommend:

Turning Data Into Stories

Turning Data Into Stories: A How-To Guide

If you need statisticians who will move beyond data charts and help you communicate the story in your data, call us for help.

–Joe Hopper, Ph.D.

1.  The pieces of the doughnut are not correctly proportional.  For some reason, the arc widths were compressed for some categories (like Media & Film Creation) distorting the true size of the relationships among categories.

2.  Information is squeezed onto the chart in such a way that some of it is unreadable.  Even if you click on the chart to get a full-sized image, it is difficult to discern the numbers for the Media & Film Creation category.

3.  Percentages are shown to two decimal places, which is unimportant.  Why add extra numbers if they don’t convey extra information?

A better way to create the chart is shown below.  It is less snazzy.  But it shows the important data (the relative sizes of each expense category) accurately.

Even so, this chart is probably not the best choice, either.  Pie charts are excellent for showing relative proportions of the whole, but once you have more than four our five slices, it is usually too much information for this type of chart, and your brain does not really process those relative proportions.

Here’s our recommendation for the best chart.  We lose the advantage of a pie chart which automatically conveys the idea that each component is a non-overlapping part of the whole.  But all the information is easy to read and compare, and all proportions are immediately and visually accessible.

Visualizing data with simple and effective charts is one critical piece of telling a good story.  Versta Research can help you with that.

–Joe Hopper, Ph.D.

Coincidentally in the same week that Versta Research published its winter newsletter on Turning Data into Stories: A How-To Guide, last week’s AMA event in Chicago was a market research panel focused on telling stories with data.  The presentations were solid, with lots of helpful ideas.  But there was also a misguided idea working its way through the room, worthy of spirited debate, if only we had more time.  It is the idea, as one panelist put it, that “clients don’t care about numbers.”

Of course his point was that too often researchers build reports and presentations from reams of data, showing table after table of numbers, and chart after chart of detail.  They fall short when it comes to interpreting the data, saying what it means, and how it should be used.  Ultimately managers and clients care about what the numbers mean and what they can do with them.  We agree 100%.

But in our experience, clients also care a great deal about numbers.  They nearly always want to know how many, how much, and how big.  They want things quantified, and even better, they want numbers visualized in charts or in other compelling ways.  Indeed, the finance people love numbers; the marketing people love numbers;  the PR people love numbers.

What customers don’t love, at least not so much, is data.  The data has to be there and it has to be solid, but it needs to be turned into a story.  The key is always to provide meaningful numbers, and to avoid diluting the story with data that overwhelms.

Here are three quick recommendations (described more fully in our article on this topic):

1.  Carefully review how the data are calculated. It is important to simply determine the base against which percentages are calculated because the specific numbers you share need to tie directly to the story you want to tell.

2.  Find the simplest numbers, and cut everything else. Think ahead in terms of which data points will make it into the CEO’s rationale for a strategy, or into a press release highlighting the research findings.

3.  Write about what the numbers point to.  Nearly every number in a report should point to something that real people do in the real world. Write about that, not about the numbers themselves.

We also recommend being a part of the conversation at events and professional meetings, like the event we helped organize for the American Marketing Association.  Our presentations last week brought together researchers from Allstate, Zurich Insurance, Tribune Company, the American Dental Association, Rockwell Automation, and market researchers from some of the world’s largest MR companies.  We anticipated a small gathering of about 20 AMA members.  Fifty four showed up.  Not bad, as far as numbers go!

–Joe Hopper, Ph.D.