It does not take super fancy (and expensive) mapping software or specialized firms to create accurate, useful, and compelling maps from market research data.  We recently created maps for a client showing where in a three-county region their best customers lived.  Everything we used to make these maps was free and publicly available for download on the Internet.  Here are the steps we used:

1.  Download shapefiles from the U.S. Census Bureau.  These files contain data to demarcate all legal and statistical geographical areas in the U.S. including states, counties, county subdivisions, census tracts, blocks, and so on.

2.  Edit the shapefiles with a program like QGIS.  There are several high quality, free, open-source software packages that you can use to read and manipulate census shapefiles.  We used QGIS, which is a program created and continually developed by the Open Source Geospatial Foundation.

3.  Link customer data to shapefile data in a spreadsheet program.  We looked at the number of customers in every zip code, then linked that data to county subdivisions in the shapefile by using a minimum distance function based on latitude and longitude coordinates.

4.  Plot the data and create the map using R.  R is quickly becoming the statistics package of choice in the academic world.  It is a free “integrated suite of software…for statistical computing and graphics” and can easily turn shapefiles and data linked to those shapefiles into visual displays.

Ultimately we created a heat map that displays customer location data for the three counties, which are divided into more than 50 townships, as shown in the map above, with darker colors signifying more customers than lighter colors.

As always, the ongoing challenge for researchers working with a burgeoning volume of data is how to interpret all that data, synthesize it, and simplify it into a story that is useful to decision makers.  Maps have always been a useful and compelling way to visually present data.  Finding the path to producing them from your data is now easier than ever.

–Joe Hopper, Ph.D.

Even if you are not involved in political polling, it is worth paying attention to the methods and best practices of political pollsters.  One reason is that few other areas of research offer a way to completely validate one’s methods.  Pollsters are using sampling and survey methods to predict the behaviors of a much larger population.  Then in just one day that population behaves, we get a near-perfect count of exactly how they behaved, and we know whether the methods worked.

Several industry colleagues have recently been debating the merits of calculating and reporting “margins of error” in political polling, and pointed us to some surprising data from The New York Times:

[The New York Times has compiled] a database consisting of thousands of primary and caucus polls dating back to the 1970s.  Each poll contains numbers for several candidates, so there are a total of about 17,000 observations. How often does a candidate’s actual vote total fall within the theoretical margin of error?  The answer is, not very often. In theory, a candidate’s actual vote total should fall outside the margin of error only 5 percent of the time [given that political polls report margins of error using a 95% confidence interval]. In reality, the candidate’s vote total was outside the margin of error 65 percent of the time! Part of this is because the database includes some polls conducted months before the actual voting took place. But even if you restrict the analysis to polls conducted within the final week of the campaign, about 40 percent of the vote totals fell outside the margin of error — eight times more often than is supposed to happen if you could take the margin of error at face value.

This does not mean that the polls were wrong, predicting wins for losing candidates and vice versa.  Rather, it means that the estimates were not as precise as the stated margins of error would have a reader believe.

The problem is that “margins of error” are based on a statistical theories that almost never line up with the messy reality of our world.  Margins of error make a number of assumptions which are rarely true in practice, including:

• Respondents are selected through simple random sampling

• All those sampled participate in the survey

• Sampling error is the only source of survey error

Indeed, Versta Research usually recommends to clients who publish survey research that they not report margins of error because the concept (and the calculations) are seriously misleading and flawed.

Calculating margins of error and looking at statistical significance should be used not because they give accurate or “scientific” predictions, but because they provide useful summary measures of how much variability there is in the data given the sample size and other critical factors that can affect one’s estimates.  At Versta Research, this helps us better interpret data and better assess what matters.  That, in turn, allows us to tell a story with the data that does not overreach or misrepresent what is going on.

–Joe Hopper, Ph.D.

Here we offer the “Versta Digest” version as a handy reference card.  Once you get the hang of it, you don’t need the examples and explanation.  You just need to know the rules.  We recommend reading the full article first, so you know what we’re talking about when it comes to “points.”  Then, when you need a refresher or a reference source, consult these rules:

• One point for each simple question or scaled response
• One point for every two response options in a multiple choice question
• One point for each row in a grid question
• Two points for any response that requires mental calculation
• Three points for every short response to an open-ended question
• One point for every three sentences of extra text that respondents must read

Then:

• Tally up the points
• Divide by 8 for online survey length (in minutes)
• Divide by 8 then multiply by 1.5 for phone survey length (in minutes)

The system is straightforward, easy to learn, and easy to execute.  It is a method that really works and that we have validated against hundreds of different types of surveys over the past several years.

Once you know how to determine survey length, you can think more strategically about the ideal survey length to optimize the value and content of a survey within your budget.  Call us at 312-348-6089 with any additional assistance you may need.

–Joe Hopper, Ph.D.

Spreadsheets have amazingly powerful and often intuitive capabilities.  They have many of the statistical functions we use every day.  Younger people entering our profession rarely know programs like SPSS or SAS, and we see them turning to Excel to generate frequencies, calculate means and proportions, create charts from data, and so on.  The same goes for our customers.  Many do not have statistical software, so when they need numbers and statistics, they often work in Excel.

But Versta Research continues to invest in advanced statistical software rather than doing our work in Excel for three important reasons:

1. Speed and efficiency.  The tools we use are designed to do exactly what we need.  Spreadsheets require more effort to manipulate data, set logic rules, and write formulas that we can otherwise do with just a few clicks.

2.  Leveraging analytical innovation.  Our statistical software leverages the newest techniques in developing areas of statistical theory and applications, especially from software developers like Sawtooth who are pioneers in choice modeling.

3.  Accuracy.  As anyone who has created a moderately complex spreadsheet knows, it is frighteningly easy to make errors in spreadsheets (data errors, sorting errors, formula errors, copy-and-paste errors, cell reference errors, and the list goes on) and it is often difficult to find, detect, and untangle those errors, if indeed they are ever found.

To be sure, Excel is a powerful tool that we use all the time and every day, in part because it can be used in so many creative and flexible ways.  We use it help us track, manipulate, and visualize statistical output, for example.  We also use it as an efficient way to write multiple lines of programming script that we then paste into our statistical programs.  But when it comes to the core of our statistical analysis, we rely on the best-in-breed software tools that continue to outpace the capabilities of a spreadsheet.

–Joe Hopper, Ph.D.

But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

–Joe Hopper, Ph.D.

Suppose you want to measure the importance of 12 attributes for a new product or service.  If you know ahead of time that consumers are going to say that all 12 are extremely important to them, then MaxDiff is an excellent method for differentiating among the attributes so you can focus on the top two or three that matter most.

But what if you don’t know that all 12 attributes are extremely important?  Maybe none of them are.  Maybe they run the gamut from unimportant to extremely important.  The problem with MaxDiff is that it only tells you the importance of attributes relative to each other, but it won’t tell you whether the attributes are important.  The MaxDiff model will assign ratio-level numbers so that you can rank and quantify the importance of each attribute vis-à-vis the others.  But it will not anchor the attributes in a meaningful way.

This week we are designing a study in which we want to differentiate among attributes, but we also want to measure the gap between satisfaction and importance for items that are truly important to our target market.  We cannot do that with data from a typical MaxDiff study.  So we are using an old fashioned importance rating scale instead.

As always, it is critical to think about the story you want to tell with your research data, and then work backwards to the design and the choice of methods.  In many cases MaxDiff is the perfect tool.  In other cases it will leave you with data that is difficult to apply to critical questions.

Feel free to give us a call if you need some help deciding among the best methods for your research, whether it be MaxDiff, other conjoint techniques, or something else entirely.  We’ll help you focus on the story you need to tell and on the research design you need to tell it.

–Joe Hopper, Ph.D.

One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits.  If these respondents are not providing thoughtful answers, the data are suspect.  All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.

But we should not rely on panel providers alone to ensure valid data.  Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases.  Here are typical indicators of potentially bad data:

1.  Speeding. Though people can legitimately whiz through surveys at varying speeds, we typically flag the fastest five percent for further investigation.

2.  Non-sense open ends.  People who have nothing to say will usually say that, so we flag respondents who type random letters, offer non-sense or vacuous answers, or skip answering entirely.

3.  Choosing all options on a screening question.  Often it  means the respondent was gaming the survey to get in, especially if some options logically exclude others.

4.  Failing quality check questions.  Usually we include a couple of questions that have only one correct response to flag respondents who are not paying attention.

5.  Inconsistent numeric values.  How long a person has worked in a profession or at a particular job, for example, must be consistent with a person’s age.

6.  Straight-lining and patterning.  If questions are laid out in grids, respondents who answer identically for all questions, or who move in a diagonal along the grid should be flagged for investigation.

7.  Logically inconsistent answers.  If attitude and behavior questions are logically related to each other (for example, multiple questions about concern for the environment), inconsistent responses may indicate bad data.

The customer who reminded us that online surveys face multiple hurdles had just gotten results from a survey that she discovered had included a respondent who took the survey 250 times.  Nobody from the research firm bothered to look at the data beyond feeding it into the data-tabulator-chart-maker-here-are-your-actionable-insights machine.

At Versta Research, our approach is the opposite.  Smart people look at your data at each step because there is no other way to turn data into a story that you can trust and then share with your management team with confidence.

–Joe Hopper, Ph.D.

Two researchers at the U.S. Census Bureau recently outlined an emerging innovation in survey research that could reverse the trend towards passive, boring, self-administered surveys that characterizes much online research.  The idea is to use internet avatars in real-time interviewing with survey respondents.

Beyond just the heightened interest of having an animated survey, the avatars would be programmed to register and interpret respondents’ verbal answers, facial expressions, and body language through webcams.

Suppose, for example, that a respondent answers a question with detailed information that answers a follow-up question as well.  The avatar would use natural language processing to insert that data into the subsequent question, and then avoid asking the follow-up.  Or if the respondent looks away from the screen and pauses for time longer than is typical, the avatar can offer a rephrased question or a reassuring comment to re-engage the participant and to put him or her at ease.  This type of innovation  could bring many of the advantages of live interviewing back into the realm of internet surveys, which are far more efficient in terms of time and cost.

The use of effective avatar interviewers is at least several years away, however, because it involves not only evolving internet technologies, but also advanced linguistic processing, facial and voice recognition technologies, and so on.  In fact, the sheer technological difficulty of truly replacing human interviewers reminds us of how absurd it is for research companies to make claims about technology replacing higher-order activities in the research process, such as providing analysis and insight.

At least for now, software and services with “actionable insight” buttons generate yet more mountains of data in need of human synthesis and interpretation.  If anything, the role for smart and experienced researchers who can turn all that data into a story is growing.  It is growing for researchers who work on the client side and who have direct accountability to the executives who need data-driven insights.  And it is growing for firms like Versta Research where the highest levels of intellectual and human capital are central to our work.

–Joe Hopper, Ph.D.

One way to analyze the data is to use a simple counting technique.  It works pretty well, by the way.  But the more common way these days is to use advanced statistical modeling that allows for stronger individual-level estimates of importance scores.  This affords better opportunities for segmenting the data and analyzing subgroups.

We typically see analysts transforming the importance scores (or utilities) to a 0—100 scale.  It is a compelling scale because all importance scores add up to 100, which mimics the technique of having respondents allocate 100 importance points to each of the features they care about.

But if you do a lot of MaxDiff studies and frequently present them to your management team, you lose the ability to provide a rule of thumb as to what counts as a “high” score vs. a “low” score.  If one MaxDiff exercise uses fifteen attributes, an importance score of 20 is quite high; if another MaxDiff exercise uses just five attributes, and importance score of 20 is just average.

The solution?  Instead, we typically transform scores to a 0—100*N scale, where N=the number of attributes tested.  If you tested fifteen attributes, transform to a 0—1500 scale.  If you tested five attributes, transform to a 0—500 scale.  With this method, a score of 100 is always the average, scores above 100 are always above average, and scores below 100 are always below average.  Sometimes we transform this into a chart showing “percent above/below  average” as shown in the third example below, but this is not always necessary because anchoring the average at 100 makes it simple to calculate those percentages mentally.

Example 1: Versta's Recommended Scaling of MaxDiff Scores

Example 2: The Typical Way of Scaling MaxDiff Scores

Example 3: Another (Sometimes Useful) Way to Scale MaxDiff Scores

Need help with the smartest ways to design, implement, and report MaxDiff studies?  Call us at (312) 348-6089 for assistance and further information.

–Joe Hopper, Ph.D.

It's pretty, but it's chartjunk

Turning data into stories involves not just words, but pictures as well.  In the world of quantitative market research, that usually means charts, graphs, and tables.  Moreover, just like poorly written sentences that often complicate rather than clarify data, charts and graphs in market research too often suffer from “chartjunk,” as Edward Tufte calls it.  Any superfluous details, design elements, or decorations that do not tell the viewer something new about the data are chartjunk.

At Versta Research we write a lot of reports.  We also revise others’ reports to help our clients find and more clearly present research stories to their management teams.  Here are three of the more common chart design mistakes we see and help our clients avoid:

1.  3-D Charts.  Few of us in market research work in multidimensional spaces, so 3-D charts have no purpose other than to “Bring more creativity to your presentations!” or “Lift your charts above the ordinary!”  In fact, 3-D charts nearly always distort proportions and make it more difficult to compare and contrast relevant data.  For the most part, we keep our charts in flatland.

2.  Grid Lines.  For some reason PowerPoint includes gridlines by default.  But gridlines are rarely needed, and usually they are distracting.  Typically we label all data points, so gridlines that pull your eyes to the axes are superfluous.  That said, when gridlines are useful, we make them light gray so that the data stands out and the grid recedes to the background.

3. Irrelevant Data. The best charts pack amazingly large amounts of data, but in elegant ways that never overwhelm with irrelevant information.  The problem with research data is that we always have more data we could put into a chart, so the key is to figure out which data helps tell the story.  For example, if just 4% of customers express dissatisfaction, there is no reason to show details down to the level of “somewhat dissatisfied” versus “very dissatisfied” versus “extremely dissatisfied.”

Data charts are easy to generate nowadays, perhaps too easy.  Too many charts (and dashboards, and “actionable” report generators) are now data dumps that fail to tell a story any more than the raw data that was dumped into them.

To get your research understood, used, and promoted by your management team, it needs to tell a story.  That requires a thoughtful, deliberate approach whether by words or by pictures.

–Joe Hopper, Ph.D.