Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/focus-groups-save-spider-man/ http://www.verstaresearch.com/blog/focus-groups-save-spider-man/#comments Wed, 11 Jan 2012 21:46:57 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1869

In last year’s cliffhanger episode of “Can a Focus Group Save Spider-Man?” we pondered whether market research was powerful enough to save a Broadway show from doom and destruction.  After crushing reviews from theater critics, the producers hired a market research firm to help them rewrite the show.

Guess what?  It worked.  Since the show re-opened in June, it is regularly among the top five earners among Broadway shows, and The Hollywood Reporter reports that in December it “set a new record for a single-week box office gross, raking in $2,941,794 for the week ending January 1.”

In a year when Steve Jobs’ quip that “it’s not the consumers’ job to know what they want” has been used to bludgeon market research, we see research doing a pretty good job identifying buyers’ needs and helping decision makers address those needs in smarter ways.

It is true that consumers can’t answer questions like, “What show do want us to produce next?” or “What great technological innovation would be most useful to you?”  But they certainly can tell you want they want, and the art of research is to ask just the right questions to give you enough insight about what to do next.  Here are three things you can ask about, and that audiences, customers, and prospects will happily tell you:

1.  What is important to them and what they care about.  Insights about what matters to buyers will help you design your product or service and will provide deep insight about how to market it.

2.  What their frustrations are and what is not working.  Most consumers are eager to critique products and services that fail to meet their needs, which highlights the opportunities for new solutions.

3.  How good or bad your idea is.  There are lots of ways to test concepts, products, and ideas, and survey research is remarkably good at predicting winners versus losers.

The lesson from Spider-Man is that it does not take superhuman powers to fix a flawed strategy nor off-the-charts creative genius to make a good product sell.  A thoughtful effort to ask questions and to listen to your customers is sometimes all it takes to turn harrowing encounters with goblins into success.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/focus-groups-save-spider-man/feed/ 0 http://www.verstaresearch.com/blog/a-path-to-better-research-with-geo-maps/ http://www.verstaresearch.com/blog/a-path-to-better-research-with-geo-maps/#comments Wed, 04 Jan 2012 22:55:11 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1855 Given how common mapping capabilities have become via the Internet and smartphones, it is surprising that we don’t see more geographic mapping in market research.  Researchers nearly always look at customer demographics, and a key component of a person’s demographic profile is where he or she lives.  This data is far more compelling if you can present it visually with maps.

It does not take super fancy (and expensive) mapping software or specialized firms to create accurate, useful, and compelling maps from market research data.  We recently created maps for a client showing where in a three-county region their best customers lived.  Everything we used to make these maps was free and publicly available for download on the Internet.  Here are the steps we used:

1.  Download shapefiles from the U.S. Census Bureau.  These files contain data to demarcate all legal and statistical geographical areas in the U.S. including states, counties, county subdivisions, census tracts, blocks, and so on.

2.  Edit the shapefiles with a program like QGIS.  There are several high quality, free, open-source software packages that you can use to read and manipulate census shapefiles.  We used QGIS, which is a program created and continually developed by the Open Source Geospatial Foundation.

3.  Link customer data to shapefile data in a spreadsheet program.  We looked at the number of customers in every zip code, then linked that data to county subdivisions in the shapefile by using a minimum distance function based on latitude and longitude coordinates.

4.  Plot the data and create the map using R.  R is quickly becoming the statistics package of choice in the academic world.  It is a free “integrated suite of software…for statistical computing and graphics” and can easily turn shapefiles and data linked to those shapefiles into visual displays.

Ultimately we created a heat map that displays customer location data for the three counties, which are divided into more than 50 townships, as shown in the map above, with darker colors signifying more customers than lighter colors.

As always, the ongoing challenge for researchers working with a burgeoning volume of data is how to interpret all that data, synthesize it, and simplify it into a story that is useful to decision makers.  Maps have always been a useful and compelling way to visually present data.  Finding the path to producing them from your data is now easier than ever.

–Joe Hopper, Ph.D.

Even if you are not involved in political polling, it is worth paying attention to the methods and best practices of political pollsters.  One reason is that few other areas of research offer a way to completely validate one’s methods.  Pollsters are using sampling and survey methods to predict the behaviors of a much larger population.  Then in just one day that population behaves, we get a near-perfect count of exactly how they behaved, and we know whether the methods worked.

Several industry colleagues have recently been debating the merits of calculating and reporting “margins of error” in political polling, and pointed us to some surprising data from The New York Times:

[The New York Times has compiled] a database consisting of thousands of primary and caucus polls dating back to the 1970s.  Each poll contains numbers for several candidates, so there are a total of about 17,000 observations. How often does a candidate’s actual vote total fall within the theoretical margin of error?  The answer is, not very often. In theory, a candidate’s actual vote total should fall outside the margin of error only 5 percent of the time [given that political polls report margins of error using a 95% confidence interval]. In reality, the candidate’s vote total was outside the margin of error 65 percent of the time! Part of this is because the database includes some polls conducted months before the actual voting took place. But even if you restrict the analysis to polls conducted within the final week of the campaign, about 40 percent of the vote totals fell outside the margin of error — eight times more often than is supposed to happen if you could take the margin of error at face value.

This does not mean that the polls were wrong, predicting wins for losing candidates and vice versa.  Rather, it means that the estimates were not as precise as the stated margins of error would have a reader believe.

The problem is that “margins of error” are based on a statistical theories that almost never line up with the messy reality of our world.  Margins of error make a number of assumptions which are rarely true in practice, including:

• Respondents are selected through simple random sampling

• All those sampled participate in the survey

• Sampling error is the only source of survey error

Indeed, Versta Research usually recommends to clients who publish survey research that they not report margins of error because the concept (and the calculations) are seriously misleading and flawed.

Calculating margins of error and looking at statistical significance should be used not because they give accurate or “scientific” predictions, but because they provide useful summary measures of how much variability there is in the data given the sample size and other critical factors that can affect one’s estimates.  At Versta Research, this helps us better interpret data and better assess what matters.  That, in turn, allows us to tell a story with the data that does not overreach or misrepresent what is going on.

–Joe Hopper, Ph.D.

We recently received and reviewed an excellent book summarizing practical findings from academic marketing research.  It is called Consumer Insights: Findings from Behavioral Research. It is published by the Marketing Science Institute and we highly recommend it.  Why is it so good and useful for corporate researchers and marketers?

• It provides a quick overview of what we know in various areas of marketing
• It is organized into 42 useful topics, most of which are relevant to nearly all marketers
• Each topic provides universal findings based on research from hundreds of studies, not just one or two
• Each topic/chapter is short and to the point (just two or three pages)
• Each topic outlines insights, the evidence base, and managerial implications

You will find important applications to almost any marketing effort you are engaged in.  For example, here is one that applies to what we are doing right now:  “Consumers are becoming more and more suspicious of online bloggers touting products.  Thus, a blogger’s attempt to tout a product may actually backfire and reduce consumer trust in both the blogger and the product.”  So to allay concerns, it is important to note that we have no connection to the authors, publishers, or sellers of this book!  Nor was the book pitched to us, or provided to us free (alas, we paid full price).

Here are the topics covered in the book:

Information Search
1. Effects of Product Knowledge on information Search
2. In-store Decision Making and Unplanned Purchases
3. Perceptions of Product Assortment
4. Variety-seeking Behavior
5. Consumer Search on the Internet
6. Buyers’ Post-purchase Information Biases Pricing
7. Perception of Price Deals
8. Biases in Processing Price Information
9. Effects of the Internet on Consumer Price Sensitivity
10. Effects of Transaction Structure on Price Perceptions & Consumption
11. Perceptions of Price Fairness

Advertising
12. Consumer Attention to Advertising
13. Effects of Ad Likability

Brand Effects
14. Consumer Brand Recall
15. Perceptions of Brand Extensions
16. Brand Dilution and Protection

Consumer Inferences
17. Consumer Inferences and Assumptions
18. Perceptions of Quality Signals
19. Causal Inferences and Consumers’ Judgments
20. Consumer Use of Persuasion Knowledge

Feelings, Attitudes, and Persuasion
21. Effects of Mere Exposure on Brand Liking
22. Influence of Feelings & Emotions on Consumers’ Judgments
23. Persuasion: Elaboration Likelihood Model
24. Consumer Goal Orientation
25. Consumer Attitudes Toward Marketing

Decision Making and Purchase
26. Pre-choice Bias in Brand Choice
27. Consumers’ Intertemporal Preferences
28. Loss Aversion and Consumer Choice
29. Protected Values
30. Purchase Intentions and Purchasing
31. Consumer Habits and Purchase Behavior
32. Impulsive and Compulsive Buying

The Social Consumer
33. Social Contagion and Word-of-mouth
34. Consumer Identity and Purchase Behavior
35. Perceptions of Advisors

Vulnerable Consumers
36. Children and Advertising
37. Aging Consumers
38. Effects of Low Literacy on Consumer Decision Making

Health and Well-being
39. Effects of Nutrition Information and Health Claims on Consumption
40. Mass-media Campaigns and Health-related Behaviors
41. Perceptions of Health Risks
42. Effects of Portion/Package Size and Shape on Consumption

The book is excellent and useful.  Get it.  It is now available through Amazon at a substantially lower price than we paid.

We learned many things we did not know, and we learned that some of the things we have come to know about marketing through our own experiences now have a wider research base to back them up.

Happy insights, from Versta Research!

–Joe Hopper, Ph.D.

Spreadsheets have amazingly powerful and often intuitive capabilities.  They have many of the statistical functions we use every day.  Younger people entering our profession rarely know programs like SPSS or SAS, and we see them turning to Excel to generate frequencies, calculate means and proportions, create charts from data, and so on.  The same goes for our customers.  Many do not have statistical software, so when they need numbers and statistics, they often work in Excel.

But Versta Research continues to invest in advanced statistical software rather than doing our work in Excel for three important reasons:

1. Speed and efficiency.  The tools we use are designed to do exactly what we need.  Spreadsheets require more effort to manipulate data, set logic rules, and write formulas that we can otherwise do with just a few clicks.

2.  Leveraging analytical innovation.  Our statistical software leverages the newest techniques in developing areas of statistical theory and applications, especially from software developers like Sawtooth who are pioneers in choice modeling.

3.  Accuracy.  As anyone who has created a moderately complex spreadsheet knows, it is frighteningly easy to make errors in spreadsheets (data errors, sorting errors, formula errors, copy-and-paste errors, cell reference errors, and the list goes on) and it is often difficult to find, detect, and untangle those errors, if indeed they are ever found.

To be sure, Excel is a powerful tool that we use all the time and every day, in part because it can be used in so many creative and flexible ways.  We use it help us track, manipulate, and visualize statistical output, for example.  We also use it as an efficient way to write multiple lines of programming script that we then paste into our statistical programs.  But when it comes to the core of our statistical analysis, we rely on the best-in-breed software tools that continue to outpace the capabilities of a spreadsheet.

–Joe Hopper, Ph.D.

But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

–Joe Hopper, Ph.D.

Suppose you want to measure the importance of 12 attributes for a new product or service.  If you know ahead of time that consumers are going to say that all 12 are extremely important to them, then MaxDiff is an excellent method for differentiating among the attributes so you can focus on the top two or three that matter most.

But what if you don’t know that all 12 attributes are extremely important?  Maybe none of them are.  Maybe they run the gamut from unimportant to extremely important.  The problem with MaxDiff is that it only tells you the importance of attributes relative to each other, but it won’t tell you whether the attributes are important.  The MaxDiff model will assign ratio-level numbers so that you can rank and quantify the importance of each attribute vis-à-vis the others.  But it will not anchor the attributes in a meaningful way.

This week we are designing a study in which we want to differentiate among attributes, but we also want to measure the gap between satisfaction and importance for items that are truly important to our target market.  We cannot do that with data from a typical MaxDiff study.  So we are using an old fashioned importance rating scale instead.

As always, it is critical to think about the story you want to tell with your research data, and then work backwards to the design and the choice of methods.  In many cases MaxDiff is the perfect tool.  In other cases it will leave you with data that is difficult to apply to critical questions.

Feel free to give us a call if you need some help deciding among the best methods for your research, whether it be MaxDiff, other conjoint techniques, or something else entirely.  We’ll help you focus on the story you need to tell and on the research design you need to tell it.

–Joe Hopper, Ph.D.

But I can tell them this:  It does get used, so keep doing it.  It allows people like me to bring new insights and new levels of rigor to the practical and sometimes urgent research questions that our customers need to have answered.

The Journal of Marketing Research has just published a special interdisciplinary issue on Consumer Financial Decision Making.  It is hot off the press, so we have yet to read it all.  But in the coming weeks we’ll be reading, reviewing, and using the findings in these articles to bring deeper insight to the work that we do for our customers.

In the meantime, here are the article titles, with links to the authors’ summaries, from the special issue of JMR that focuses on research in consumer finance:

1. Misunderstanding Savings Growth: Implications for Retirement Savings Behavior
2. Earmarking and Partitioning: Increasing Saving by Low-Income Households
3. Increasing Saving Behavior Through Age-Progressed Renderings of the Future Self
4. Winning the Battle but Losing the War: The Psychology of Debt Management
5. Using Loan Plus Lender Literacy Information to Combat One-Sided Marketing of Debt Consolidation Loans
6. Minimum Required Payment and Supplemental Information Disclosure Effects on Consumer Debt Repayment Decisions
7. Leave Home Without It? The Effects of Credit Card Debt and Available Credit on Spending
8. Axe the Tax: Taxes Are Disliked More than Equivalent Costs
9. Once Burned, Twice Shy: How Naive Learning, Counterfactuals, and Regret Affect the Repurchase of Stocks Previously Sold
10. Fear, Social Projection, and Financial Decision Making
11. Microfinance Decision Making: A Field Study of Prosocial Lending
12. Tell Me a Good Story and I May Lend You Money: The Role of Narratives in Peer-to-Peer Lending Decisions
13. Marketing Complex Financial Products in Emerging Markets: Evidence from Rainfall Insurance in India
14. Are Consumers Too Trusting? The Effects of Relationships with Expert Advisers

Need help interpreting and applying these academic findings to the research questions your financial services team has?  Give us a call at 312-348-6089 and we would be happy to help you think about how to bring more insight to your research, and then how to find useful stories in your data that can be put into action.

–Joe Hopper, Ph.D.

Earlier this month Arthur C. Nielsen, Jr. died.  He left behind a giant and reputable market research company and a brand name recognized throughout the world.  The A.C. Nielsen company was started by his father and in its early years tracked the sales of goods through grocery and drug stores.  The company then moved into media tracking and became the authoritative source for measuring audience size and demographics.  Nearly every company with an advertising budget continues to rely on Nielsen data to determine where to advertise and how much to spend.

Nielsen’s legacy is that he demonstrated the value of collecting and tracking data, and lots of it.  Every item we purchase is now logged, counted, and tracked.  Every television and radio show is tracked for how many viewers it has and in what markets they live.  And of course everything we do on the Internet is recorded and tracked.  Even our bodily locations are tracked via GPS or cell phone signals.  Most market research firms today generate the bulk of their revenue simply by collecting, tracking, tabulating, and reporting data.

This important legacy has left us with tons of data, growing at an exponential rate,  and a monumental challenge of how to synthesize it and move beyond mere tabulation and reporting.  The question is, how do we meet that challenge and take Nielsen’s legacy to the next frontier?  In our view, it will involve two key efforts:

1.  Understanding data in much deeper ways and analyzing it with data mining tools, new algorithms, and new approaches that go beyond traditional statistics, including Bayesian analysis, neural networks, and machine-learning techniques.

2. Interpreting and communicating data in ways that are more practical, relevant, meaningful, and useful.  In other words, turning data into stories that real people, real managers, and real businesses understand and can use.

To be sure, much of the research industry is still (appropriately) focused on implementing technologies to better manage, tabulate, and report volumes of data.  But automated tables and charts with ever-expanding levels of detail are reaching their limits of utility.  Versta Research is proud to be on the next frontier, where better interpretation and understanding of data is key.

–Joe Hopper, Ph.D.