Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research.  Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.

This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls.  We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do.

The webinar (scheduled for January 26, 2012 with archived replay available afterwards) is being co-sponsored by the American Association of Public Opinion Research, and will be led by Claudia Deane, associate director for public opinion and survey research at the Kaiser Family Foundation.  It is designed for non-researchers, and will focus on:

• The science of how polling and survey research works
• How to determine the legitimacy of a poll and the rigor of survey research
• How to assess the quality and usefulness of survey questions

If you miss the webinar or hunger for more, Versta Research has (and can point you toward) additional resources to help PR professionals, marketing professionals, and market researchers understand, keep abreast of, and communicate the importance of research.  We are one of the few research firms in the industry with a background in university teaching and academic research striving to bring that expertise to the practical worlds of communications campaigns and marketing insight.  Please give us a call and we would be happy to help you further.

–Joe Hopper, Ph.D.

Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/focus-groups-save-spider-man/ http://www.verstaresearch.com/blog/focus-groups-save-spider-man/#comments Wed, 11 Jan 2012 21:46:57 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1869

In last year’s cliffhanger episode of “Can a Focus Group Save Spider-Man?” we pondered whether market research was powerful enough to save a Broadway show from doom and destruction.  After crushing reviews from theater critics, the producers hired a market research firm to help them rewrite the show.

Guess what?  It worked.  Since the show re-opened in June, it is regularly among the top five earners among Broadway shows, and The Hollywood Reporter reports that in December it “set a new record for a single-week box office gross, raking in $2,941,794 for the week ending January 1.”

In a year when Steve Jobs’ quip that “it’s not the consumers’ job to know what they want” has been used to bludgeon market research, we see research doing a pretty good job identifying buyers’ needs and helping decision makers address those needs in smarter ways.

It is true that consumers can’t answer questions like, “What show do want us to produce next?” or “What great technological innovation would be most useful to you?”  But they certainly can tell you want they want, and the art of research is to ask just the right questions to give you enough insight about what to do next.  Here are three things you can ask about, and that audiences, customers, and prospects will happily tell you:

1.  What is important to them and what they care about.  Insights about what matters to buyers will help you design your product or service and will provide deep insight about how to market it.

2.  What their frustrations are and what is not working.  Most consumers are eager to critique products and services that fail to meet their needs, which highlights the opportunities for new solutions.

3.  How good or bad your idea is.  There are lots of ways to test concepts, products, and ideas, and survey research is remarkably good at predicting winners versus losers.

The lesson from Spider-Man is that it does not take superhuman powers to fix a flawed strategy nor off-the-charts creative genius to make a good product sell.  A thoughtful effort to ask questions and to listen to your customers is sometimes all it takes to turn harrowing encounters with goblins into success.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/focus-groups-save-spider-man/feed/ 0 http://www.verstaresearch.com/blog/a-path-to-better-research-with-geo-maps/ http://www.verstaresearch.com/blog/a-path-to-better-research-with-geo-maps/#comments Wed, 04 Jan 2012 22:55:11 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1855 Given how common mapping capabilities have become via the Internet and smartphones, it is surprising that we don’t see more geographic mapping in market research.  Researchers nearly always look at customer demographics, and a key component of a person’s demographic profile is where he or she lives.  This data is far more compelling if you can present it visually with maps.

It does not take super fancy (and expensive) mapping software or specialized firms to create accurate, useful, and compelling maps from market research data.  We recently created maps for a client showing where in a three-county region their best customers lived.  Everything we used to make these maps was free and publicly available for download on the Internet.  Here are the steps we used:

1.  Download shapefiles from the U.S. Census Bureau.  These files contain data to demarcate all legal and statistical geographical areas in the U.S. including states, counties, county subdivisions, census tracts, blocks, and so on.

2.  Edit the shapefiles with a program like QGIS.  There are several high quality, free, open-source software packages that you can use to read and manipulate census shapefiles.  We used QGIS, which is a program created and continually developed by the Open Source Geospatial Foundation.

3.  Link customer data to shapefile data in a spreadsheet program.  We looked at the number of customers in every zip code, then linked that data to county subdivisions in the shapefile by using a minimum distance function based on latitude and longitude coordinates.

4.  Plot the data and create the map using R.  R is quickly becoming the statistics package of choice in the academic world.  It is a free “integrated suite of software…for statistical computing and graphics” and can easily turn shapefiles and data linked to those shapefiles into visual displays.

Ultimately we created a heat map that displays customer location data for the three counties, which are divided into more than 50 townships, as shown in the map above, with darker colors signifying more customers than lighter colors.

As always, the ongoing challenge for researchers working with a burgeoning volume of data is how to interpret all that data, synthesize it, and simplify it into a story that is useful to decision makers.  Maps have always been a useful and compelling way to visually present data.  Finding the path to producing them from your data is now easier than ever.

–Joe Hopper, Ph.D.

We recently received and reviewed an excellent book summarizing practical findings from academic marketing research.  It is called Consumer Insights: Findings from Behavioral Research. It is published by the Marketing Science Institute and we highly recommend it.  Why is it so good and useful for corporate researchers and marketers?

• It provides a quick overview of what we know in various areas of marketing
• It is organized into 42 useful topics, most of which are relevant to nearly all marketers
• Each topic provides universal findings based on research from hundreds of studies, not just one or two
• Each topic/chapter is short and to the point (just two or three pages)
• Each topic outlines insights, the evidence base, and managerial implications

You will find important applications to almost any marketing effort you are engaged in.  For example, here is one that applies to what we are doing right now:  “Consumers are becoming more and more suspicious of online bloggers touting products.  Thus, a blogger’s attempt to tout a product may actually backfire and reduce consumer trust in both the blogger and the product.”  So to allay concerns, it is important to note that we have no connection to the authors, publishers, or sellers of this book!  Nor was the book pitched to us, or provided to us free (alas, we paid full price).

Here are the topics covered in the book:

Information Search
1. Effects of Product Knowledge on information Search
2. In-store Decision Making and Unplanned Purchases
3. Perceptions of Product Assortment
4. Variety-seeking Behavior
5. Consumer Search on the Internet
6. Buyers’ Post-purchase Information Biases Pricing
7. Perception of Price Deals
8. Biases in Processing Price Information
9. Effects of the Internet on Consumer Price Sensitivity
10. Effects of Transaction Structure on Price Perceptions & Consumption
11. Perceptions of Price Fairness

Advertising
12. Consumer Attention to Advertising
13. Effects of Ad Likability

Brand Effects
14. Consumer Brand Recall
15. Perceptions of Brand Extensions
16. Brand Dilution and Protection

Consumer Inferences
17. Consumer Inferences and Assumptions
18. Perceptions of Quality Signals
19. Causal Inferences and Consumers’ Judgments
20. Consumer Use of Persuasion Knowledge

Feelings, Attitudes, and Persuasion
21. Effects of Mere Exposure on Brand Liking
22. Influence of Feelings & Emotions on Consumers’ Judgments
23. Persuasion: Elaboration Likelihood Model
24. Consumer Goal Orientation
25. Consumer Attitudes Toward Marketing

Decision Making and Purchase
26. Pre-choice Bias in Brand Choice
27. Consumers’ Intertemporal Preferences
28. Loss Aversion and Consumer Choice
29. Protected Values
30. Purchase Intentions and Purchasing
31. Consumer Habits and Purchase Behavior
32. Impulsive and Compulsive Buying

The Social Consumer
33. Social Contagion and Word-of-mouth
34. Consumer Identity and Purchase Behavior
35. Perceptions of Advisors

Vulnerable Consumers
36. Children and Advertising
37. Aging Consumers
38. Effects of Low Literacy on Consumer Decision Making

Health and Well-being
39. Effects of Nutrition Information and Health Claims on Consumption
40. Mass-media Campaigns and Health-related Behaviors
41. Perceptions of Health Risks
42. Effects of Portion/Package Size and Shape on Consumption

The book is excellent and useful.  Get it.  It is now available through Amazon at a substantially lower price than we paid.

We learned many things we did not know, and we learned that some of the things we have come to know about marketing through our own experiences now have a wider research base to back them up.

Happy insights, from Versta Research!

–Joe Hopper, Ph.D.

Suppose you want to measure the importance of 12 attributes for a new product or service.  If you know ahead of time that consumers are going to say that all 12 are extremely important to them, then MaxDiff is an excellent method for differentiating among the attributes so you can focus on the top two or three that matter most.

But what if you don’t know that all 12 attributes are extremely important?  Maybe none of them are.  Maybe they run the gamut from unimportant to extremely important.  The problem with MaxDiff is that it only tells you the importance of attributes relative to each other, but it won’t tell you whether the attributes are important.  The MaxDiff model will assign ratio-level numbers so that you can rank and quantify the importance of each attribute vis-à-vis the others.  But it will not anchor the attributes in a meaningful way.

This week we are designing a study in which we want to differentiate among attributes, but we also want to measure the gap between satisfaction and importance for items that are truly important to our target market.  We cannot do that with data from a typical MaxDiff study.  So we are using an old fashioned importance rating scale instead.

As always, it is critical to think about the story you want to tell with your research data, and then work backwards to the design and the choice of methods.  In many cases MaxDiff is the perfect tool.  In other cases it will leave you with data that is difficult to apply to critical questions.

Feel free to give us a call if you need some help deciding among the best methods for your research, whether it be MaxDiff, other conjoint techniques, or something else entirely.  We’ll help you focus on the story you need to tell and on the research design you need to tell it.

–Joe Hopper, Ph.D.

But I can tell them this:  It does get used, so keep doing it.  It allows people like me to bring new insights and new levels of rigor to the practical and sometimes urgent research questions that our customers need to have answered.

The Journal of Marketing Research has just published a special interdisciplinary issue on Consumer Financial Decision Making.  It is hot off the press, so we have yet to read it all.  But in the coming weeks we’ll be reading, reviewing, and using the findings in these articles to bring deeper insight to the work that we do for our customers.

In the meantime, here are the article titles, with links to the authors’ summaries, from the special issue of JMR that focuses on research in consumer finance:

1. Misunderstanding Savings Growth: Implications for Retirement Savings Behavior
2. Earmarking and Partitioning: Increasing Saving by Low-Income Households
3. Increasing Saving Behavior Through Age-Progressed Renderings of the Future Self
4. Winning the Battle but Losing the War: The Psychology of Debt Management
5. Using Loan Plus Lender Literacy Information to Combat One-Sided Marketing of Debt Consolidation Loans
6. Minimum Required Payment and Supplemental Information Disclosure Effects on Consumer Debt Repayment Decisions
7. Leave Home Without It? The Effects of Credit Card Debt and Available Credit on Spending
8. Axe the Tax: Taxes Are Disliked More than Equivalent Costs
9. Once Burned, Twice Shy: How Naive Learning, Counterfactuals, and Regret Affect the Repurchase of Stocks Previously Sold
10. Fear, Social Projection, and Financial Decision Making
11. Microfinance Decision Making: A Field Study of Prosocial Lending
12. Tell Me a Good Story and I May Lend You Money: The Role of Narratives in Peer-to-Peer Lending Decisions
13. Marketing Complex Financial Products in Emerging Markets: Evidence from Rainfall Insurance in India
14. Are Consumers Too Trusting? The Effects of Relationships with Expert Advisers

Need help interpreting and applying these academic findings to the research questions your financial services team has?  Give us a call at 312-348-6089 and we would be happy to help you think about how to bring more insight to your research, and then how to find useful stories in your data that can be put into action.

–Joe Hopper, Ph.D.

Earlier this month Arthur C. Nielsen, Jr. died.  He left behind a giant and reputable market research company and a brand name recognized throughout the world.  The A.C. Nielsen company was started by his father and in its early years tracked the sales of goods through grocery and drug stores.  The company then moved into media tracking and became the authoritative source for measuring audience size and demographics.  Nearly every company with an advertising budget continues to rely on Nielsen data to determine where to advertise and how much to spend.

Nielsen’s legacy is that he demonstrated the value of collecting and tracking data, and lots of it.  Every item we purchase is now logged, counted, and tracked.  Every television and radio show is tracked for how many viewers it has and in what markets they live.  And of course everything we do on the Internet is recorded and tracked.  Even our bodily locations are tracked via GPS or cell phone signals.  Most market research firms today generate the bulk of their revenue simply by collecting, tracking, tabulating, and reporting data.

This important legacy has left us with tons of data, growing at an exponential rate,  and a monumental challenge of how to synthesize it and move beyond mere tabulation and reporting.  The question is, how do we meet that challenge and take Nielsen’s legacy to the next frontier?  In our view, it will involve two key efforts:

1.  Understanding data in much deeper ways and analyzing it with data mining tools, new algorithms, and new approaches that go beyond traditional statistics, including Bayesian analysis, neural networks, and machine-learning techniques.

2. Interpreting and communicating data in ways that are more practical, relevant, meaningful, and useful.  In other words, turning data into stories that real people, real managers, and real businesses understand and can use.

To be sure, much of the research industry is still (appropriately) focused on implementing technologies to better manage, tabulate, and report volumes of data.  But automated tables and charts with ever-expanding levels of detail are reaching their limits of utility.  Versta Research is proud to be on the next frontier, where better interpretation and understanding of data is key.

–Joe Hopper, Ph.D.

Two researchers at the U.S. Census Bureau recently outlined an emerging innovation in survey research that could reverse the trend towards passive, boring, self-administered surveys that characterizes much online research.  The idea is to use internet avatars in real-time interviewing with survey respondents.

Beyond just the heightened interest of having an animated survey, the avatars would be programmed to register and interpret respondents’ verbal answers, facial expressions, and body language through webcams.

Suppose, for example, that a respondent answers a question with detailed information that answers a follow-up question as well.  The avatar would use natural language processing to insert that data into the subsequent question, and then avoid asking the follow-up.  Or if the respondent looks away from the screen and pauses for time longer than is typical, the avatar can offer a rephrased question or a reassuring comment to re-engage the participant and to put him or her at ease.  This type of innovation  could bring many of the advantages of live interviewing back into the realm of internet surveys, which are far more efficient in terms of time and cost.

The use of effective avatar interviewers is at least several years away, however, because it involves not only evolving internet technologies, but also advanced linguistic processing, facial and voice recognition technologies, and so on.  In fact, the sheer technological difficulty of truly replacing human interviewers reminds us of how absurd it is for research companies to make claims about technology replacing higher-order activities in the research process, such as providing analysis and insight.

At least for now, software and services with “actionable insight” buttons generate yet more mountains of data in need of human synthesis and interpretation.  If anything, the role for smart and experienced researchers who can turn all that data into a story is growing.  It is growing for researchers who work on the client side and who have direct accountability to the executives who need data-driven insights.  And it is growing for firms like Versta Research where the highest levels of intellectual and human capital are central to our work.

–Joe Hopper, Ph.D.

One way to analyze the data is to use a simple counting technique.  It works pretty well, by the way.  But the more common way these days is to use advanced statistical modeling that allows for stronger individual-level estimates of importance scores.  This affords better opportunities for segmenting the data and analyzing subgroups.

We typically see analysts transforming the importance scores (or utilities) to a 0—100 scale.  It is a compelling scale because all importance scores add up to 100, which mimics the technique of having respondents allocate 100 importance points to each of the features they care about.

But if you do a lot of MaxDiff studies and frequently present them to your management team, you lose the ability to provide a rule of thumb as to what counts as a “high” score vs. a “low” score.  If one MaxDiff exercise uses fifteen attributes, an importance score of 20 is quite high; if another MaxDiff exercise uses just five attributes, and importance score of 20 is just average.

The solution?  Instead, we typically transform scores to a 0—100*N scale, where N=the number of attributes tested.  If you tested fifteen attributes, transform to a 0—1500 scale.  If you tested five attributes, transform to a 0—500 scale.  With this method, a score of 100 is always the average, scores above 100 are always above average, and scores below 100 are always below average.  Sometimes we transform this into a chart showing “percent above/below  average” as shown in the third example below, but this is not always necessary because anchoring the average at 100 makes it simple to calculate those percentages mentally.

Example 1: Versta's Recommended Scaling of MaxDiff Scores

Example 2: The Typical Way of Scaling MaxDiff Scores

Example 3: Another (Sometimes Useful) Way to Scale MaxDiff Scores

Need help with the smartest ways to design, implement, and report MaxDiff studies?  Call us at (312) 348-6089 for assistance and further information.

–Joe Hopper, Ph.D.