One lesson I have learned serving on the board of the American Marketing Association in Chicago is that for most companies, research and marketing can no longer be separated.  Why not?  Because marketers now deal with immense volumes of data.  If they want to make smart decisions and execute campaigns effectively, they need to use that data.  And who best to help them find, interpret, and use data than the research team?

This became even more clear at our AMA event two weeks ago, where Luanne Calvert, vice president of marketing at Virgin America talked about her team’s effort to build an airline customer loyalty program from scratch.  One of the challenges, she noted, was making sense of their data.  “We have tons of data,”  Calvert said.  “There’s so much we barely know what to do with it.  We are just starting to get a handle on it.”

Effectively dealing with tons of data requires a thoughtful research approach, which, by the way, does not necessarily mean doing yet another survey or creating a dashboard of mindless metrics.  In our view, it means the following:

1.  Asking questions.  No matter what the software salespeople tell you, there are no hidden patterns in your data that you can reveal simply by clicking a “predictive analytics” button.  But data can be an immensely powerful resource for answering specific, well-formulated questions.

2.  Using the right tools.  Good data querying and analysis nearly always involves a variety of methods and tools, including statistics programs, database tools, spreadsheets, conjoint models, text analyzers, tabulation programs, and so on.  There is no enterprise-wide platform that can do it all.

3.  Telling a story.  Just as hidden patterns are never revealed by themselves, data likewise never speaks for itself.  The art and science of research is interpreting the data and communicating the story so that it answers key business questions.

The idea of research finding a seat at the executive decision making table is worth striving for.  But it is far more important and far more useful for research to be sitting at the table with the marketing, communications, operations, customer service, innovation, and product development teams.  They are the ones who most urgently need our help to use their wealth of data in truly smart ways.

–Joe Hopper, Ph.D.

Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research.  Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.

This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls.  We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do.

The webinar (scheduled for January 26, 2012 with archived replay available afterwards) is being co-sponsored by the American Association of Public Opinion Research, and will be led by Claudia Deane, associate director for public opinion and survey research at the Kaiser Family Foundation.  It is designed for non-researchers, and will focus on:

• The science of how polling and survey research works
• How to determine the legitimacy of a poll and the rigor of survey research
• How to assess the quality and usefulness of survey questions

If you miss the webinar or hunger for more, Versta Research has (and can point you toward) additional resources to help PR professionals, marketing professionals, and market researchers understand, keep abreast of, and communicate the importance of research.  We are one of the few research firms in the industry with a background in university teaching and academic research striving to bring that expertise to the practical worlds of communications campaigns and marketing insight.  Please give us a call and we would be happy to help you further.

–Joe Hopper, Ph.D.

Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/focus-groups-save-spider-man/ http://www.verstaresearch.com/blog/focus-groups-save-spider-man/#comments Wed, 11 Jan 2012 21:46:57 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1869

In last year’s cliffhanger episode of “Can a Focus Group Save Spider-Man?” we pondered whether market research was powerful enough to save a Broadway show from doom and destruction.  After crushing reviews from theater critics, the producers hired a market research firm to help them rewrite the show.

Guess what?  It worked.  Since the show re-opened in June, it is regularly among the top five earners among Broadway shows, and The Hollywood Reporter reports that in December it “set a new record for a single-week box office gross, raking in $2,941,794 for the week ending January 1.”

In a year when Steve Jobs’ quip that “it’s not the consumers’ job to know what they want” has been used to bludgeon market research, we see research doing a pretty good job identifying buyers’ needs and helping decision makers address those needs in smarter ways.

It is true that consumers can’t answer questions like, “What show do want us to produce next?” or “What great technological innovation would be most useful to you?”  But they certainly can tell you want they want, and the art of research is to ask just the right questions to give you enough insight about what to do next.  Here are three things you can ask about, and that audiences, customers, and prospects will happily tell you:

1.  What is important to them and what they care about.  Insights about what matters to buyers will help you design your product or service and will provide deep insight about how to market it.

2.  What their frustrations are and what is not working.  Most consumers are eager to critique products and services that fail to meet their needs, which highlights the opportunities for new solutions.

3.  How good or bad your idea is.  There are lots of ways to test concepts, products, and ideas, and survey research is remarkably good at predicting winners versus losers.

The lesson from Spider-Man is that it does not take superhuman powers to fix a flawed strategy nor off-the-charts creative genius to make a good product sell.  A thoughtful effort to ask questions and to listen to your customers is sometimes all it takes to turn harrowing encounters with goblins into success.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/focus-groups-save-spider-man/feed/ 0 http://www.verstaresearch.com/blog/a-path-to-better-research-with-geo-maps/ http://www.verstaresearch.com/blog/a-path-to-better-research-with-geo-maps/#comments Wed, 04 Jan 2012 22:55:11 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1855 Given how common mapping capabilities have become via the Internet and smartphones, it is surprising that we don’t see more geographic mapping in market research.  Researchers nearly always look at customer demographics, and a key component of a person’s demographic profile is where he or she lives.  This data is far more compelling if you can present it visually with maps.

It does not take super fancy (and expensive) mapping software or specialized firms to create accurate, useful, and compelling maps from market research data.  We recently created maps for a client showing where in a three-county region their best customers lived.  Everything we used to make these maps was free and publicly available for download on the Internet.  Here are the steps we used:

1.  Download shapefiles from the U.S. Census Bureau.  These files contain data to demarcate all legal and statistical geographical areas in the U.S. including states, counties, county subdivisions, census tracts, blocks, and so on.

2.  Edit the shapefiles with a program like QGIS.  There are several high quality, free, open-source software packages that you can use to read and manipulate census shapefiles.  We used QGIS, which is a program created and continually developed by the Open Source Geospatial Foundation.

3.  Link customer data to shapefile data in a spreadsheet program.  We looked at the number of customers in every zip code, then linked that data to county subdivisions in the shapefile by using a minimum distance function based on latitude and longitude coordinates.

4.  Plot the data and create the map using R.  R is quickly becoming the statistics package of choice in the academic world.  It is a free “integrated suite of software…for statistical computing and graphics” and can easily turn shapefiles and data linked to those shapefiles into visual displays.

Ultimately we created a heat map that displays customer location data for the three counties, which are divided into more than 50 townships, as shown in the map above, with darker colors signifying more customers than lighter colors.

As always, the ongoing challenge for researchers working with a burgeoning volume of data is how to interpret all that data, synthesize it, and simplify it into a story that is useful to decision makers.  Maps have always been a useful and compelling way to visually present data.  Finding the path to producing them from your data is now easier than ever.

–Joe Hopper, Ph.D.

Even if you are not involved in political polling, it is worth paying attention to the methods and best practices of political pollsters.  One reason is that few other areas of research offer a way to completely validate one’s methods.  Pollsters are using sampling and survey methods to predict the behaviors of a much larger population.  Then in just one day that population behaves, we get a near-perfect count of exactly how they behaved, and we know whether the methods worked.

Several industry colleagues have recently been debating the merits of calculating and reporting “margins of error” in political polling, and pointed us to some surprising data from The New York Times:

[The New York Times has compiled] a database consisting of thousands of primary and caucus polls dating back to the 1970s.  Each poll contains numbers for several candidates, so there are a total of about 17,000 observations. How often does a candidate’s actual vote total fall within the theoretical margin of error?  The answer is, not very often. In theory, a candidate’s actual vote total should fall outside the margin of error only 5 percent of the time [given that political polls report margins of error using a 95% confidence interval]. In reality, the candidate’s vote total was outside the margin of error 65 percent of the time! Part of this is because the database includes some polls conducted months before the actual voting took place. But even if you restrict the analysis to polls conducted within the final week of the campaign, about 40 percent of the vote totals fell outside the margin of error — eight times more often than is supposed to happen if you could take the margin of error at face value.

This does not mean that the polls were wrong, predicting wins for losing candidates and vice versa.  Rather, it means that the estimates were not as precise as the stated margins of error would have a reader believe.

The problem is that “margins of error” are based on a statistical theories that almost never line up with the messy reality of our world.  Margins of error make a number of assumptions which are rarely true in practice, including:

• Respondents are selected through simple random sampling

• All those sampled participate in the survey

• Sampling error is the only source of survey error

Indeed, Versta Research usually recommends to clients who publish survey research that they not report margins of error because the concept (and the calculations) are seriously misleading and flawed.

Calculating margins of error and looking at statistical significance should be used not because they give accurate or “scientific” predictions, but because they provide useful summary measures of how much variability there is in the data given the sample size and other critical factors that can affect one’s estimates.  At Versta Research, this helps us better interpret data and better assess what matters.  That, in turn, allows us to tell a story with the data that does not overreach or misrepresent what is going on.

–Joe Hopper, Ph.D.

If you HAVE done the hard work of phone surveys, then besides deeper research insights into your business, we expect you will have a deeper appreciation for this holiday survey, written by Mitch Pravatiner, which is written in exactly the format that telephone interviewers typically see.  We have worked with Mitch in the past, and he is a true pro when it comes to telephone interviewing. He recently shared this via aapor-net, which is a discussion group of research professionals who belong to the American Association of Public Opinion Research.

‘Twas the Night Before Christmas . . .

Hello, this is [NAME OF INTERVIEWER] from [NAME OF ORGANIZATION], an opinion research organization. Tonight we’re calling people across the country to find out what’s going on in their homes on the night before Christmas. Please rest assured that this is not a sales call, and that everything you tell us will be kept confidential. Would you be willing to participate?

Yes (CONTINUE)
No (THANK AND TERMINATE)

Q1 First of all, have you noticed any creatures stirring in your house tonight?

Yes (SKIP TO Q3)
No
DON’T KNOW (SKIP TO Q3)
REFUSED (SKIP TO Q3)

Q2 Not even a mouse? (DO NOT READ LIST)

YES, NOTICED MOUSE STIRRING
NO, NOT EVEN A MOUSE
DON’T KNOW
REFUSED

Q3 Have you hung any stockings by your chimney?

Yes
No (SKIP TO Q6)

Q4 How much care did you use in hanging the stockings by the chimney? Please rate it on a scale from 0 to 10, where 0 means “no care at all” and 10 means “extreme care.”

0
1
2
3
4
5
6
7
8
9
10
DON’T KNOW
REFUSED

Q5 What would you say was the single main reason you hung stockings by the chimney? Was it:

In the hope that Santa Claus would drop in
To dry them after you had to walk home through a series of deep snowdrifts
They had developed runs, so there was no other use for them, or
Some other reason (SPECIFY)
DON’T KNOW
REFUSED

Q6 As far as you know, what visions have your children experienced in bed tonight? (RANDOMIZE)

(NO CHILDREN)
Barbie dolls
GI Joes
Chatty Cathys
Sugar plums
Betsy Wetsys
Cabbage Patch Kids
Tickle Me Elmos
Easy Bake Ovens
Candy canes
Radio Flyers
Fruitcake
Red Ryder BB guns
Other (if “yes” go to Q6a, otherwise skip to q7)

Q6a What other visions? (PROBE AND CLARIFY)

DON’T KNOW
REFUSED

Q7 Have you tried to get any sleep so far tonight?

Yes
No
DON’T KNOW
REFUSED

Q8 Were you awakened from your sleep at any time tonight?

Yes
No
DON’T KNOW
REFUSED

Q9 What woke you up? (READ LIST)

A loud noise outside the house
A loud noise inside the house
Indigestion from something you ate at the office party earlier today
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q10 What did you do after you woke up? (READ LIST)

Got up to see what was going on
Shrugged it off and went back to sleep
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q11 What did you notice after that?

A sleigh pulled by reindeer parked on the front lawn
A UPS truck that overturned after skidding on black ice on the street in front of your house
Several drunks who passed out on the lawn after leaving the party next door
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q12 Was Santa Claus with the sleigh and the reindeer?

Yes
No
DON’T KNOW
REFUSED

Q13 How many reindeer did you see?

(RECORD NUMBER)
DON’T KNOW
REFUSED

Q14 Did Santa mention any of them by name?

Yes
No
DON’T KNOW
REFUSED

Q15 Which reindeer do you recall him mentioning? (DO NOT READ LIST; MULTIPLE ANSWERS PERMITTED)

DASHER
DANCER
PRANCER
VIXEN
COMET
CUPID
DONNER
BLITZEN
OTHER (SPECIFY)
DON’T KNOW
REFUSED

Q16 (ASK FOR EACH REINDEER NOT MENTIONED IN Q15)

Did Santa also mention [NAME OF REINDEER] by name?

Yes
No
DON’T KNOW
REFUSED

Q17 Did Santa move the sleigh anywhere after that?

Yes
No
DON’T KNOW
REFUSED

Q18 Where did he move it to?

The roof
The back yard
The alley behind the house
Somewhere else (SPECIFY)
DON’T KNOW
REFUSED

Q19 Did Santa enter your house after that?

Yes
No
DON’T KNOW
REFUSED

Q20 How did he enter your house?

Through the front door
Through the back door
Through the chimney
Some other way (SPECIFY)
DON’T KNOW
REFUSED

Q21 Can you describe Santa while he was in your house? (PROBE AND CLARIFY)

DON’T KNOW
REFUSED

Q22 Did you and he speak to each other while he was in the house?

Yes
No
DON’T KNOW
REFUSED

Q23 Which of the following best describes what Santa Claus did while in your house?

Filled the stockings and then left
Filled the stockings, then ate your Christmas cookies, and then left
Filled the stockings, ate your Christmas cookies, polished off your eggnog, and left
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q24 How did he leave your house?

Through the front door
Through the back door
Through the chimney
Some other way (SPECIFY)
DON’T KNOW
REFUSED

Thank you. Those are all the questions we have for you this evening. Merry Christmas to all, and to all a good night.

–Joe Hopper, Ph.D.

We recently received and reviewed an excellent book summarizing practical findings from academic marketing research.  It is called Consumer Insights: Findings from Behavioral Research. It is published by the Marketing Science Institute and we highly recommend it.  Why is it so good and useful for corporate researchers and marketers?

• It provides a quick overview of what we know in various areas of marketing
• It is organized into 42 useful topics, most of which are relevant to nearly all marketers
• Each topic provides universal findings based on research from hundreds of studies, not just one or two
• Each topic/chapter is short and to the point (just two or three pages)
• Each topic outlines insights, the evidence base, and managerial implications

You will find important applications to almost any marketing effort you are engaged in.  For example, here is one that applies to what we are doing right now:  “Consumers are becoming more and more suspicious of online bloggers touting products.  Thus, a blogger’s attempt to tout a product may actually backfire and reduce consumer trust in both the blogger and the product.”  So to allay concerns, it is important to note that we have no connection to the authors, publishers, or sellers of this book!  Nor was the book pitched to us, or provided to us free (alas, we paid full price).

Here are the topics covered in the book:

Information Search
1. Effects of Product Knowledge on information Search
2. In-store Decision Making and Unplanned Purchases
3. Perceptions of Product Assortment
4. Variety-seeking Behavior
5. Consumer Search on the Internet
6. Buyers’ Post-purchase Information Biases Pricing
7. Perception of Price Deals
8. Biases in Processing Price Information
9. Effects of the Internet on Consumer Price Sensitivity
10. Effects of Transaction Structure on Price Perceptions & Consumption
11. Perceptions of Price Fairness

Advertising
12. Consumer Attention to Advertising
13. Effects of Ad Likability

Brand Effects
14. Consumer Brand Recall
15. Perceptions of Brand Extensions
16. Brand Dilution and Protection

Consumer Inferences
17. Consumer Inferences and Assumptions
18. Perceptions of Quality Signals
19. Causal Inferences and Consumers’ Judgments
20. Consumer Use of Persuasion Knowledge

Feelings, Attitudes, and Persuasion
21. Effects of Mere Exposure on Brand Liking
22. Influence of Feelings & Emotions on Consumers’ Judgments
23. Persuasion: Elaboration Likelihood Model
24. Consumer Goal Orientation
25. Consumer Attitudes Toward Marketing

Decision Making and Purchase
26. Pre-choice Bias in Brand Choice
27. Consumers’ Intertemporal Preferences
28. Loss Aversion and Consumer Choice
29. Protected Values
30. Purchase Intentions and Purchasing
31. Consumer Habits and Purchase Behavior
32. Impulsive and Compulsive Buying

The Social Consumer
33. Social Contagion and Word-of-mouth
34. Consumer Identity and Purchase Behavior
35. Perceptions of Advisors

Vulnerable Consumers
36. Children and Advertising
37. Aging Consumers
38. Effects of Low Literacy on Consumer Decision Making

Health and Well-being
39. Effects of Nutrition Information and Health Claims on Consumption
40. Mass-media Campaigns and Health-related Behaviors
41. Perceptions of Health Risks
42. Effects of Portion/Package Size and Shape on Consumption

The book is excellent and useful.  Get it.  It is now available through Amazon at a substantially lower price than we paid.

We learned many things we did not know, and we learned that some of the things we have come to know about marketing through our own experiences now have a wider research base to back them up.

Happy insights, from Versta Research!

–Joe Hopper, Ph.D.

Here we offer the “Versta Digest” version as a handy reference card.  Once you get the hang of it, you don’t need the examples and explanation.  You just need to know the rules.  We recommend reading the full article first, so you know what we’re talking about when it comes to “points.”  Then, when you need a refresher or a reference source, consult these rules:

• One point for each simple question or scaled response
• One point for every two response options in a multiple choice question
• One point for each row in a grid question
• Two points for any response that requires mental calculation
• Three points for every short response to an open-ended question
• One point for every three sentences of extra text that respondents must read

Then:

• Tally up the points
• Divide by 8 for online survey length (in minutes)
• Divide by 8 then multiply by 1.5 for phone survey length (in minutes)

The system is straightforward, easy to learn, and easy to execute.  It is a method that really works and that we have validated against hundreds of different types of surveys over the past several years.

Once you know how to determine survey length, you can think more strategically about the ideal survey length to optimize the value and content of a survey within your budget.  Call us at 312-348-6089 with any additional assistance you may need.

–Joe Hopper, Ph.D.

Spreadsheets have amazingly powerful and often intuitive capabilities.  They have many of the statistical functions we use every day.  Younger people entering our profession rarely know programs like SPSS or SAS, and we see them turning to Excel to generate frequencies, calculate means and proportions, create charts from data, and so on.  The same goes for our customers.  Many do not have statistical software, so when they need numbers and statistics, they often work in Excel.

But Versta Research continues to invest in advanced statistical software rather than doing our work in Excel for three important reasons:

1. Speed and efficiency.  The tools we use are designed to do exactly what we need.  Spreadsheets require more effort to manipulate data, set logic rules, and write formulas that we can otherwise do with just a few clicks.

2.  Leveraging analytical innovation.  Our statistical software leverages the newest techniques in developing areas of statistical theory and applications, especially from software developers like Sawtooth who are pioneers in choice modeling.

3.  Accuracy.  As anyone who has created a moderately complex spreadsheet knows, it is frighteningly easy to make errors in spreadsheets (data errors, sorting errors, formula errors, copy-and-paste errors, cell reference errors, and the list goes on) and it is often difficult to find, detect, and untangle those errors, if indeed they are ever found.

To be sure, Excel is a powerful tool that we use all the time and every day, in part because it can be used in so many creative and flexible ways.  We use it help us track, manipulate, and visualize statistical output, for example.  We also use it as an efficient way to write multiple lines of programming script that we then paste into our statistical programs.  But when it comes to the core of our statistical analysis, we rely on the best-in-breed software tools that continue to outpace the capabilities of a spreadsheet.

–Joe Hopper, Ph.D.