They are not happy with how poorly their polls have fared (who would be?), and they have teams of smart people trying to figure out what is wrong.  Given their high profile, they are making the process and findings of their investigations public, and we have much to learn from that.

Last week they released their findings from an extensive review, which involved outside experts as well as internal ones.  It is fascinating to read, because they identify 13 suspects in their survey process that all companies who do survey research should always be thinking about:

 1. Tracking design.  Did daily sampling quotas, interviews, and weighting (versus aggregating quotas and weighting protocols over several days) affect the results?  Verdict: innocent.

 2. RDD list-assisted landline vs. listed landline samples.  Gallup used random digit dialing only for its mobile phone sample, but not for landlines.  For landline phones, they called only numbers published in directories. Verdict: guilty.

 3. Company name.  Does the brand name of the polling firm influence who is willing to participate in surveys and how they answer?  Verdict: innocent.

 4. Race of the interviewer.  African American interviewers win better cooperation among African American voters, but there was no discernible effect on outcomes.  Verdict: innocent.

 5. Gender of the interviewer.  Women interviewers were somewhat more likely to get respondents who supported Obama, but there was no discernible effect on outcomes.  Verdict: innocent.

 6. Neutral probing of “don’t know” and “refused” responses.  Gently pushing respondents to reveal an underlying preference even if they say “I don’t know” might have affected statistical estimates.  Verdict: innocent.

 7. Geographic distribution of interviews.  National samples usually set quotas by four large regions within the U.S., but not by areas within those regions.  Verdict: guilty.

 8. Interview completion time.  Which day of the week polling is done, and at what times of day, can affect the types of respondents who are reached.  Verdict: innocent.

 9. Cellphone and landline phone distribution.  The most accurate telephone polling now requires companies to call both cellphones and landlines.  Was the 50%-50% mix appropriate?  Verdict: innocent.

 10. Measuring and weighting race.  The most accurate ways to ask about race and to compensate for skews with statistical weighting is always changing, even at the U.S. Census Bureau.  Did Gallup’s protocols affect their results?  Verdict: guilty.

 11. Handling of third-party candidates.  Not explicitly reading third-party candidate names and party affiliations could affect the accuracy of polling predictions.  Verdict: innocent.

 12. Candidate name order in questions.  The order in which names are read does affect outcomes, but randomization helps to ensure that any effects are equal across all candidates.  Verdict: innocent.

 13. Likely voter estimating.  Some opinions should be ignored in election polls if there are good reasons to believe those people will not vote.  But how good were the criteria for deciding who the most likely voters are?  Verdict: guilty.

Conducting valid and reliable surveys is not easy.  There are multiple places where design, fieldwork, data collection, and analysis can go wrong.  There are many places where experience and professional judgment play critical roles.

Every survey research and measurement firm and every internal market research team should be keeping their eyes on suspects like these — and additional suspects that are unique to each research effort — with the highest levels of vigilance and care.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

The Excel coding error, courtesy of The Roosevelt Institute

Or maybe we should say: why vendors SHOULD do everything twice BEFORE their work hits your desk and you send it back because you found errors. When it comes to something as complex and exacting as market research or public opinion polling, there are almost certainly mistakes the first time around. If a company does not have processes to validate data and deliverables, unfortunately those mistakes end up with you.

Recent errors in an economics paper that laid the foundation for Europe’s austerity programs provides a dramatic and painful example, as outlined by Paul Krugman of the New York Times:

Other researchers, using seemingly comparable data on debt and growth, couldn’t replicate the Reinhart-Rogoff results. They typically found some correlation between high debt and slow growth — but nothing that looked like a tipping point at 90 percent or, indeed, any particular level of debt. Finally, Ms. Reinhart and Mr. Rogoff allowed researchers at the University of Massachusetts to look at their original spreadsheet — and the mystery of the irreproducible results was solved. First, they omitted some data; second, they used unusual and highly questionable statistical procedures; and finally, yes, they made an Excel coding error.

Coding errors like this are the ones that most insidiously creep into research, which is precisely why vendors should do everything twice. And not just twice, but twice with different tools. Some examples:

• If using Excel to produce calculations or statistics, pull the data into another statistics program (like SPSS) and replicate the calculations.
• Likewise, if using SPSS to produce calculations or statistics, pull the data into another statistics program (like WinCross or R) and replicate the calculations.
• When writing syntax that constructs new variables from existing data, cross tabulate the new variable against the old one(s) to ensure the logic was correct.
• Proofread (and read backwards) all written text in reports and press releases that cite numbers to ensure that numbers were correctly pulled from the data.

Sometimes doing everything twice feels ridiculous. We have computers to eliminate errors and help us work faster, right? Indeed they do, but complex computing tools also compound and hide simple errors that are easy to find. Once we set up the processes to replicate results and check for errors, it takes very little time. Our customers get flawless (and correct!) insights, and they never even know that we did everything twice.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

Others are following his lead, though not always to the same superb effect.  Here is another animated statistical graph, this one documenting drone strikes and fatalities in Pakistan:

In my view, this second one is interesting and it represents a worthy effort to explore animated presentations of data.  But the animation does not add anything beyond the full, finished graph at the end.  It is like animated PowerPoint presentations: They (sometimes) look cool, but so what?  It is a great reminder that new innovations in market research and public opinion polling must be carefully matched with specific needs.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

On Monday I attended an update of sorts, via presentations in Chicago sponsored by our AMA local chapter and the Chicago chapter of the Market Research Association.  Annie Pettit of Research Now, an expert on social media analysis and listening, spoke about the current state of quantitative social media research.  I was struck by these points she made, in particular:

1.  Automated sentiment coding of social media content (text analysis and text mining) has limited accuracy and validity.  Even the best machine-learning algorithms cannot untangle the layers of nuance and sarcasm and the rapid shifts in linguistic usage, especially when it comes to how people converse online.

2.  The sources of social media data matter a lot.  Twitter, LinkedIn, Facebook, Tumblr, and Blogger  have dramatically different content, and the inferences one might draw from each can be jarringly different.  This means that effective sampling and representation matter just as much as ever.

3.   Brand mentions on social media reflect buzz, not brand awareness.  P90X is mentioned far more often than Pfizer on social media platforms.  But which do you think has higher brand awareness?

4.  Demographic data on social media is lousy, when it exists at all.  While people mostly tell the truth on surveys, they either omit demographic data or lie about it on social media platforms.  Annie’s advice?  If you need to look at demographics, do a survey.

5.  Social media analysis is slow — or at least as slow as more traditional research design, data collection, and analysis.  The reason is that high quality, useful social media analysis requires all the same thoughtful effort with sampling, data collection, data cleaning, analysis, and report writing.  There are no short cuts, no matter what the techno wizard purveyors promise!

The verdict, at least for now, is that social media is a new source of data that is here to stay, and surely it can be mined for rich ideas and insights.  But it is not better, faster, richer, cheaper, or more accurate than other kinds of research.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

We can’t do this with self-administered online surveys, but there is a way to minimize error associated with respondents jumping to conclusions:  Put all clarifying instructions before asking the question, not after.  A recent study published in Public Opinion Quarterly documented that if you put instructions before the question, respondents spend more time answering because they are reading the instructions and answering more carefully.  Not surprisingly, their answers are more accurate.

Here is an example of a typical question that has clarifying instructions after the question:

The next question is about your retirement savings.

How much did you contribute to your retirement savings in 2012?  For the purpose of this question, include any IRA contributions, any pre or post-tax contributions to a workplace retirement account, plus any contributions from your employer, as well as any other savings specifically for retirement.

Here is a better alternative that puts the clarifying instructions first:

The next question is about your retirement savings.

For the purpose of this question, consider any IRA contributions, any pre or post-tax contributions to a workplace retirement account, plus any contributions from your employer, as well as any other savings specifically for retirement.  How much did you contribute to your retirement savings in 2012?

I confess that my inclination in writing survey questions is always to get to the point quickly in as few words as possible (always a good thing!)  But if I start adding qualifiers and asking for nuance, I remind myself that the respondents are no longer even listening or reading.  That’s when it is time to revise.

Need help?  We know a lot about how to ask the right questions that generate insightful answers.  Give us a call at (312) 348-6089.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

I rarely miss the world of academia after having jumped ship twelve years ago, but one thing I always appreciated was that teaching involved distilling ideas down to their simplest and most essential form.  Students often led the way for me, asking questions that forced me to clarify, deepen, and condense.  So it was again in Professor Alan Malter’s market research class for MBA students last month at the University of Illinois in Chicago.

My annual task is to offer a business-side view of market research.  But this year I asked students to imagine that you, dear readers—Versta Research’s valued clients—were standing in front of them instead of me.  What would they want to ask you about your work and about client-side research?

I was struck by how good their questions were.  None of them were naïve.  Many were questions that we ask ourselves and our clients all the time—questions that need continual probing to make research succeed.  Fundamental questions like: Why are you doing research?  What are you trying to solve?  Are you sure you need research?  How will you and your internal clients know if the answers are right?  And so on.

Here is the full list:

• What are the nuts and bolts—the everyday process—of how you support your company with research?
• What specific problems are you trying to solve?  How important are those problems?
• Do you think you need market research to solve those problems?
• How important is market research to your company?
• Is it really possible to calculate the ROI of a specific research effort?
• Where do you focus most of your research: new markets or current relationships?
• How do you measure the un-measurable, like what’s inside people’s minds?
• With so much secondary data now available, why do you collect primary data?
• Is research a continual investment, or is it something you do just when there is a specific problem?
• How are budgets for research determined—as needed vs. fixed vs. percentage of marketing budget?
• Do you think a separate market research function will exist in your company in ten years?
• How do you identify what your internal clients want and need?
• Is everything you do filled with numbers and statistics?
• Do your internal clients really believe the statistics you present?
• What qualities do you look for when hiring internal market research professionals?
• Is it better to have an internal market research person, versus relying entirely on outside companies?
• What is the most difficult part about communicating with outside research companies?
• Which problems do you outsource to research companies, and which do you keep in house?
• Does using a third party research company add to (or diminish) the credibility of your work?
• What kinds of expertise or credentials do you look for in research companies?
• When you get research back from a vendor, how do you know if it is good?

This exercise reminded me that, fundamentally, research is not about methods and numbers and stories and data, but about formulating questions—the right questions—and then finding useful answers by whatever means and methods we have available.

For next year’s talk, I am thinking it would be excellent to share with the students a joint talk by Versta Research and one or two of its client-side research partners.  Any volunteers?

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

It’s our job to deliver bad news as well as good news, right?  To tell clients what they’re doing wrong so they can fix their problems and leap to the next level of profitability, right?  Why would they spend money collecting data if they just wanted to hear how much customers love them?  In fact, why would they want to hear how much customers love them, if the research says otherwise?

A recent study published in the Journal of Consumer Research suggests some answers that may surprise you.  There are  situations in which good news can be far more useful to the recipients than bad news, even when bad news is presented constructively with specific ways to improve.  We cited this research in a recent post, “Tell Me What I’m Doing Wrong,” which was about the best kind of feedback for clients to give vendors.  But the implications go in the other direction as well:  when you have a client who wants evaluation research or customer satisfaction research, what kind of feedback is best?  Quoting again, from the research article:

“Novices are more likely than experts to seek positive feedback on their strengths and alter their behaviors and attitudes when they get such feedback, whereas experts are more likely than novices to seek negative feedback on their weaknesses and alter their behaviors and attitudes when they get this feedback.”

So when it comes to organizations or business groups, some clients need and want to hear what’s working and what they’re doing right.  It helps them define effective strategies and build organizational commitment to those strategies.  Other clients need and want to hear what’s wrong.  It helps them improve upon strategies already in place in order to reach goals to which they are already committed.

The idea of telling clients what they want to hear to has gotten a bad rap.  Most want to hear news and information that helps.  Sometimes it’s good news that helps and sometimes it’s bad.  Either way, great research moves beyond best-practice-boilerplates to deliver insights tailored to exactly what can help.  Whether it is good news or bad, that’s exactly what clients want to hear.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

It is hard to find an appropriate use for Google Surveys, because, as we outlined in a review article last fall, its capabilities are limited.  But last week we needed a quick incidence test of how many U.S. adults own a certain type of investment product.  Google Surveys seemed perfect.  It was not fast, by the way.  It took five days to collect data from 200 respondents.  Google says this is because we asked a screening question before asking about product ownership.  Even so, this survey took longer than a standard omnibus.

But what struck me most about my trial run with Google Surveys was the Creepy Factor.  It made me realize in a most uncomfortable way that Google tracks everything I do.  I knew this already, and I follow ongoing discussions about online privacy.  I have a personal g-mail account, a G+ page, and I use Google as the starting point for almost everything I do on the Internet.  I know that they track everything I do.  But it was never so creepy and apparent until I fielded a Google survey.  How was it creepy?

1.  Two months ago I was offered $75 off my first survey via PRSA (the Public Relations Society of America), a professional association to which I belong.  I learned about it through my work e-mail and followed the link for more details.  I had no use for it at the time, so decided against it and navigated away.  Bam.  A personalized coupon showed up in my (non-work) g-mail account.

2.  So last week I decided to give it a try.  I did the design, made the appropriate selections for sample size, etc.  When it came time to submit and pay, Google pulled up credit card information, asking for any updates and an authorization.  How did they have all that information?  I must have used Google Wallet sometime in distant past.  I honestly can’t recall, but having an old (expired) credit card number automatically pulled up for a service I have never used was creepy.

3.  I never created an account, logged in, or provided any type of password to manage the project or see results.  Information was instantly displayed by typing “google consumer survey” on my home page search bar.  Yes, I know it’s because I remained logged into my g-mail account on my computer.  But even at my office, being “in” does not automatically give me access to all computers, accounts, data, services and projects.  I value additional layers of security for my work and for the work I am doing on behalf of clients.

4.  Most important of all: who do you suppose owns the data from my survey?  Stupid question, right?  Google is tracking every survey I do and every survey question I field.  No doubt they are tracking every person who responded to my survey.  Google claims not to track how those individuals are answering survey questions, but surely they are using all of this data for their purposes, not mine.

I will think twice about fielding another Google survey.  It works well, and recent election data shows that the surveys are highly accurate.  But in most cases this will not out-weigh the cost of having my survey data tracked, sold, and leveraged by a company that proudly claims it is not evil.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

We’re all for ditching research when you have strong intuitions about your business and have no pressing questions that need to be answered.  But too many business leaders are now following in the footsteps of Steve Jobs who was notorious for believing that consumers can’t tell you what they want. Aggressively ignoring market research has become a strange point of pride for some.

Did Ron Johnson, formerly of Apple and now formerly of J.C. Penney, do any research before running his new company into the ground?  It’s doubtful.  Even a super simple research effort like watching people enter stores from parking lots (“Look, they’re all clutching coupons!”) might have given him some insight about how his new strategy would fare.  In fact he didn’t even have to do research.  He could have asked his managers to review research published over the last twenty years to help him understand how consumers respond to various pricing strategies.

Here is an excellent resource that should have been his starting point: Consumer Insights: Findings from Behavioral Research. It is published by the Marketing Science Institute and summarizes practical findings from academic market research.  It has an entire section on pricing with five chapters:

• Perceptions of Price Deals

• Biases in Processing Price Information

• Effects of the Internet on Consumer Price Sensitivity

• Effects of Transaction Structure on Price Perceptions & Consumption

• Perceptions of Price Fairness

These five chapters provide a quick overview of what is known when it comes to pricing and how consumers respond, based on hundreds of research studies.  Each is short and to the point (just two or three pages!).  Each outlines insights, the evidence base, and managerial implications.

Shucks, Mr. Johnson could have hired Versta Research and we would have happily synthesized these findings and presented them in one afternoon.  We would have wrapped up our presentation by acknowledging the limitations of market research, but suggesting that while his intuition about transforming JCP may be right, we ought to ask his customers ahead of time how they might respond.

Why?  Because high quality research, both qualitative (such as focus groups) and quantitative (such as surveys) certainly can provide deep insights into how customers and audiences think, perceive, react, and respond to products, services, and opportunities.  Almost certainly JCP’s customers would have told us that they would take their coupons and cash elsewhere.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research

RELATED POSTS

Can a Focus Group Save Spider-Man?

Focus Groups Save Spider-Man!

Conjoint Analysis Helps Apple Win $1B in Lawsuit

]]> http://www.verstaresearch.com/blog/research-could-have-spared-j-c-penney/feed/ 0 http://www.verstaresearch.com/blog/tell-me-what-im-doing-wrong/ http://www.verstaresearch.com/blog/tell-me-what-im-doing-wrong/#comments Thu, 11 Apr 2013 21:59:36 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=2722

This is my dog Lancelot on his first day at his new home

If you have ever done “clicker training” with a dog, you know how amazingly effective positive rewards are in training, versus the old-fashioned method of “correction” and negative feedback.  Identify and reward the behavior you want, and you can teach an old dog new tricks within hours.  It works for people too, which is why so many HR and business experts talk about the power of praise in teaching and motivating employees.  I can relate to this.  I love my work most when our clients offer generous praise and tell us that we exceeded their expectations.

But I also worry:  Is the work really good?  Is it as good as it can be?  Is it finding its way up to other managers and decision makers, and is it helping them, too?  Maybe this reflects professional insecurity, but new research published in the Journal of Consumer Research shows it is common as people gain expertise in their fields.  Quoting the study published a few months ago:

“Novices are more likely than experts to seek positive feedback on their strengths and alter their behaviors and attitudes when they get such feedback, whereas experts are more likely than novices to seek negative feedback on their weaknesses and alter their behaviors and attitudes when they get this feedback.”

So while I appreciate knowing that our work is good, I appreciate it even more when a client tells me how it could be better.  We want our work moving from good to great, and for that, we need clients like you telling us how to make it better.

Part of me is the eager-to-please puppy who needs just a few positive strokes to make me wag my tail.  But the “expert” in me is always asking for more: “What could make this even better and more useful?”  Your straight-on critiques will help us transform simple tricks into really spectacular research findings that will amaze you and your managers.

By Joe Hopper, Ph.D.

Versta Research Website

More articles from Versta Research