Even if you are not involved in political polling, it is worth paying attention to the methods and best practices of political pollsters.  One reason is that few other areas of research offer a way to completely validate one’s methods.  Pollsters are using sampling and survey methods to predict the behaviors of a much larger population.  Then in just one day that population behaves, we get a near-perfect count of exactly how they behaved, and we know whether the methods worked.

Several industry colleagues have recently been debating the merits of calculating and reporting “margins of error” in political polling, and pointed us to some surprising data from The New York Times:

[The New York Times has compiled] a database consisting of thousands of primary and caucus polls dating back to the 1970s.  Each poll contains numbers for several candidates, so there are a total of about 17,000 observations. How often does a candidate’s actual vote total fall within the theoretical margin of error?  The answer is, not very often. In theory, a candidate’s actual vote total should fall outside the margin of error only 5 percent of the time [given that political polls report margins of error using a 95% confidence interval]. In reality, the candidate’s vote total was outside the margin of error 65 percent of the time! Part of this is because the database includes some polls conducted months before the actual voting took place. But even if you restrict the analysis to polls conducted within the final week of the campaign, about 40 percent of the vote totals fell outside the margin of error — eight times more often than is supposed to happen if you could take the margin of error at face value.

This does not mean that the polls were wrong, predicting wins for losing candidates and vice versa.  Rather, it means that the estimates were not as precise as the stated margins of error would have a reader believe.

The problem is that “margins of error” are based on a statistical theories that almost never line up with the messy reality of our world.  Margins of error make a number of assumptions which are rarely true in practice, including:

• Respondents are selected through simple random sampling

• All those sampled participate in the survey

• Sampling error is the only source of survey error

Indeed, Versta Research usually recommends to clients who publish survey research that they not report margins of error because the concept (and the calculations) are seriously misleading and flawed.

Calculating margins of error and looking at statistical significance should be used not because they give accurate or “scientific” predictions, but because they provide useful summary measures of how much variability there is in the data given the sample size and other critical factors that can affect one’s estimates.  At Versta Research, this helps us better interpret data and better assess what matters.  That, in turn, allows us to tell a story with the data that does not overreach or misrepresent what is going on.

–Joe Hopper, Ph.D.

But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates.  Here are some key takeaways from the research:

• A pre-notification postcard sent by mail will significantly improve response rates to your online survey

• Offering a token cash incentive (a couple dollars) sent by mail in advance will significantly improve response rates to your online survey

• As a final step, offering a mail survey to non-respondents will boost response rates even further

So why even bother with online surveys, you may wonder?  Indeed, even with all this, response rates are no better than conducting a mail-only survey.  The answer is that online surveys have a number of advantages worth keeping.  Programmed skip logic and constraints ensure no missing data or misinterpretation of skip patterns.  Data are entered into a database real-time, eliminating errors from manual data entry or optical scanning.  The process is faster, more efficient, and usually less expensive.

While many people overestimate the importance of response rates on a survey’s validity, there is no doubt that higher response rates are better in terms of cost, efficiency, and feasibility.  So if you are thinking about deploying an online survey to your customers or members, it is probably worth investing in some old fashioned mail-based outreach.

Need help?  We can be reached by phone (312-348-6089), or Internet, or mail (919 Forest Ave, Evanston, IL  60202).  Whichever way you contact us, you’ll get a 100% response rate within a few hours.

—Joe Hopper, Ph.D.

Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process.  We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.

But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing.  That’s an unusually high ratio of “suspends” as we call them.  So what was the problem?  Were we just getting lousy respondents who did not want to seriously participate in a survey?  Was the survey was too difficult, tedious, boring, or confusing?  One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.

The story in this data: Something is wrong with your survey

Nearly every respondent who quit got close to finishing and then dropped off at exactly the same point, which was odd because the most difficult questions were earlier in the survey.  In fact, the question where most ended up quitting was an interesting drag-and-drop interactive exercise.  Ah, that was the problem.  The programming for the interactive piece was flawed and respondents were being kicked out.

It wasn’t without a good deal of angst that the programming team tested, re-tested, and confirmed the error.  Everybody involved in this effort resisted: the sampling provider, the programmers, the survey tool developers, the questionnaire designers—they have all done this work hundreds of times, so there can’t be anything wrong with their piece of it, right?

Let the data speak.  It will tell you where the mistakes are. There are lots of places and moments where things may go wrong.  If the top people responsible for the project do not have immediate visibility into the data, they are unable to suggest smart solutions, and bad research will just keep happening.

—Joe Hopper, Ph.D.

Last week, theater critics roundly panned the show, calling it among the worst Broadway shows ever. The New York Times described is as “so grievously broken in every respect that it is beyond repair.”  But days after these negative reviews, a market research firm was brought in, soliciting volunteers to help fix the show:

If you love Broadway shows, we would like to invite you to be a test audience member and participate in either Act 1 (first part of show) or Act 2 (second part of show) of the all new Broadway musical “Spider-Man: Turn Off the Dark.”

We will be asking you to fill out a survey after the performance, plus stay for a 15 minute discussion in the Spider-Man V.I.P. room. In turn for your feedback and participation you will receive a Spider-Man goodie bag worth over $60.00, on us.

Can market research really rescue the show?  Possibly.  High quality research, both qualitative (such as focus groups) and quantitative (such as surveys) can provide deep insights into how customers and audiences think, perceive, react, and respond to products, services, and opportunities.

If market research can help rescue the show, here are two suggestions we offer that will be critical to keep in mind:

1.  Keep the focus groups focused. Too often clients or managers want their customers to tell them what would make them love their product.  That’s not something customers can reliably tell you.  They can tell you what they care about, what products they purchase and why, what needs and problems they are trying to solve, and what frustrations they face.  Our advice generally: Keep the focus group participants focused on the things that matter to them, then connect the insights gained to the products being offered.

2.  Ask only about things that can be fixed. For all the talk about “actionable” research, there is a great deal of research that offers supposedly actionable ideas that clients or managers simply cannot and will not act upon.  How does the audience feel about Bono’s music for the show?  Unless management is prepared to dump Bono or have him re-write (again), don’t ask.  The key is to determine during the research design which specific decisions can and cannot be made, and then to focus the research on collecting data that lends specific insight to those decisions.

Asking market research to help rescue a $65 million investment at the last minute seems like a tall order, but fortunately great research does not take super-human powers.  At Versta Research we rely on brains, experience, thoughtfulness, and a commitment to working closely with clients to ensure that we deliver insightful stories that are understood and acted upon.

So what will become of Spider-Man?  Will he die under the crushing weight of debt and harsh reviews?  Or will the critics groan in agony as Americans happily pay for the worst show on earth?  Stay tuned for the next exciting episode of “Can a Focus Group Save Spider-Man?”

—Joe Hopper, Ph.D.

Though relatively low response rates can yield statistically sound data (see our article, Do Response Rates Really Matter?) you need to have some data to work with.  One organization we recently advised found itself with just two respondents for a do-it-yourself survey they launched, and unfortunately they lost the opportunity to go back into the field for another shot.

Based on our experience and what we have learned from the ever-evolving scientific literature on survey methods, here are three important steps you can take to improve your survey response rates:

1. Communicate. When you are ready to launch a survey, it is critical that you provide basic information to the people you are asking to fill it out, including:

• What the survey is about, and why you are fielding it
• Why they, in particular, were chosen to participate
• Why it is important for them to participate
• How the information will be used and reported
• With whom the information will be shared
• How long the survey will take for them to complete
• Whether their participation is anonymous or confidential
• What they will get in return

You need to keep your survey invitation short and to the point, but be sure to include these needed pieces of information.

2. Track.  It is best if you have way of knowing who has responded to your survey so that you can follow up with reminders to those who have not responded.  Even though low response rates can often yield good data, research shows that people who respond quickly or who respond to first invitations vs. second or third invitations are often unique and may skew your findings.

Depending on the survey, we typically suggest up to two or three reminders beyond the initial invitation.  If possible, vary the mode (using phone, postcards, e-mail, etc.) and for the final reminders convey some sense of urgency (when the study is closing, for example, with a reminder of how valuable their input is).

Of course tracking means that participation is not anonymous.  You’ll need to weigh your promise of confidentiality against this.  One solution is to work with a third-party research firm like Versta Research that adheres to CASRO and AAPOR guidelines on ethics and privacy.  This keeps you “arms-length” from the research, while still allowing a trusted source to track and follow up as needed.

3. Reward.  Whenever possible, we suggest offering an incentive for people to participate in surveys.  Why?  Because it demonstrates that you are serious about needing them to participate, that you truly value their input, and that you recognize their time as being valuable.

In most cases the reward is a small token of thanks.  It can take many forms: direct cash, internet gift cards, entry into a sweepstakes for a large prize, and so on.  For B2B research, information is often a more valuable incentive than money.  Business decision makers usually have an invested interest in the topic you are studying (that’s why you want them in your survey) and offering them a summary of some key data may be all the incentive they need to participate.

Need help?  We have experience fielding hundreds of surveys, everything from consumer surveys and membership surveys to top-level business executive surveys – in person, by phone, online, fax, and by mail.  Give us a call, and we are happy to advise you on sampling strategies, response rates, and other key issues that will affect the rigor of your survey.

—Joe Hopper, Ph.D.

Is this possible?  No.  It unfortunately confuses the tools of market research and public opinion polling with the interpretation and outcomes of research.  To be sure, better tools and technology help us do our work faster, smarter, and cheaper.  They bring sophisticated tools into the hands of smaller organizations who can then help businesses and media outlets with nimble and cost efficient solutions.  Market researchers are benefiting enormously from these new technologies and tools.

But ultimately these easy-to-use applications that deliver real time data and pie-charts give us … well, just data and pie-charts.  They don’t give us insights.  For insights, we need smart people who bring expertise, brainpower, and thoughtful creativity to the research effort — people who know how to design and implement studies, and then interpret and communicate information to answer critical questions.  When managers and clients see their own desktops stuffed with “auto-alerts-sent-to-key-stakeholders-enterprise-wide” from our newest suite of tools, they need people to answer challenging questions like, “So what?  Is this really true?  What does it mean?  How should I proceed?”

These are the kinds of questions we can help you with.  We can help you sort through the latest tools and can help you implement them within  your organization.  But don’t forget that tools are a means something bigger and more fundamental, like a story with a context, a puzzle, a question that needs to be answered.

—Joe Hopper, Ph.D.

In 2004, Sandra Rathod and Andrea la Bruna conducted experiments to examine the effects of survey length on response rates, drop-out rates, respondent fatigue, speed of answering, and data quality.  In 2009, researchers at Survey Sampling International replicated the experiments and presented their findings at the recent 2010 ARF Re-Think Conference.

Their findings confirm what we know from the research five years earlier:

1.  With longer surveys, respondents get more fatigued, pay less attention, and increase their speed of response as they progress through the survey

2.  With longer surveys, data quality declines as the survey length increases (questions are skipped, open-ends are less complete, less effort is devoted to questions at the end of the survey compared to the start)

3.  With longer surveys, respondents are more likely to cheat by answering untruthfully to avoid multiple follow-up questions

What is a “long” survey?  The consistent answer from this research and other research in years past is that surveys over 20 minutes are too long.  In our experience, you can nearly always get detailed data that gives you deep insights into your questions with surveys under 20 minutes.  It is just a matter of focusing on the right questions (and only those questions) and then skillfully designing the survey instrument to answer those questions.

Need help?  Give us a call.  We would be happy to help you find the right focus and an efficient research design that delivers high quality data to answer your critical questions.

—Joe Hopper, Ph.D.

Various surveys, all rigorously  done, showed support for including a public option in health care reform varying from 44% to 66%.  All asked neutral, unbiased questions, and all provided appropriate answer categories.  But each used slightly different wording, which dramatically affected the results.  Comparing the proposed reforms to the Medicare program boosted support, while referring to it as a government-run insurance plan killed support.  If you are the marketing or communications team charged with the task of selling health care reform, these are exactly the kinds of conflicting results that help you.  They give you deep insight about how to position your product or service, and what kind of messaging you need to develop.

Of course this example also highlights the critical importance of questionnaire design when launching a survey.  Ostensibly similar efforts to measure the same thing can lead to different answers.  It is easy to write questions and field surveys, but not so easy to nail down exactly what needs to be measured, and not so easy to anticipate how your audience will respond to the nuances of words and design.

The lesson?  First, pay a great deal of attention to question wording, and get input from many members of your team: survey experts, outsiders, insiders, the brand team, your business executives, and so on.  Second, when you get results that don’t make sense because they contradict other data, look for opportunities where this conflict can help enhance your understanding, because usually it can.  If you need help with either, give Versta Research a call.  We are happy to bounce around ideas and provide some initial thinking at no charge.

—Joe Hopper, Ph.D.

Keeping your survey efforts honest is critical.  Sincere efforts to document issues through rigorous surveys succeed, and when done correctly, people are influenced by numbers.  But trying to shock or manipulate with sensationalist, biased, and phony data gets you only so far.  For a great survey, keep it honest, rigorous, and valid, and carefully control for bias.  We, at Versta, can help you succeed with this.

-Joe Hopper, Ph.D.

Phone vs. Online Surveys: Strengths & Weaknesses

As reflected in the table above, the article is rather technical.  But there are two key summary points worth learning from the study:

One, online surveys fielded through panel providers may not fully represent the population.  But we know from other research that they can come close, so the key is to understand the sources of potential bias and adjust for them as needed.

Two, online surveys fielded through panel providers may result in more accurate data than other types of surveys.  There are two reasons for this.  First, online panelists are more engaged in the process and interested in the topics, so they exhibit less cognitive laziness.  Second, online surveys feel more private, so respondents provide answers that are more honest.  This is in contrast to phone surveys in which people usually want to be nice and therefore provide socially desirable answers, even if it’s not how they really feel.

Which should you do, a phone survey or an online survey?  And what type of sampling should you employ, probability or non-probability?  It depends on your objectives.  Each of these issues can matter a lot or not at all depending on the specific questions you are trying to answer.  We would be happy to help you think through your options and the strengths of each.  Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

-Joe Hopper, Ph.D.