The cartoon brings to mind three lessons worth pondering:

1. Customer satisfaction research is often “not fun”—but it can be
2. Internal data can be a goldmine of insight and there is often a lot of it lying around
3. Ethical considerations dictate that just because research can be done does not mean it should be done

1. Customer satisfaction research is often “not fun.” Why?  In our view it is because so many customer satisfaction surveys are done for the wrong reasons, focus on the wrong issues, and ask about the wrong people.  What could be more boring—and more unenlightening—than tracking the percentage of customers who like (or dislike) you month after month and year after year?

To make customer satisfaction research far more interesting and useful, focus not on your products and not on you, but instead on your customers, what they need, and what will delight them.  Then assemble a team that is committed to thinking hard and adding value to every report they deliver.  Tracking studies can be fun if focused on the challenge of consistently providing an insightful story.

2. Internal data can be a goldmine of insight.  Organizations often send out survey after survey without ever taking stock of what they have already learned, and without considering the wealth of data they have internally.  Yet, the volume of useful, untapped internal data that most organizations have is astonishing.  You can learn a great deal about your customers and satisfaction, for example, simply by analyzing historical data.  Who are you losing or gaining as customers?  What segments, industries, or geographies characterize them?  You don’t always need surveys to answer these questions.

3. Ethical approaches to research are essential.  Data should never be collected and archived without the people supplying that data knowing about it and knowing how it will be used.  This is a fundamental ethical requirement all social scientific research, and it extends to all market research and public opinion polling as well.  We recommend that research teams adhere strictly to the strictest standards of ethics and privacy as outlined by The Council of American Survey Research Organizations (CASRO) and the American Association of Public Opinion Research (AAPOR).

Need help on any or all of these fronts?  Definitely stay away from Mordac the Preventer of Information Services, and give us a call instead.

—Joe Hopper, Ph.D.

This shift in telephone usage is critically important to survey research, because telephone surveys have set a gold standard for rigorous research over the last two decades.  Given the trends, most survey researchers would agree that cell phone sample must now be included in the most rigorous research designs for an RDD survey.  But doing so introduces new difficulties in design, sampling, weighting, and cost, and our collective experiences are not yet sufficient for the industry to  agree on a set of “best practices” when it comes to including cell phones in survey research.

Our leading industry association, AAPOR (The American Association of Public Opinion Research) recently published a comprehensive update from the AAPOR Cell Phone Task Force that offers a good overview of the issues, complications, guidelines, and considerations every researcher should take into account when designing a telephone survey of the general population.  Briefly, it covers and highlights the following issues:

Coverage and Sampling: It is increasingly difficult to reach young men and minorities via landlines, so cell phone coverage is critical.  But using dual and overlapping sampling frames (both landline and cell phone) introduces new methodological complications in sampling.

Nonresponse: Response rates are lower among cell phones users, though response rates among landline users continue to decline as well.

Measurement: Data gathered via cell phone interviews is generally not of lower quality (despite concerns about audio quality, talking in public places, and distractions from multitasking).

Weighting: If dual and overlapping sampling frames are used (to include both landlines and cell phones) then data need to be statistically weighted to account for the complicated differences in probabilities that each person in the population will be recruited into the survey.

Legal and Ethical Issues: Including cell phones in surveying introduces a new set of legal, regulatory, and ethical considerations regarding auto-dialers, texting, caller ID, and do-not-call lists.

Operational Issues: Cell phone interviewing requires a unique set of protocols for recruiting, call-backs, and interviewing, all of which require special training for the people conducting the interviews.

Cost Issues: Including cell phones at least doubles the cost of a survey, and many times may triple or quadruple the cost.

Should you include cell phones in your survey?  It depends on who you are trying to reach and for what purposes.  When it comes to effective surveying, there are no absolutely right answers about cell phones vs. landlines, just as there are no absolutely right answers about online vs. phone surveys.  Moreover, the rapidly changing landscape of how people communicate means that the best answer today may be different in six months.

Need some help thinking through the options and figuring out the best approach for your research?  Feel free to give us a call.  We have decades of experience and are committed to helping our clients take full advantage of important changes and innovations in research.

—Joe Hopper, Ph.D.

So whatever you do, please don’t take Dogbert’s approach to conducting a survey:

Versta Research believes strongly in protecting the privacy of all research respondents.  We adhere to the strictest standards of ethics and privacy as outlined by The Council of American Survey Research Organizations (CASRO), the American Association of Public Opinion Research (AAPOR), and other industry organizations to which we belong.  The following two paragraphs nicely summarize our views and approach:

“Since individuals who are interviewed are the lifeblood of the Survey Research Industry, it is essential that Survey Research Organizations be responsible for protecting from disclosure to third parties–including Clients and members of the Public–the identity of individual Respondents as well as Respondent-identifiable information, unless the Respondent expressly requests or permits such disclosure.”  From the CASRO (Council of American Survey Research Organizations) Code of Standards and Ethics for Survey Research.

“Unless the respondent explicitly waives confidentiality for specified uses, we shall hold as privileged and confidential all information that could be used, alone or in combination with other reasonably available information, to identify a respondent with his or her responses. We also shall not disclose or use the names of respondents or any other personally-identifying information for non-research purposes unless the respondents grant us permission to do so.”  From the AAPOR (American Association for Public Opinion Research) Code of Professional Ethics & Practice.

In the spirit of tomorrow’s holiday, we extend warm thanks to the many people who participate in our research, and we pledge to continue our work to protect research from the likes of Dogbert.  We also promise not to call you tomorrow during Thanksgiving dinner, hoping that you’ll answer just a few short questions for our survey.

Happy Thanksgiving!

—Joe Hopper, Ph.D.

Should you state a survey’s margin of error in your press release when pitching a story to the media?  In our view, the answer is no.  Why not?  Because margins of error refer to sampling error only, not about the overall accuracy or error of the survey itself.

But how many readers of your news story understand what sampling error is?  For that matter, how many researchers understand what sampling error is?  It was only after four college semesters of advanced mathematical statistics that I finally “got it.”  More importantly, how many readers understand that there are many other potential sources of survey error?  Most undoubtedly assume that all error is somehow accounted for when you confidently proclaim the margin of error being ±4%.  (Or, more absurdly, ±3.6% or even ±3.57% — examples of phony accuracy like this are all too easy to find.)

In short, margins of error are misleading because they deal with only one source of error.  They convey a false sense of accuracy.  And they should not be used.  This idea is not always popular among colleagues and clients.  But take comfort:  One of the giants of public polling, Harris Interactive, refuses to report margins of error in its work, for precisely the reasons outlined above.  Here we quote their methodological statement that accompanies every press release and report they issue:

All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading. All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100% response rates. These are only theoretical because no published polls come close to this ideal.

We agree 100% with this statement, and advise our clients accordingly.  In our view, the polling industry  has an obligation to re-educate the media about this issue as well.  Transparency requires that you (and we) report samples size, composition, and source.  But transparency also requires a willingness to do away with misleading statements, no matter how much a reporter wants them.

—Joe Hopper, Ph.D.

As of this writing, Versta is one of 67 prominent survey and polling organizations currently supporting the initiative. The initiative is designed to create protocols and recognition for regular disclosure of methods when survey organizations conduct public polls.

Trust in survey research and public polls is critical not only for the public good, but for the important work that we do for our clients.  Our clients rely on us to give them accurate data and truthful assessments. They can’t make good decisions without that. And they rely on us to ensure that when they share data with the public, it is backed by the most rigorous and defensible methods available in the social sciences. Transparency is one means of ensuring that all organizations are continually held to the highest standards of rigor and accuracy.

AAPOR began the initiative in 2009 “to encourage routine disclosure of methodological information from polls and surveys whose findings are released to the public.” The organization lists the following as key elements of the efforts now underway:

• The association’s public recognition of excellence in transparency;
• A system for collecting and archiving disclosure information;
• Education for organizations in how to make transparency a routine part of their work;
• Outreach to survey sponsors, users of survey data and the public to reinforce the value of transparency and to publicize the organizations that participate in the initiative;
• Collaboration with other associations and academic disciplines in these activities.

One benefit of working with an organization like Versta Research is that you can fully trust the outcomes of the research, and you can trust that we have the expertise to defend it.  You can also trust that we work diligently within the industry to protect the value of our collective investments in good research.

—Joe Hopper, Ph.D.

One of the tenets of rigorous market research is that respondent confidentiality is key.  Why?  Primarily because it benefits you as a client.  To make smart decisions, you need honest and thoughtful information from your customers and constituents.  In most cases, your customers want to give us that information (because they want you to do a better job) as long as it won’t come back to haunt them.

Versta Research adheres to the ethics guidelines for privacy as outlined by CASRO, AAPOR, and other industry organizations to which we belong.  The following two paragraphs nicely summarize our views and approach:

“Since individuals who are interviewed are the lifeblood of the Survey Research Industry, it is essential that Survey Research Organizations be responsible for protecting from disclosure to third parties–including Clients and members of the Public–the identity of individual Respondents as well as Respondent-identifiable information, unless the Respondent expressly requests or permits such disclosure.”  From the CASRO (Council of American Survey Research Organizations) Code of Standards and Ethics for Survey Research.

“Unless the respondent explicitly waives confidentiality for specified uses, we shall hold as privileged and confidential all information that could be used, alone or in combination with other reasonably available information, to identify a respondent with his or her responses. We also shall not disclose or use the names of respondents or any other personally-identifying information for non-research purposes unless the respondents grant us permission to do so.”  From the AAPOR (American Association for Public Opinion Research) Code of Professional Ethics & Practice.

If you need help deciding whether your research plan and approach appropriately protect the privacy of those participating in the research, we would be happy to advise.  We have deep experience with these issues in market research, membership research, political polling, and academic IRB (Institutional Review Board) mandates governing research under federal funding.

—Joe Hopper, Ph.D.

Two researchers recently came up with methods of testing whether polling data is legitimate in a case where a research firm is accused of falsifying publicly released data. The researchers examined the last digit of the numbers released by the polling firm, and calculated the probability of seeing these digits in their specific combinations and frequencies if the polls were legitimate.  It was an ingenious way of using sophisticated statistical methods to examine data where the accused firm refused to disclose details about its methods.  The New York Times calls it “forensic polling analysis” and notes that in this case, the odds suggest that the polling data were fraudulent.

There are a number of leading industry organizations that have outlined ethical guidelines and principles of disclosure for survey research, including:

• AAPOR, the American Association of Public Opinion Research
• NCPP, the National Council on Public Polls
• CASRO, the Council of American Survey Research Organizations

Versta Research believes these guidelines are important to the health and integrity of research.  We adhere to them and encourage our clients to do so as well.  If you are unsure about your ethical obligations when releasing polling data to the public, give us a call and we can help you make sense of the guidelines.

–Joe Hopper, Ph.D.