Surveys have gotten easier and easier, and worse and worse, and perhaps the two go hand in hand.

With so many powerful and inexpensive tools available, even people challenged by science and statistics can now distribute polls, crunch numbers, extract metrics and report “insights.” So they churn out lots and lots of surveys (and we do, too). Some of the results are good, some are okay, many are terrible. While I like to think that none of our work is terrible, sometimes we are tasked with the impossible: trying to fix or improve a survey when other parts of the work are fundamentally flawed.

But it’s worth understanding what makes for great surveys versus okay surveys versus terrible ones because we learn from the disasters as well as the triumphs. The disasters are one star surveys. Surprisingly many organizations do them, and we stumble upon them every day. The triumphs are five star surveys. Few organizations do them (and for good reasons) but they provide superlative standards that we can emulate.

In between the one star surveys and the five star surveys there are two star, three star, and four star surveys as well. Here are the defining characteristics of each, proceeding from best to worst:

★★★★★ 5 Star Surveys

Five star surveys are the gold standard for what all surveys could be if only we had the time and money. Think U.S. Bureau of Labor Statistics, or the U.S. Census Bureau’s American Community Survey. Several large, academic research centers, most often funded through large federal grants, produce surveys of this caliber. They are characterized by:

• Rigorous scientific protocols with surveys that are painstakingly designed, tested, and executed in all phases of work including sampling, questionnaire wording, scale construction, recruitment, and data management.
• Academic expertise that draws from the very top tiers of social scientists who are defining and advancing the boundaries of shared knowledge.
• Important topics that are widely useful to both private businesses and public institutions, and that establish critical benchmarks for other research and measurement.
• Huge budgets and long timelines, as in millions of dollars and many months if not years of planning and fieldwork to execute.

Even if you never come close to fielding a five star survey yourself, it is important to know about them. They provide amazing public resources you can use in your own work, and they set methodological standards against which all other surveys should be measured.

★★★★☆ 4 Star Surveys

Four star surveys are the five star wannabes, and that’s a good thing. They come as close as they can, with $30K instead of $3M, and with eight-week timelines instead of two years. They are characterized by:

• Specialized expertise with personnel trained in academic five star research and who continue to follow and apply important trends in social scientific methods and knowledge.
• Applied rigor that balances theory and practice, and carefully weighs the costs and benefits of competing priorities when it comes to sampling and fieldwork.
• Relevant topics that are carefully linked to internal business objectives, or that answer specific questions of general public interest.
• Affordable budgets and timelines but not super cheap and super fast, because the guiding needs are rigor, credibility, and defensibility.

If you are serious about investing in research, have important decisions to make, or important markets to reach, then a four star survey ought to be your goal. Who does four star surveys? The Pew Research Center, Versta Research, Gallup, Harris, Ipsos, GfK, NORC, and other research groups (maybe your group, too?) that have a sincere focus on bringing scientific social research to business.

★★★☆☆ 3 Star Surveys

Three star surveys offer rough, quick, and dirty quantitative views of a market. They can be valuable in the right circumstances. But most three star surveys masquerade as being more, in part because those who are conducting the surveys are not fully versed in the statistical and fieldwork issues that underlie more rigorous research. We consult for many firms who do these surveys, and we are always surprised that few understand how flimsy they are.

Three star surveys are characterized by one or more of the following:

• Weaker expertise because the firms involved usually specialize in marketing, brand innovation, strategic planning, advertising, qualitative research, or technology.
• Narrowly focused on issues that are specific to an immediate need but lack relevant and robust data that can speak to larger business questions or objectives.
• Cheaper and faster because there is minimal effort to build a careful design, manage sample, control fieldwork quality, and clean data.
• Good enough (one hopes) but inexact, and often just plain wrong, because shortcuts are embedded in the design, errors are overlooked, analysis is automated and simplified.

We did one of these surveys two months ago, but our objective was clear and limited: we needed a rough idea of brand awareness and purchase frequency to estimate costs for a larger survey about new products. Indeed, three star surveys can deliver a general sense of your market and give direction for better research (if needed), but please don’t rely on them for decision-making or for building credibility.

★★☆☆☆ 2 Star Surveys

A survey we came upon recently described having to recruit “like crazy” to achieve “a statistically significant dataset.” How did they do it? By reaching out to all of their friends and contacts, which was cool because contacts turn into leads and sales. And then the survey results “drive traffic, shares, and likes” which was the goal of their research.

This is a two star survey. The authors get credit for picking a topic they believed in and wanted to learn about. But otherwise, two star surveys are characterized by:

• Zero expertise as evidenced by nonsense statements about methods, samples, margins of error, and statistical significance.
• Biased samples with an obsessive focus on size, no understanding of representation, and seemingly no awareness of multiple sources of error.
• Numbers-focused, with statistics reported to absurd levels of precision, weak interpretations of what numbers might mean, and little or no depth about why the data matter.
• Gimmickry that goes beyond genuine thought leadership and that moves dangerously toward SUGGING and FRUGGING with surveys that are primarily tools to persuade and sell to people.

If you care about quality, two star surveys are, in some ways, more depressing than one star surveys. They perpetuate ignorance and misunderstanding. They bumble along, ignore best practices, and declare false conclusions. But they are wrapped in sophisticated language that sounds semi-real. In no circumstances should you do them.

★☆☆☆☆ 1 Star Surveys

If two star surveys are so bad, is there anything left for a one star survey? Yes. One star surveys embody willful disregard, even mockery, of what surveys can and should be. They deliberately employ worst practices to grab attention from respondents or from an audience.

You know those offensive questionnaires you get from your congressional representatives asking if you support a strong America or a weak one? Those are one star surveys. They are characterized by:

• Willful rejection of rigor with phony questions designed to elicit already-known answers, and with convenience-based or deliberately biased sampling.
• Unimportant topics or questions about things like space aliens invading America during presidential elections, designed to be fun or humorous or simply to score political points.
• Chart-junk graphics that are distorted, cartoonish, or silly, and that violate multiple principles of effective graphic design for the display of quantitative information.
• Easy to execute, fast, and cheap because there are no governing principles of sampling, data quality, coding, or effective analysis.

Politicians aren’t the only ones promulgating one star surveys. We saw one several weeks ago in the New York Times Magazine about cursing in conversation. I have the print version in front of me, but it seems to be absent from any kind of online archive. Maybe the New York Times staff, too, realized how embarrassingly bad it was?