Is this possible?  No.  It unfortunately confuses the tools of market research and public opinion polling with the interpretation and outcomes of research.  To be sure, better tools and technology help us do our work faster, smarter, and cheaper.  They bring sophisticated tools into the hands of smaller organizations who can then help businesses and media outlets with nimble and cost efficient solutions.  Market researchers are benefiting enormously from these new technologies and tools.

But ultimately these easy-to-use applications that deliver real time data and pie-charts give us … well, just data and pie-charts.  They don’t give us insights.  For insights, we need smart people who bring expertise, brainpower, and thoughtful creativity to the research effort — people who know how to design and implement studies, and then interpret and communicate information to answer critical questions.  When managers and clients see their own desktops stuffed with “auto-alerts-sent-to-key-stakeholders-enterprise-wide” from our newest suite of tools, they need people to answer challenging questions like, “So what?  Is this really true?  What does it mean?  How should I proceed?”

These are the kinds of questions we can help you with.  We can help you sort through the latest tools and can help you implement them within  your organization.  But don’t forget that tools are a means something bigger and more fundamental, like a story with a context, a puzzle, a question that needs to be answered.

—Joe Hopper, Ph.D.

In 2004, Sandra Rathod and Andrea la Bruna conducted experiments to examine the effects of survey length on response rates, drop-out rates, respondent fatigue, speed of answering, and data quality.  In 2009, researchers at Survey Sampling International replicated the experiments and presented their findings at the recent 2010 ARF Re-Think Conference.

Their findings confirm what we know from the research five years earlier:

1.  With longer surveys, respondents get more fatigued, pay less attention, and increase their speed of response as they progress through the survey

2.  With longer surveys, data quality declines as the survey length increases (questions are skipped, open-ends are less complete, less effort is devoted to questions at the end of the survey compared to the start)

3.  With longer surveys, respondents are more likely to cheat by answering untruthfully to avoid multiple follow-up questions

What is a “long” survey?  The consistent answer from this research and other research in years past is that surveys over 20 minutes are too long.  In our experience, you can nearly always get detailed data that gives you deep insights into your questions with surveys under 20 minutes.  It is just a matter of focusing on the right questions (and only those questions) and then skillfully designing the survey instrument to answer those questions.

Need help?  Give us a call.  We would be happy to help you find the right focus and an efficient research design that delivers high quality data to answer your critical questions.

—Joe Hopper, Ph.D.

Various surveys, all rigorously  done, showed support for including a public option in health care reform varying from 44% to 66%.  All asked neutral, unbiased questions, and all provided appropriate answer categories.  But each used slightly different wording, which dramatically affected the results.  Comparing the proposed reforms to the Medicare program boosted support, while referring to it as a government-run insurance plan killed support.  If you are the marketing or communications team charged with the task of selling health care reform, these are exactly the kinds of conflicting results that help you.  They give you deep insight about how to position your product or service, and what kind of messaging you need to develop.

Of course this example also highlights the critical importance of questionnaire design when launching a survey.  Ostensibly similar efforts to measure the same thing can lead to different answers.  It is easy to write questions and field surveys, but not so easy to nail down exactly what needs to be measured, and not so easy to anticipate how your audience will respond to the nuances of words and design.

The lesson?  First, pay a great deal of attention to question wording, and get input from many members of your team: survey experts, outsiders, insiders, the brand team, your business executives, and so on.  Second, when you get results that don’t make sense because they contradict other data, look for opportunities where this conflict can help enhance your understanding, because usually it can.  If you need help with either, give Versta Research a call.  We are happy to bounce around ideas and provide some initial thinking at no charge.

—Joe Hopper, Ph.D.

Keeping your survey efforts honest is critical.  Sincere efforts to document issues through rigorous surveys succeed, and when done correctly, people are influenced by numbers.  But trying to shock or manipulate with sensationalist, biased, and phony data gets you only so far.  For a great survey, keep it honest, rigorous, and valid, and carefully control for bias.  We, at Versta, can help you succeed with this.

-Joe Hopper, Ph.D.

Phone vs. Online Surveys: Strengths & Weaknesses

As reflected in the table above, the article is rather technical.  But there are two key summary points worth learning from the study:

One, online surveys fielded through panel providers may not fully represent the population.  But we know from other research that they can come close, so the key is to understand the sources of potential bias and adjust for them as needed.

Two, online surveys fielded through panel providers may result in more accurate data than other types of surveys.  There are two reasons for this.  First, online panelists are more engaged in the process and interested in the topics, so they exhibit less cognitive laziness.  Second, online surveys feel more private, so respondents provide answers that are more honest.  This is in contrast to phone surveys in which people usually want to be nice and therefore provide socially desirable answers, even if it’s not how they really feel.

Which should you do, a phone survey or an online survey?  And what type of sampling should you employ, probability or non-probability?  It depends on your objectives.  Each of these issues can matter a lot or not at all depending on the specific questions you are trying to answer.  We would be happy to help you think through your options and the strengths of each.  Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

-Joe Hopper, Ph.D.

How does it save you money?  The biggest cost in conducting a survey is getting it set up and then finding a statistically representative sample of respondents to answer the questions.  Once you’ve got them on the phone or online answering survey questions, it is costs little to ask them “just one more question.”  You want to avoid asking questions for more than fifteen minutes, but suppose that you have fifteen questions you want to ask, and Acme Sports has fifteen questions, and Best Wireless has another fifteen questions.  Rather than each of you conducting your own short survey and duplicating all the work (and cost) of set-up, sampling, and communicating with respondents, you can pool your resources to do one survey.  Each of you pays proportionally to how much of the survey is devoted to your questions.

Here is an example how an omnibus survey works (shown second) versus traditional custom surveys (shown first) — the costs shown are for illustrative purposes only:

Since Versta Research and other research firms have clients asking for short and inexpensive surveys all the time, we do the work of finding the other companies to partner with you on an omnibus.  Versta will advise you on all phases of the work:  How many questions to ask, what to ask, how to ask, and we’ll provide you with several cost and survey-length options so that you can find the right approach for your budget and strategic needs.

Have more questions?  Give us a call at 312-348-6089.  We would be happy to answer your questions and to advise on an approach that works for you.

-Joe Hopper, Ph.D.

Looking back, here are what we consider to be the five biggest changes that shaped current challenges faced by market research and opinion polling:

1. Online data collection using sample panels became the dominant mode of surveying, displacing a huge industry in telephone surveying.  Plus there has been a dramatic decline in households owning landlines, which makes the ideal of probability sampling extremely difficult to achieve.
2. Survey technology made data collection easy and cheap which means that a lot of market research became a commodity, resulting in industry consolidation and need to find new ways of adding value.
3. An explosion of too many surveys, which are everywhere, powered by simple tools like Survey Monkey.  Along with this there has been a dramatic decline in response rates, bringing the issue of quality to the fore like never before.
4. Statistical and mathematical expertise grew in demand.  There is so much data and information, and it is so easy to get.  But who knows what to do with it all?
5. “Audience-ready” reports became more valuable.  Newspapers laid off staff and market research departments downsized, demanding a level of professional involvement to ensure that research reports are easy to read, interpret, and ready to deliver to the audiences who need them most.

One product of these five trends is Versta Research’s focus on high-level expertise and academic brainpower so that you get smart, creative, and flexible market research.  Another is the skill and commitment we have in helping you communicate research to managers, clients, reporters, and the audiences you need to reach.

In our next post we’ll highlight what we predict will be the five biggest trends that will shape market research challenges in the decade to come.

–Joe Hopper, Ph.D.

There will always be a need for good storytelling. This point was eloquently made by Susan Howe, President of Weber Shandwick Chicago.  The media through which stories are told is shifting, but the fundamentals of good PR remain.  Likewise for research, the tools we use are changing every day, but turning data into stories is a constant that underlies change.

You know the story better than anyone. Rich Jernstedt, EVP of Fleishman-Hillard argued that good PR professionals own and communicate the client’s story at every phase because a reporter will never know that story as deeply.   Likewise, an effective research partner helps a client communicate data from beginning to end, long after the “report” is delivered.  Nobody knows the research as thoroughly as we do.

We need to add value. Janet Cabot, Co-President of the Chicago Office of Edelman, highlighted the importance of research and intellectual capital because effective tactics are now a “given” and clients are looking for more.   The same goes for research itself.  Focusing on data collection and tabulation no longer adds value.  We need to leverage our considerable intellectual capital by helping clients interpret, grasp and communicate the research.

It was an optimistic and inspiring forum that was consistent with our thinking at Versta Research:    Change is an opportunity to focus on (1) the fundamentals of what we do (designing rigorous research to answer critical questions) and (2) the value we add (turning data into stories), both of which make for strong and lasting partnerships with our clients.

–Joe Hopper, Ph.D.

Unfortunately these companies are teaching customers to ignore them.  Their surveys are not about listening to customers, but about internal systems for benchmarking or trending.  Worse yet, sometimes nobody even pretends to listen.  The data from these surveys are ignored because there is too much of it, nobody knows what to make of it, or because it always says the same thing.  If they listened, they would notice that their customers are increasingly irritated.  We recently helped a client analyze such data and read customer comments like: “Why are you asking me this?  Do you really care? Why are you annoying me with a survey before I have even finished?”  Irritating your customers is too high a price to pay for this kind of research.

There are more efficient, more respectful, and more insightful ways to listen to your customers with survey research.  Here are a few simple guidelines:

• Define your “need to know” information, and don’t ask anything beyond that
• Know ahead of time how much data is needed, and put a limit on how much is enough
• Determine an end point for data collection, instead of gathering data forever just because it is cheap and easy
• Keep surveys relevant so that you are respectful of your customers’ time and goodwill
• Use sampling strategies so that each customer is surveyed no more than once or twice a year

Research and insight are important, but it is critical to understand the cost of teaching your customers to ignore you.  Be smart about your research, and you will get the insight you need.

If you are unsure, give Versta Research a call and we will help evaluate your current survey program to ensure that you are getting valuable information without undue burden on your customers.

–Joe Hopper, Ph.D.

The first approach is to mine data.  Find data that you already have, analyze that data, use it to answer interesting questions, and then tell the story.  For example, financial services, HR, benefits, and other outsourcing companies like Hewitt Associates track a wealth of data about employees and employers as part of their business.  They can then use that data to write interesting stories about how many people cash out their 401(k) plans when leaving a job, or how many companies are switching healthcare plans as costs continue to rise.

The second approach is to generate new data by commissioning a survey that answers interesting questions to drive your story.  For example, one client I worked for got significant story placements by surveying people about skin cancer myths and reasons for not taking simple precautions like wearing sunscreen.  Another gets ongoing coverage for a survey of IT professionals about future trends in the marketplace.

With both approaches, the trick is to (1) build a credible foundation with expert research and data analysis, and (2) turn data into stories. Versta Research can help you with both pieces.  We can help you answers questions with expertise, turn data into stories, and ultimately help you  communicate those stories to the audiences who need them most.

–Joe Hopper, Ph.D.