The idea seems crazy, but as reported in Science, researchers at Carnegie Mellon University have found that certain kinds of twitter data can give them a good read on public sentiment. The looked at things like positive or negative comments about President Obama and found results that aligned with traditional polls. Comments about finances and savings aligned with consumer confidence polls. What does the polling industry think? Here’s what one colleague says: “I believe that I am now going to run to a dark corner of the house and cower in fear of what may come.”

To be sure, we are not recommending that you rely on social media to accurately measure overall public opinion. No way. But in our view, these findings may lead to new insights about how people, individually and collectively, behave and think, and how research — both academic and practical — can harness these new forms of data to measure markets and the social world.  The findings may also force us to re-think our theories of statistical inference that rely on random sampling. There is much research to be done before we will know, but possibilities for social media someday offering insights that are as statistically valid as our current methods is intriguing and nothing to cower about.

Thinking about measuring social media as part of your research plan? We would be happy to advise you. We can help you explore new options in research while ensuring that your research and findings remain rigorous and defensible.

—Joe Hopper, Ph.D.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.

Looking back, here are what we consider to be the five biggest changes that shaped current challenges faced by market research and opinion polling:

1. Online data collection using sample panels became the dominant mode of surveying, displacing a huge industry in telephone surveying.  Plus there has been a dramatic decline in households owning landlines, which makes the ideal of probability sampling extremely difficult to achieve.
2. Survey technology made data collection easy and cheap which means that a lot of market research became a commodity, resulting in industry consolidation and need to find new ways of adding value.
3. An explosion of too many surveys, which are everywhere, powered by simple tools like Survey Monkey.  Along with this there has been a dramatic decline in response rates, bringing the issue of quality to the fore like never before.
4. Statistical and mathematical expertise grew in demand.  There is so much data and information, and it is so easy to get.  But who knows what to do with it all?
5. “Audience-ready” reports became more valuable.  Newspapers laid off staff and market research departments downsized, demanding a level of professional involvement to ensure that research reports are easy to read, interpret, and ready to deliver to the audiences who need them most.

One product of these five trends is Versta Research’s focus on high-level expertise and academic brainpower so that you get smart, creative, and flexible market research.  Another is the skill and commitment we have in helping you communicate research to managers, clients, reporters, and the audiences you need to reach.

In our next post we’ll highlight what we predict will be the five biggest trends that will shape market research challenges in the decade to come.

–Joe Hopper, Ph.D.

The first approach is to mine data.  Find data that you already have, analyze that data, use it to answer interesting questions, and then tell the story.  For example, financial services, HR, benefits, and other outsourcing companies like Hewitt Associates track a wealth of data about employees and employers as part of their business.  They can then use that data to write interesting stories about how many people cash out their 401(k) plans when leaving a job, or how many companies are switching healthcare plans as costs continue to rise.

The second approach is to generate new data by commissioning a survey that answers interesting questions to drive your story.  For example, one client I worked for got significant story placements by surveying people about skin cancer myths and reasons for not taking simple precautions like wearing sunscreen.  Another gets ongoing coverage for a survey of IT professionals about future trends in the marketplace.

With both approaches, the trick is to (1) build a credible foundation with expert research and data analysis, and (2) turn data into stories. Versta Research can help you with both pieces.  We can help you answers questions with expertise, turn data into stories, and ultimately help you  communicate those stories to the audiences who need them most.

–Joe Hopper, Ph.D.

Surveys and polls can be powerful tools to understand what people are thinking and doing, and they can provide good data for public relations efforts and community outreach.  Unfortunately they can also be gimmicks, which erodes trust in polling and in the organizations sponsoring them.

Here is an example.  Walmart wants to build a new store in the city of Chicago, and so far the city council has said “no thank you.”  So Walmart conducted a “poll” of city residents to prove that the residents back them.  Eric Zorn at the Chicago Tribune has been following the story (so has Chicagoist).  Zorn describes several problems with the poll:  It was conducted via computerized telephone calls; there was only one question; the wording of the question was biased.

Here’s another issue to add to the list.  Walmart says they called every resident listed in the directory in the city of Chicago.  They made over ONE MILLION phone calls in one day.  Smart and effective pollsters do not do this.  A legitimate and accurate poll of Chicago city residents would have relied on a carefully selected and statistically adjusted sample of 800 residents.  Each phone call needs to be carefully handled, managed, and tracked.  You need to know who answered and who did not, where they live, how old they are, and so on.

Imagine the US Census Bureau conducting its annual American Community Survey by robo-calling every household or every telephone number in the nation (how long would it take — a week at most?) then declaring they have accurate data on the population.  Would you trust these results?

If you want to conduct an opinion poll for public relations efforts and community outreach, devote some time and thought to these key issues in designing your poll:

• Write neutral, well-worded questions
• Use an appropriate data collection method
• Design a credible sampling plan
• Carefully manage of the sample, outreach, and data collection

These will help ensure your findings can withstand media scrutiny.

–Joe Hopper, Ph.D.