As of this writing, Versta is one of 67 prominent survey and polling organizations currently supporting the initiative. The initiative is designed to create protocols and recognition for regular disclosure of methods when survey organizations conduct public polls.

Trust in survey research and public polls is critical not only for the public good, but for the important work that we do for our clients.  Our clients rely on us to give them accurate data and truthful assessments. They can’t make good decisions without that. And they rely on us to ensure that when they share data with the public, it is backed by the most rigorous and defensible methods available in the social sciences. Transparency is one means of ensuring that all organizations are continually held to the highest standards of rigor and accuracy.

AAPOR began the initiative in 2009 “to encourage routine disclosure of methodological information from polls and surveys whose findings are released to the public.” The organization lists the following as key elements of the efforts now underway:

• The association’s public recognition of excellence in transparency;
• A system for collecting and archiving disclosure information;
• Education for organizations in how to make transparency a routine part of their work;
• Outreach to survey sponsors, users of survey data and the public to reinforce the value of transparency and to publicize the organizations that participate in the initiative;
• Collaboration with other associations and academic disciplines in these activities.

One benefit of working with an organization like Versta Research is that you can fully trust the outcomes of the research, and you can trust that we have the expertise to defend it.  You can also trust that we work diligently within the industry to protect the value of our collective investments in good research.

—Joe Hopper, Ph.D.

The idea seems crazy, but as reported in Science, researchers at Carnegie Mellon University have found that certain kinds of twitter data can give them a good read on public sentiment. The looked at things like positive or negative comments about President Obama and found results that aligned with traditional polls. Comments about finances and savings aligned with consumer confidence polls. What does the polling industry think? Here’s what one colleague says: “I believe that I am now going to run to a dark corner of the house and cower in fear of what may come.”

To be sure, we are not recommending that you rely on social media to accurately measure overall public opinion. No way. But in our view, these findings may lead to new insights about how people, individually and collectively, behave and think, and how research — both academic and practical — can harness these new forms of data to measure markets and the social world.  The findings may also force us to re-think our theories of statistical inference that rely on random sampling. There is much research to be done before we will know, but possibilities for social media someday offering insights that are as statistically valid as our current methods is intriguing and nothing to cower about.

Thinking about measuring social media as part of your research plan? We would be happy to advise you. We can help you explore new options in research while ensuring that your research and findings remain rigorous and defensible.

—Joe Hopper, Ph.D.

The number of people without home access to landline telephones is increasing.

Almost half of adults under age 30 live in a household with only wireless telephone service.

In response, our industry has developed newer (and complicated) methods to include cell-only households along with land-based phone sampling.  But do-not-call rules, the cost to respondents of receiving calls on wireless phones, and the fact that exchanges no longer map to geographic regions have been significant challenges.

Now there is a new challenge:  Even though the number cell phones and the number of people who carry them is increasing exponentially, people are not talking on them.  More than nine out of ten households now has cell phone service, but recent data indicate that voice usage is not increasing, and for the first time “the amount of data in text, e-mail messages, streaming video, music and other services on mobile devices in 2009 surpassed the amount of voice data in cellphone calls” (see NYT article).

As people use their phones more and more in multiple other ways, the opportunity for public polling and market research is to find new ways of engaging people who are willing to share data and opinions.  There are now surveys designed for mobile devices and real-time enthnographies using video, photography, and voice from cell phones.  Mobile devices are also increasingly used for purchases, data monitoring, and loyalty programs, all of which can be rich sources of insight for market research.

Need help thinking about the best way to conduct your survey or research?  The best way will depend on your specific questions and the group of people you want to understand.  Give us a call (we like to talk on the phone); we will help you sort out your options for an optimal approach.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/trouble-for-phone-surveys-nobody-talks/feed/ 0 http://www.verstaresearch.com/blog/practical-statistics-vs-theoretical-statistics/ http://www.verstaresearch.com/blog/practical-statistics-vs-theoretical-statistics/#comments Fri, 09 Apr 2010 12:33:19 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=457 If something works and it keeps on working but you don’t know exactly why it works, what would you do?  Our view is that you should keep doing it.  Not everyone agrees with us.  The American Association of Public Opinion Research (AAPOR) convened a task force to study online survey panels, and released their report last month (we posted a summary of findings last week).  To us, the most jarring statement in the report was this:

“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”

Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters.  Why?  Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works.

Their conclusion is particularly surprising because although statistical inference based on probability sampling has a solid theoretical underpinning, in practice pure probability samples are almost never achieved.  Nearly always we are faced with low response rates and non-response biases.  We work around this in practical ways (not always supported by extensive theory) to weight the data, adjust for biases, understand the sources of biases, add caveats to our findings, and so on.  And sure enough, our efforts tend to work, so we keep doing it.

There is substantial data to show that carefully managed non-probability online panels can be used to estimate certain population parameters depending on the nature of the study and how exact those estimates need to be.  Someday our academics and social theorists will help us understand why.  Here’s our theory: Social bodies are not composed of individual units (the assumption underlying inferential statistical theory)  but nodes of networks such that non-random entry points to measure social forces can provide as much information as a random selection of individuals.  We’ll leave it to the future theorists to tell us if we’re right.

In the meantime, there is a lot of research for which using non-probability online panels makes good sense (which the AAPOR report acknowledges), so we’ll keep doing it and we’ll keep extending it into new areas even if the theory can’t keep up.  And we’ll keep refining our techniques based on experience and practice to make it work better, and we’ll keep thinking about why it works and lending that insight to the work we do for you.

If you need help or expertise designing and executing your research based on probability or nonprobability samples, online panels or phone, let us know.  We know our statistical theory.  More importantly, we know our practical statistics.  We can help you sort through it all to ensure rigorous research and practical results.

—Joe Hopper, Ph.D.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.

Looking back, here are what we consider to be the five biggest changes that shaped current challenges faced by market research and opinion polling:

1. Online data collection using sample panels became the dominant mode of surveying, displacing a huge industry in telephone surveying.  Plus there has been a dramatic decline in households owning landlines, which makes the ideal of probability sampling extremely difficult to achieve.
2. Survey technology made data collection easy and cheap which means that a lot of market research became a commodity, resulting in industry consolidation and need to find new ways of adding value.
3. An explosion of too many surveys, which are everywhere, powered by simple tools like Survey Monkey.  Along with this there has been a dramatic decline in response rates, bringing the issue of quality to the fore like never before.
4. Statistical and mathematical expertise grew in demand.  There is so much data and information, and it is so easy to get.  But who knows what to do with it all?
5. “Audience-ready” reports became more valuable.  Newspapers laid off staff and market research departments downsized, demanding a level of professional involvement to ensure that research reports are easy to read, interpret, and ready to deliver to the audiences who need them most.

One product of these five trends is Versta Research’s focus on high-level expertise and academic brainpower so that you get smart, creative, and flexible market research.  Another is the skill and commitment we have in helping you communicate research to managers, clients, reporters, and the audiences you need to reach.

In our next post we’ll highlight what we predict will be the five biggest trends that will shape market research challenges in the decade to come.

–Joe Hopper, Ph.D.

Two researchers recently came up with methods of testing whether polling data is legitimate in a case where a research firm is accused of falsifying publicly released data. The researchers examined the last digit of the numbers released by the polling firm, and calculated the probability of seeing these digits in their specific combinations and frequencies if the polls were legitimate.  It was an ingenious way of using sophisticated statistical methods to examine data where the accused firm refused to disclose details about its methods.  The New York Times calls it “forensic polling analysis” and notes that in this case, the odds suggest that the polling data were fraudulent.

There are a number of leading industry organizations that have outlined ethical guidelines and principles of disclosure for survey research, including:

• AAPOR, the American Association of Public Opinion Research
• NCPP, the National Council on Public Polls
• CASRO, the Council of American Survey Research Organizations

Versta Research believes these guidelines are important to the health and integrity of research.  We adhere to them and encourage our clients to do so as well.  If you are unsure about your ethical obligations when releasing polling data to the public, give us a call and we can help you make sense of the guidelines.

–Joe Hopper, Ph.D.

There will always be a need for good storytelling. This point was eloquently made by Susan Howe, President of Weber Shandwick Chicago.  The media through which stories are told is shifting, but the fundamentals of good PR remain.  Likewise for research, the tools we use are changing every day, but turning data into stories is a constant that underlies change.

You know the story better than anyone. Rich Jernstedt, EVP of Fleishman-Hillard argued that good PR professionals own and communicate the client’s story at every phase because a reporter will never know that story as deeply.   Likewise, an effective research partner helps a client communicate data from beginning to end, long after the “report” is delivered.  Nobody knows the research as thoroughly as we do.

We need to add value. Janet Cabot, Co-President of the Chicago Office of Edelman, highlighted the importance of research and intellectual capital because effective tactics are now a “given” and clients are looking for more.   The same goes for research itself.  Focusing on data collection and tabulation no longer adds value.  We need to leverage our considerable intellectual capital by helping clients interpret, grasp and communicate the research.

It was an optimistic and inspiring forum that was consistent with our thinking at Versta Research:    Change is an opportunity to focus on (1) the fundamentals of what we do (designing rigorous research to answer critical questions) and (2) the value we add (turning data into stories), both of which make for strong and lasting partnerships with our clients.

–Joe Hopper, Ph.D.

The first approach is to mine data.  Find data that you already have, analyze that data, use it to answer interesting questions, and then tell the story.  For example, financial services, HR, benefits, and other outsourcing companies like Hewitt Associates track a wealth of data about employees and employers as part of their business.  They can then use that data to write interesting stories about how many people cash out their 401(k) plans when leaving a job, or how many companies are switching healthcare plans as costs continue to rise.

The second approach is to generate new data by commissioning a survey that answers interesting questions to drive your story.  For example, one client I worked for got significant story placements by surveying people about skin cancer myths and reasons for not taking simple precautions like wearing sunscreen.  Another gets ongoing coverage for a survey of IT professionals about future trends in the marketplace.

With both approaches, the trick is to (1) build a credible foundation with expert research and data analysis, and (2) turn data into stories. Versta Research can help you with both pieces.  We can help you answers questions with expertise, turn data into stories, and ultimately help you  communicate those stories to the audiences who need them most.

–Joe Hopper, Ph.D.

In years past, data collection was a slow, complicated, and expensive task.  If you had just a few survey questions, you could share the cost of data collection by cooperating with others who had just a few questions.  That was the birth of the omnibus.  Today, however, most surveys are conducted online, and the data collection process is automated, fast, and inexpensive.  The cost of a custom survey with just a few survey questions, just for you, fielded exactly when and how you want it,  is often no higher than an omnibus.

Sometimes an omnibus survey is the best way to go, but it all depends on the issues you’re trying to understand and the audience you’re trying to reach.  If you think an omnibus survey might be right for you, here are Versta’s recommendations about how to proceed:

• Focus on the issues you’re trying to understand and the budget you have to work with

• Ask us to recommend all options that answer your questions within your budget

• Do not worry about whether the study is an omnibus or not — an omnibus survey is a money-saving technique for the researcher that should be invisible to you

• If an omnibus survey makes sense, work through a firm like Versta Research – even if we field it through a big-name provider for you, it will cost less, and we can design it in a customized way that helps you turn data into stories

–Joe Hopper, Ph.D.