Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

AAPOR and the ASA are offering a great way for non-statisticians to learn more about sampling.  AAPOR is the American Association of Public Opinion Research, and the ASA is the American Statistical Association.  In February they are hosting an introductory webinar on sampling for non-statisticians that that we recommend.  It is being taught by a senior statistician at NORC, and will cover such topics as:

• The logic of sampling
• The concept of a sampling frame
• The difference between a sample and a census
• The process of sampling
• Different types of samples
• Probability vs. non-probability sampling
• Sampling error and non-sampling error
• Sample size considerations
• Post-sampling steps

The webinar is described as “a course to expose non-statisticians to sampling so that they are able to read and understand articles or documents describing sampling designs and communicate with statisticians about their research needs.”

The work that we do at Versta Research always starts with the art of asking questions (so we know what our research must “do”) then moves into the rigors of research design, which typically involves the science of sampling, then back again into the art and science of turning data into stories.  If you have always had a nagging feeling that you ought to know just a little bit more about that middle piece—the science of sampling—without having to get a four year degree in statistics, here’s your chance.

—Joe Hopper, Ph.D.

While the Census provides basic tabulations and counts on its website, you can also query the data yourself to answer specific, customized questions that you are unable to find otherwise.  In fact, using this data is so easy that Versta no longer uses the Census’ rather difficult and ever-changing “fact finder” features.  We have downloaded the data along with a codebook, and use a statistics program to run specific queries and counts of anything we need.

Want to know how many single mothers there are between the ages of 40 and 45, with four children living at home?  We can tell you.  We recently had a client who was having a hard time finding the percentage of people in the U.S. age 40 and over.  With the data from the census at our fingertips, it took just a few minutes to get this number, and ship off the answer to our client.

Versta Research Report Using Census Data

The data (called the Public Use Microdata Sample, or PUMS) is available to anyone, for free.  You’ll need a decent database, spreadsheet, or statistics program work to with it (we use SPSS).  Or, to get the exact information you are looking for, contact Versta Research and we can help.

Here is a partial list of attributes covered in the data:

• Age
• Ancestry
• Disability status
• Educational attainment
• Fertility
• Hispanic origin
• Hours worked
• Income
• Industry
• Location
• Marital status
• Mobility status
• Occupation
• Race
• Sex
• Family, subfamily, and household relationships
• Mortgage status and selected monthly owner costs
• Presence and age of own children
• Property value

–Joe Hopper, Ph.D.

Surveys and polls can be powerful tools to understand what people are thinking and doing, and they can provide good data for public relations efforts and community outreach.  Unfortunately they can also be gimmicks, which erodes trust in polling and in the organizations sponsoring them.

Here is an example.  Walmart wants to build a new store in the city of Chicago, and so far the city council has said “no thank you.”  So Walmart conducted a “poll” of city residents to prove that the residents back them.  Eric Zorn at the Chicago Tribune has been following the story (so has Chicagoist).  Zorn describes several problems with the poll:  It was conducted via computerized telephone calls; there was only one question; the wording of the question was biased.

Here’s another issue to add to the list.  Walmart says they called every resident listed in the directory in the city of Chicago.  They made over ONE MILLION phone calls in one day.  Smart and effective pollsters do not do this.  A legitimate and accurate poll of Chicago city residents would have relied on a carefully selected and statistically adjusted sample of 800 residents.  Each phone call needs to be carefully handled, managed, and tracked.  You need to know who answered and who did not, where they live, how old they are, and so on.

Imagine the US Census Bureau conducting its annual American Community Survey by robo-calling every household or every telephone number in the nation (how long would it take — a week at most?) then declaring they have accurate data on the population.  Would you trust these results?

If you want to conduct an opinion poll for public relations efforts and community outreach, devote some time and thought to these key issues in designing your poll:

• Write neutral, well-worded questions
• Use an appropriate data collection method
• Design a credible sampling plan
• Carefully manage of the sample, outreach, and data collection

These will help ensure your findings can withstand media scrutiny.

–Joe Hopper, Ph.D.