These days most researchers agree that if you want to do a random sample phone survey of the U.S. population, you ought to include cell phones.  More than one-quarter of the population do not have landline telephones at home.  Those who do have landline telephones are less likely than ever to answer them, and less likely than ever to participate in surveys.

But it is not easy to include cell phones.  The sampling protocols and the post-stratification weighting become more complicated.  You need to account for a higher probability of cell phone owners being in your sample, because most of them also have landlines.  You can’t use automated or predictive dialing to call cell phone numbers.  You can’t target geography as well, because area codes and exchanges have become mobile.  And people get mad at you if they have to pay for incoming calls, so you need to offer cash.

What’s the bottom line effect on costs for a survey that includes cell phones?  A recent study sponsored by the American Association for Public Opinion Research (AAPOR) documents the following:

The cost of fieldwork for cell phone interviews is double if you don’t screen out those who also have landlines, and more than double (2.6 times higher) if you do screen them out.

Of course if you are surveying the whole population, only a portion of your sampling and interviews will be cell-phone based.  Currently, we recommend that 20% to 40% of interviews be cell-based.  But there are additional professional costs to remember as well, such as purchasing, managing, merging, weighting, and analyzing different types and sources of sample, and training interviewers to work with different sources and types of respondents.

There are still plenty of phone surveys being done that do not include cell-phones, and for many types of studies landline-only surveys produce information that is good enough for what needs to be learned.  But it is getting increasingly difficult for these surveys to achieve true representation and surely their days numbered.

Feel free to give us a call if you need help figuring out the best approach for your research.  We can advise you on the most cost-effective, feasible, and rigorous approaches to getting the data, stories, and level of understanding you need.

—Joe Hopper, Ph.D.

What does Versta Research recommend when it comes to sample size?  Well, the answer always depends on the type of study, the objectives of the study, the audience to whom it will be presented, and of course budget. We can make legitimate statistical calculations based on any sample size, but there are dramatic trade-offs in precision and cost no matter what sample size is chosen.

Here, however, are some general guidelines:

If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision.  The benefit of doing this will almost never exceed the cost of including that many more people in your sample.

The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.

With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point.  But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points.  Another twenty five, fifty, or one hundred respondents can make an important difference when you are looking at small samples.

Most sample sizes will range from about 100 and 1,200 and within this range, for each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates.  So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people.  Some of the healthcare research we do relies on samples sizes of fifty to one-hundred.  Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200.

Unfortunately there is no magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand.  But there is a magic phone number worth trying:  (312) 348-6089.  Versta Research has a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget.  As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/an-interactive-graph-for-choosing-sample-size/feed/ 0 http://www.verstaresearch.com/blog/how-to-conduct-a-telephone-survey-for-gold-standard-research/ http://www.verstaresearch.com/blog/how-to-conduct-a-telephone-survey-for-gold-standard-research/#comments Thu, 25 Feb 2010 15:35:01 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=397 Telephone surveys are still considered the gold standard for rigorous public opinion polling and market research.  The reason is that virtually every household in the U.S. can be reached by telephone, and therefore we have careful methods of determining the probability that any individual person is included in a sample to be surveyed.  Knowing this probability is at the core of statistical inference, which makes mathematical purists very happy.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.

Phone vs. Online Surveys: Strengths & Weaknesses

As reflected in the table above, the article is rather technical.  But there are two key summary points worth learning from the study:

One, online surveys fielded through panel providers may not fully represent the population.  But we know from other research that they can come close, so the key is to understand the sources of potential bias and adjust for them as needed.

Two, online surveys fielded through panel providers may result in more accurate data than other types of surveys.  There are two reasons for this.  First, online panelists are more engaged in the process and interested in the topics, so they exhibit less cognitive laziness.  Second, online surveys feel more private, so respondents provide answers that are more honest.  This is in contrast to phone surveys in which people usually want to be nice and therefore provide socially desirable answers, even if it’s not how they really feel.

Which should you do, a phone survey or an online survey?  And what type of sampling should you employ, probability or non-probability?  It depends on your objectives.  Each of these issues can matter a lot or not at all depending on the specific questions you are trying to answer.  We would be happy to help you think through your options and the strengths of each.  Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.

-Joe Hopper, Ph.D.

While the Census provides basic tabulations and counts on its website, you can also query the data yourself to answer specific, customized questions that you are unable to find otherwise.  In fact, using this data is so easy that Versta no longer uses the Census’ rather difficult and ever-changing “fact finder” features.  We have downloaded the data along with a codebook, and use a statistics program to run specific queries and counts of anything we need.

Want to know how many single mothers there are between the ages of 40 and 45, with four children living at home?  We can tell you.  We recently had a client who was having a hard time finding the percentage of people in the U.S. age 40 and over.  With the data from the census at our fingertips, it took just a few minutes to get this number, and ship off the answer to our client.

Versta Research Report Using Census Data

The data (called the Public Use Microdata Sample, or PUMS) is available to anyone, for free.  You’ll need a decent database, spreadsheet, or statistics program work to with it (we use SPSS).  Or, to get the exact information you are looking for, contact Versta Research and we can help.

Here is a partial list of attributes covered in the data:

• Age
• Ancestry
• Disability status
• Educational attainment
• Fertility
• Hispanic origin
• Hours worked
• Income
• Industry
• Location
• Marital status
• Mobility status
• Occupation
• Race
• Sex
• Family, subfamily, and household relationships
• Mortgage status and selected monthly owner costs
• Presence and age of own children
• Property value

–Joe Hopper, Ph.D.

Surveys and polls can be powerful tools to understand what people are thinking and doing, and they can provide good data for public relations efforts and community outreach.  Unfortunately they can also be gimmicks, which erodes trust in polling and in the organizations sponsoring them.

Here is an example.  Walmart wants to build a new store in the city of Chicago, and so far the city council has said “no thank you.”  So Walmart conducted a “poll” of city residents to prove that the residents back them.  Eric Zorn at the Chicago Tribune has been following the story (so has Chicagoist).  Zorn describes several problems with the poll:  It was conducted via computerized telephone calls; there was only one question; the wording of the question was biased.

Here’s another issue to add to the list.  Walmart says they called every resident listed in the directory in the city of Chicago.  They made over ONE MILLION phone calls in one day.  Smart and effective pollsters do not do this.  A legitimate and accurate poll of Chicago city residents would have relied on a carefully selected and statistically adjusted sample of 800 residents.  Each phone call needs to be carefully handled, managed, and tracked.  You need to know who answered and who did not, where they live, how old they are, and so on.

Imagine the US Census Bureau conducting its annual American Community Survey by robo-calling every household or every telephone number in the nation (how long would it take — a week at most?) then declaring they have accurate data on the population.  Would you trust these results?

If you want to conduct an opinion poll for public relations efforts and community outreach, devote some time and thought to these key issues in designing your poll:

• Write neutral, well-worded questions
• Use an appropriate data collection method
• Design a credible sampling plan
• Carefully manage of the sample, outreach, and data collection

These will help ensure your findings can withstand media scrutiny.

–Joe Hopper, Ph.D.