If you HAVE done the hard work of phone surveys, then besides deeper research insights into your business, we expect you will have a deeper appreciation for this holiday survey, written by Mitch Pravatiner, which is written in exactly the format that telephone interviewers typically see.  We have worked with Mitch in the past, and he is a true pro when it comes to telephone interviewing. He recently shared this via aapor-net, which is a discussion group of research professionals who belong to the American Association of Public Opinion Research.

‘Twas the Night Before Christmas . . .

Hello, this is [NAME OF INTERVIEWER] from [NAME OF ORGANIZATION], an opinion research organization. Tonight we’re calling people across the country to find out what’s going on in their homes on the night before Christmas. Please rest assured that this is not a sales call, and that everything you tell us will be kept confidential. Would you be willing to participate?

Yes (CONTINUE)
No (THANK AND TERMINATE)

Q1 First of all, have you noticed any creatures stirring in your house tonight?

Yes (SKIP TO Q3)
No
DON’T KNOW (SKIP TO Q3)
REFUSED (SKIP TO Q3)

Q2 Not even a mouse? (DO NOT READ LIST)

YES, NOTICED MOUSE STIRRING
NO, NOT EVEN A MOUSE
DON’T KNOW
REFUSED

Q3 Have you hung any stockings by your chimney?

Yes
No (SKIP TO Q6)

Q4 How much care did you use in hanging the stockings by the chimney? Please rate it on a scale from 0 to 10, where 0 means “no care at all” and 10 means “extreme care.”

0
1
2
3
4
5
6
7
8
9
10
DON’T KNOW
REFUSED

Q5 What would you say was the single main reason you hung stockings by the chimney? Was it:

In the hope that Santa Claus would drop in
To dry them after you had to walk home through a series of deep snowdrifts
They had developed runs, so there was no other use for them, or
Some other reason (SPECIFY)
DON’T KNOW
REFUSED

Q6 As far as you know, what visions have your children experienced in bed tonight? (RANDOMIZE)

(NO CHILDREN)
Barbie dolls
GI Joes
Chatty Cathys
Sugar plums
Betsy Wetsys
Cabbage Patch Kids
Tickle Me Elmos
Easy Bake Ovens
Candy canes
Radio Flyers
Fruitcake
Red Ryder BB guns
Other (if “yes” go to Q6a, otherwise skip to q7)

Q6a What other visions? (PROBE AND CLARIFY)

DON’T KNOW
REFUSED

Q7 Have you tried to get any sleep so far tonight?

Yes
No
DON’T KNOW
REFUSED

Q8 Were you awakened from your sleep at any time tonight?

Yes
No
DON’T KNOW
REFUSED

Q9 What woke you up? (READ LIST)

A loud noise outside the house
A loud noise inside the house
Indigestion from something you ate at the office party earlier today
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q10 What did you do after you woke up? (READ LIST)

Got up to see what was going on
Shrugged it off and went back to sleep
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q11 What did you notice after that?

A sleigh pulled by reindeer parked on the front lawn
A UPS truck that overturned after skidding on black ice on the street in front of your house
Several drunks who passed out on the lawn after leaving the party next door
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q12 Was Santa Claus with the sleigh and the reindeer?

Yes
No
DON’T KNOW
REFUSED

Q13 How many reindeer did you see?

(RECORD NUMBER)
DON’T KNOW
REFUSED

Q14 Did Santa mention any of them by name?

Yes
No
DON’T KNOW
REFUSED

Q15 Which reindeer do you recall him mentioning? (DO NOT READ LIST; MULTIPLE ANSWERS PERMITTED)

DASHER
DANCER
PRANCER
VIXEN
COMET
CUPID
DONNER
BLITZEN
OTHER (SPECIFY)
DON’T KNOW
REFUSED

Q16 (ASK FOR EACH REINDEER NOT MENTIONED IN Q15)

Did Santa also mention [NAME OF REINDEER] by name?

Yes
No
DON’T KNOW
REFUSED

Q17 Did Santa move the sleigh anywhere after that?

Yes
No
DON’T KNOW
REFUSED

Q18 Where did he move it to?

The roof
The back yard
The alley behind the house
Somewhere else (SPECIFY)
DON’T KNOW
REFUSED

Q19 Did Santa enter your house after that?

Yes
No
DON’T KNOW
REFUSED

Q20 How did he enter your house?

Through the front door
Through the back door
Through the chimney
Some other way (SPECIFY)
DON’T KNOW
REFUSED

Q21 Can you describe Santa while he was in your house? (PROBE AND CLARIFY)

DON’T KNOW
REFUSED

Q22 Did you and he speak to each other while he was in the house?

Yes
No
DON’T KNOW
REFUSED

Q23 Which of the following best describes what Santa Claus did while in your house?

Filled the stockings and then left
Filled the stockings, then ate your Christmas cookies, and then left
Filled the stockings, ate your Christmas cookies, polished off your eggnog, and left
Something else (SPECIFY)
DON’T KNOW
REFUSED

Q24 How did he leave your house?

Through the front door
Through the back door
Through the chimney
Some other way (SPECIFY)
DON’T KNOW
REFUSED

Thank you. Those are all the questions we have for you this evening. Merry Christmas to all, and to all a good night.

–Joe Hopper, Ph.D.

Here we offer the “Versta Digest” version as a handy reference card.  Once you get the hang of it, you don’t need the examples and explanation.  You just need to know the rules.  We recommend reading the full article first, so you know what we’re talking about when it comes to “points.”  Then, when you need a refresher or a reference source, consult these rules:

• One point for each simple question or scaled response
• One point for every two response options in a multiple choice question
• One point for each row in a grid question
• Two points for any response that requires mental calculation
• Three points for every short response to an open-ended question
• One point for every three sentences of extra text that respondents must read

Then:

• Tally up the points
• Divide by 8 for online survey length (in minutes)
• Divide by 8 then multiply by 1.5 for phone survey length (in minutes)

The system is straightforward, easy to learn, and easy to execute.  It is a method that really works and that we have validated against hundreds of different types of surveys over the past several years.

Once you know how to determine survey length, you can think more strategically about the ideal survey length to optimize the value and content of a survey within your budget.  Call us at 312-348-6089 with any additional assistance you may need.

–Joe Hopper, Ph.D.

Is it true that the Associated Press refuses to carry stories from online surveys?  Yes, as odd as that seems nowadays.  But news media face a difficult problem given how easy it is to conduct biased public opinion polling, especially now with online panels and social networks.  So some news organizations like the Associated Press (AP), The New York Times, and ABC News have developed guidelines that specify for a survey or public opinion poll to be valid and reliable, it must be conducted by telephone.

It is worth understanding their rationale even if you do not pitch research stories to the media because the guidelines provide a fascinating glimpse into current methodological debates about probability sampling, inferential statistics, and the rapidly changing world of online data collection and analysis. If your organization does use research for public relations and marketing, then a deeper understanding will help you offer recommendations to your organization about conducting PR research.

In our Fall 2011 Newsletter, we outline media guidelines for reporting on survey research.  Whether your goal is getting research into the board room to influence top decision makers, or in front of the public to promote your brand, you need to know the standards of rigor against which the research will be judged.

Our commitment at Versta is to advise you on the best research mode for your campaign strategy, and to conduct rigorous research that can withstand the highest levels of scrutiny.

–Joe Hopper, Ph.D.

• Thirty percent of U.S. households do not have a landline telephone
• An additional 16% have a landline telephone, but never or rarely use it to receive calls
• The percentage of households without landlines is increasing by about five to six percentage points each year
• Half of young adults under age 30 have no landline in their homes
• Half of adult renters have no landline in their homes
• Nearly four out of ten Hispanic adults have no landline in their homes

For anyone doing random sample surveys of consumers, the implications are huge.  Methodological purists insist that only phone-based surveys are rigorous, but clearly the biases of phone-based research can be severe, and no doubt they often are.

Putting aside the issue of research methods, however, we believe a more significant issue over the next few years will be the explosive growth in smartphone usage.  Although not captured in government surveys that track phone usage and availability, the Pew center estimates that one-third of all U.S. adults currently own a smartphone, and industry analysts predict that by the end of 2011 half of all cell phones in the U.S. will be smartphones.

Why is this is so critical for researchers?  Because more than any other technology since the widespread adoption of the Internet, smartphones are changing how consumers behave.  Consumers get up in the morning and check media apps before brushing their teeth.  Then while commuting to work or waiting for their computer to boot up, they purchase coupons for goods and services they might never have bought otherwise.  They map and track their locations, and expect product features and services that can anticipate what they need.

Smartphones are important for reasons far beyond narrow methodological concerns about how to execute surveys.  They are important because they are changing the very nature of what we research: how people behave and think, and what they buy, believe, want, and aspire to.  Indeed, just as we nearly always ask our research respondents to tell us their gender, age, and income, we now often ask whether they have smartphones.

Beyond anything else, market research is about understanding people (the verstehen in Versta), which means our focus at Versta is always on the “what, how, and why” of what matters in peoples’ lives in a rapidly changing world.  Smartphones matter in a more profound way than cell phones ever did.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/smartphones-matter-more-than-cell-phones/feed/ 0 http://www.verstaresearch.com/blog/cell-phones-may-double-your-survey-costs/ http://www.verstaresearch.com/blog/cell-phones-may-double-your-survey-costs/#comments Tue, 09 Aug 2011 14:13:02 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1528

These days most researchers agree that if you want to do a random sample phone survey of the U.S. population, you ought to include cell phones.  More than one-quarter of the population do not have landline telephones at home.  Those who do have landline telephones are less likely than ever to answer them, and less likely than ever to participate in surveys.

But it is not easy to include cell phones.  The sampling protocols and the post-stratification weighting become more complicated.  You need to account for a higher probability of cell phone owners being in your sample, because most of them also have landlines.  You can’t use automated or predictive dialing to call cell phone numbers.  You can’t target geography as well, because area codes and exchanges have become mobile.  And people get mad at you if they have to pay for incoming calls, so you need to offer cash.

What’s the bottom line effect on costs for a survey that includes cell phones?  A recent study sponsored by the American Association for Public Opinion Research (AAPOR) documents the following:

The cost of fieldwork for cell phone interviews is double if you don’t screen out those who also have landlines, and more than double (2.6 times higher) if you do screen them out.

Of course if you are surveying the whole population, only a portion of your sampling and interviews will be cell-phone based.  Currently, we recommend that 20% to 40% of interviews be cell-based.  But there are additional professional costs to remember as well, such as purchasing, managing, merging, weighting, and analyzing different types and sources of sample, and training interviewers to work with different sources and types of respondents.

There are still plenty of phone surveys being done that do not include cell-phones, and for many types of studies landline-only surveys produce information that is good enough for what needs to be learned.  But it is getting increasingly difficult for these surveys to achieve true representation and surely their days numbered.

Feel free to give us a call if you need help figuring out the best approach for your research.  We can advise you on the most cost-effective, feasible, and rigorous approaches to getting the data, stories, and level of understanding you need.

—Joe Hopper, Ph.D.

Why?  Because their research challenges the beliefs of many AAPOR-ites who disregard most online research as being theoretically indefensible since it is not based on probability sampling.  The research presented was based on parallel surveys conducted last year, designed to allow careful comparison of three survey modes:  a “gold standard” phone survey that included both landline and cell phone RDD sampling, an opt-in Internet panel survey that used careful weighting techniques to ensure a comparably representative sample of the U.S. population, and a traditional paper-based mail survey.  The questionnaires for each mode were nearly identical and elicited data on variety of topics including several that could be validated against the most rigorous U.S. Census and NIH data.

Here is how the authors summarized their results:

Comparing the findings from the modes to each other and the validated benchmarks, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates. (download full paper)

Their research adds to a growing body of research showing that rigorously done online surveys can be as useful and effective as rigorously done phone surveys.

True, most online surveys are not rigorously executed (but they can be), and the practical successes of online surveys have outpaced our understanding of the statistical, theoretical, and social underpinnings of why they work.  But it is both intellectually closed-minded and pragmatically foolish to dismiss all online surveys as “invalid” and “not newsworthy” as some of our colleagues in AAPOR do.

Our approach at Versta Research is rooted in our training and continuing engagement in academic research, but we are also rigorously focused on doing what works to get the urgent answers and compelling business stories that our clients need.  Online survey research is one crucial piece of that.

—Joe Hopper, Ph.D.

As newly-appointed director of market research for the American Marketing Association in Chicago, I oversee efforts to develop and organize events related to market research.  I also oversee efforts to measure, track, and analyze customer satisfaction for all other events and activities sponsored by the AMA.  For the past few events, the AMA collected feedback via an online survey sent by e-mail immediately after the events.  They were also using a less-than-stellar do-it-yourself survey tool that promises to deliver “actionable insights!” at the click of a button.

The results were abysmal.  For every 50 attendees, we were lucky to get one response, which meant our final sample sizes ranged from one to four respondents.  Even a magic “actionable insights” button can’t do much with that.  For good reasons, we did not want to send out multiple e-mail reminders, and there was no money or staff to reach out with telephone calls.

So what’s a market researcher to do?  First, we on the market research committee joked about doing multivariate analysis with a sample of n=1.  OK, not so hilarious.  Then we decided that for the next MR event that we were organizing, we would go back to basics.  The plan:  Distribute surveys by hand during the last ten minutes of the event; make an announcement about it; explain why feedback is important; tell people how the data will be used; ask them to do it; and collect their surveys as they headed out the door.

We considered offering a raffle incentive as well, but we didn’t need to.  We got responses back from two-thirds of attendees.  True, we lost some advantages of doing an online survey.  The process was less anonymous, and we were stuck having to enter all the data into a computer by hand.  But we got robust data that we can work with, analyze, and develop a story to share with our client.

Sometimes paper surveys make a lot of sense, and in this case it made sense because our respondent base was  right there, a captive audience in one room.  Evolving technologies don’t always replace the old.  Telephones, computers, the Internet, e-mail, social media, and smartphones are not necessarily replacing paper surveys, but rather opening new opportunities and finding their appropriate niches.  Each mode has an important and valuable place in market research.  The key is to know when to use each and for what purposes, because all you really care about is getting information to make smart decisions.

Need help sorting through all the options?  Feel free to give us a call.  We can help you optimize your research plan by choosing the best methods, whether old or new, and then putting the best minds to work in answering your critical questions.

—Joe Hopper, Ph.D.

This shift in telephone usage is critically important to survey research, because telephone surveys have set a gold standard for rigorous research over the last two decades.  Given the trends, most survey researchers would agree that cell phone sample must now be included in the most rigorous research designs for an RDD survey.  But doing so introduces new difficulties in design, sampling, weighting, and cost, and our collective experiences are not yet sufficient for the industry to  agree on a set of “best practices” when it comes to including cell phones in survey research.

Our leading industry association, AAPOR (The American Association of Public Opinion Research) recently published a comprehensive update from the AAPOR Cell Phone Task Force that offers a good overview of the issues, complications, guidelines, and considerations every researcher should take into account when designing a telephone survey of the general population.  Briefly, it covers and highlights the following issues:

Coverage and Sampling: It is increasingly difficult to reach young men and minorities via landlines, so cell phone coverage is critical.  But using dual and overlapping sampling frames (both landline and cell phone) introduces new methodological complications in sampling.

Nonresponse: Response rates are lower among cell phones users, though response rates among landline users continue to decline as well.

Measurement: Data gathered via cell phone interviews is generally not of lower quality (despite concerns about audio quality, talking in public places, and distractions from multitasking).

Weighting: If dual and overlapping sampling frames are used (to include both landlines and cell phones) then data need to be statistically weighted to account for the complicated differences in probabilities that each person in the population will be recruited into the survey.

Legal and Ethical Issues: Including cell phones in surveying introduces a new set of legal, regulatory, and ethical considerations regarding auto-dialers, texting, caller ID, and do-not-call lists.

Operational Issues: Cell phone interviewing requires a unique set of protocols for recruiting, call-backs, and interviewing, all of which require special training for the people conducting the interviews.

Cost Issues: Including cell phones at least doubles the cost of a survey, and many times may triple or quadruple the cost.

Should you include cell phones in your survey?  It depends on who you are trying to reach and for what purposes.  When it comes to effective surveying, there are no absolutely right answers about cell phones vs. landlines, just as there are no absolutely right answers about online vs. phone surveys.  Moreover, the rapidly changing landscape of how people communicate means that the best answer today may be different in six months.

Need some help thinking through the options and figuring out the best approach for your research?  Feel free to give us a call.  We have decades of experience and are committed to helping our clients take full advantage of important changes and innovations in research.

—Joe Hopper, Ph.D.

The idea seems crazy, but as reported in Science, researchers at Carnegie Mellon University have found that certain kinds of twitter data can give them a good read on public sentiment. The looked at things like positive or negative comments about President Obama and found results that aligned with traditional polls. Comments about finances and savings aligned with consumer confidence polls. What does the polling industry think? Here’s what one colleague says: “I believe that I am now going to run to a dark corner of the house and cower in fear of what may come.”

To be sure, we are not recommending that you rely on social media to accurately measure overall public opinion. No way. But in our view, these findings may lead to new insights about how people, individually and collectively, behave and think, and how research — both academic and practical — can harness these new forms of data to measure markets and the social world.  The findings may also force us to re-think our theories of statistical inference that rely on random sampling. There is much research to be done before we will know, but possibilities for social media someday offering insights that are as statistically valid as our current methods is intriguing and nothing to cower about.

Thinking about measuring social media as part of your research plan? We would be happy to advise you. We can help you explore new options in research while ensuring that your research and findings remain rigorous and defensible.

—Joe Hopper, Ph.D.

The number of people without home access to landline telephones is increasing.

Almost half of adults under age 30 live in a household with only wireless telephone service.

In response, our industry has developed newer (and complicated) methods to include cell-only households along with land-based phone sampling.  But do-not-call rules, the cost to respondents of receiving calls on wireless phones, and the fact that exchanges no longer map to geographic regions have been significant challenges.

Now there is a new challenge:  Even though the number cell phones and the number of people who carry them is increasing exponentially, people are not talking on them.  More than nine out of ten households now has cell phone service, but recent data indicate that voice usage is not increasing, and for the first time “the amount of data in text, e-mail messages, streaming video, music and other services on mobile devices in 2009 surpassed the amount of voice data in cellphone calls” (see NYT article).

As people use their phones more and more in multiple other ways, the opportunity for public polling and market research is to find new ways of engaging people who are willing to share data and opinions.  There are now surveys designed for mobile devices and real-time enthnographies using video, photography, and voice from cell phones.  Mobile devices are also increasingly used for purchases, data monitoring, and loyalty programs, all of which can be rich sources of insight for market research.

Need help thinking about the best way to conduct your survey or research?  The best way will depend on your specific questions and the group of people you want to understand.  Give us a call (we like to talk on the phone); we will help you sort out your options for an optimal approach.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/trouble-for-phone-surveys-nobody-talks/feed/ 0