Archive for the ‘Survey Tips’ Category
Friday, May 17th, 2013
If you have ever been called to participate in a phone survey, you probably know the routine where you hear a question and then jump in with an answer. A good interviewer will remind you that she needs to read the entire question and all the answer options just to be sure that you offered the best response option.
We can’t do this with self-administered online surveys, but there is a way to minimize error associated with respondents jumping to conclusions: Put all clarifying instructions before asking the question, not after. A recent study published in Public Opinion Quarterly documented that if you put instructions before the question, respondents spend more time answering because they are reading the instructions and answering more carefully. Not surprisingly, their answers are more accurate.
Here is an example of a typical question that has clarifying instructions after the question: (more…)
Tags: Online Surveys, research, Survey Design, survey respondents
Posted in Data Collection, Methods & Tools, Online Surveys, Survey Design, Survey Tips | Comments Off
Wednesday, January 30th, 2013
It is good to use multiple survey questions to measure a particular attitude or opinion, and for many years experts on survey design recommended reversing the wording on some questions to get a stronger and more valid measure. For example, if we want to measure retirement confidence we might ask respondents how much they agree with the statement “I am confident that I will be able to live comfortably in retirement” and “I worry about being able to make ends meet in retirement.”
A recent meta-analysis in the Journal of Marketing Research shows that fewer and fewer researchers are using reverse wordings. Why? Because reversed wordings introduce ambiguities, make surveys more difficult for respondents, and often lead to increased levels of measurement error. Still, survey item reversals are considered a best practice because (1) they ensure fuller measurement of an attitude or opinion, (2) they keep respondents from answering carelessly, and (3) help correct for agreement bias. For this reason, the authors advocate “the continued use of reversed items in survey research” but they also caution researchers to “use them with care.”
Here are several of their specific recommendations we found most useful: (more…)
Tags: best practices, measurement, response scales, Survey Design
Posted in Market Research, Survey Design, Survey Tips | Comments Off
Thursday, October 11th, 2012
Figure 1. Example of a Daily Field Report (Click Image)
A mistake often made by both professional and do-it-yourself researchers is letting a survey sit in the field without actively monitoring it. Once we design a survey and put it out there for people to respond, we just wait patiently (or get busy on another project) until we have data for analysis, right? But collecting data is never straightforward. It nearly always requires daily adjustments and decisions from the most senior members of a research team.
So at Versta Research, all fieldwork we conduct or oversee requires a daily and detailed fieldwork report that gives us visibility into all kinds of technical and conceptual issues that might affect the quality and outcomes of research. Figure 1 shows an example of a report; nothing fancy, but full of crucial data. As we review these reports, we watch for several warning signs and intervene where needed:
(more…)
Tags: data, Data Collection, data quality, Online Surveys, Sampling
Posted in Data Collection, Market Research, Methods & Tools, Omnibus Surveys, Online Surveys, Survey Tips | Comments Off
Thursday, July 12th, 2012
In a typical graduate-level research course on survey methods, students are taught to ask demographic questions about race, ethnicity, age, gender, income, etc. at the end of a survey. The questions are typically off-topic, and they sometimes scare away respondents who feel such questions are intrusive.
But with most surveys now fielded through online survey panels and with researchers trying to reach increasingly narrow targets, putting demographic questions at the beginning of surveys is common. Often we have strict demographic quotas and screening criteria, so demographics need to be assessed immediately to decide whether potential respondents qualify for inclusion in a survey.
(more…)
Tags: best practices, demographics, Market Research, Survey Design
Posted in Data Collection, Market Research, Survey Design, Survey Tips | Comments Off
Wednesday, April 25th, 2012
Most people who take surveys want to share their opinions, which is important for researchers hoping to get a few nuggets of data from willing respondents. The trouble is, if a survey it not written carefully, a respondent’s urgent desire to share their feelings may bias their answers to other questions.
Two marketing professors at Northwestern University’s Kellogg School of Management recently published a paper in the AMA’s Journal of Marketing Research identifying and documenting this unique kind of bias. They call it “response substitution.” (more…)
Tags: bias, customer satisfaction, satisfaction research, survey, survey respondents
Posted in Market Research, Methods & Tools, Survey Design, Survey Tips | Comments Off
Wednesday, February 15th, 2012
Data quality will suffer if respondents are bored with long surveys
It is hard to resist the temptation of asking “just one more question” when you’ve got an engaged respondent answering your survey questions online or on the phone. But it is crucial to do so because plenty of research shows that longer surveys result in bad data. Survey respondents may be willing to answer just one more question, but at some point the quality of information you get from them declines. Survey respondents become inattentive and offer lazy answers, or worse, they offer quick random answers just to get the survey over with.
At Versta Research we have a few rules of thumb for survey length based on (1) academic and industry research measuring data quality, (2) conversations with colleagues and suppliers throughout the industry, and (3) our ongoing experience of what works and what does not work. The maximum survey lengths we typically recommend are: (more…)
Tags: data, Data Collection, data quality, Market Research, Online Surveys, phone surveys, social media, survey, Survey Design, survey respondents, survey technology
Posted in Data Collection, Survey Design, Survey Tips | Comments Off
Wednesday, November 16th, 2011
Friends often solicit from me quick advice about conducting do-it-yourself customer satisfaction surveys. What questions should they ask? How many questions should they ask? What measures and scales should they use? And, of course, shouldn’t they be using NPS (Net Promoter Score) like everyone else?
I tell them that, by far, the most useful question they can ask is an open ended question that would be something like this: (more…)
Tags: insight, open-ends, satisfaction research, Survey Design
Posted in Survey Design, Survey Tips | Comments Off
Wednesday, October 5th, 2011
In response to last week’s newsletter, Is Your Research Good Enough for The New York Times?, which discussed hurdles of getting online survey research reported by some news organizations, a customer reminded us that online surveys can be difficult to sell internally as well. Too many people have been burned by junk data from online surveys.
One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits. If these respondents are not providing thoughtful answers, the data are suspect. All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.
But we should not rely on panel providers alone to ensure valid data. Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases. Here are typical indicators of potentially bad data:
(more…)
Tags: data quality, Online Surveys, open-ends, survey respondents
Posted in Data Collection, Methods & Tools, Online Surveys, Survey Tips | Comments Off
Wednesday, June 29th, 2011
One of the surprises of multi-mode research is that offering people a choice of how they want to complete a survey (online, by mail, by phone, etc.) does not necessarily boost response rates. An article in the most recent issue of Public Opinion Quarterly provides new evidence of this. The study showed that even in a population with full access to both mail and Internet options (and full literacy in both modes), a full mail survey achieves a higher response rate than a web-based survey. It also achieves a higher response rate than a choice of either mail or web. So much for our recent article on The Myth of Too Many Choices!
But the study went further to explore some ways of sequentially deploying multiple survey options and multiple modes of information that can substantially boost web-based survey response rates. Here are some key takeaways from the research: (more…)
Tags: Data Collection, Internet, Online Surveys, response rates, survey, survey respondents
Posted in Data Collection, Online Surveys, Survey Tips | Comments Off
Thursday, June 2nd, 2011
We have always been big fans of the Net Promoter Score (NPS) metric because it has convinced many firms to begin using customer satisfaction measurement scales that work better and that are tied to what people do rather than what people think. Eleven point scales (with points zero to ten) allow for optimal variation. They are intuitive and appealing: people quickly grasp the idea of rating something on a zero to ten scale, and are familiar with the idea from grade school. They also have a neutral mid-point, which is important for many customer satisfaction and loyalty studies.
But NPS questions do not make sense in many situations. Here’s one we saw last week—it’s a survey sent by Amazon to sellers who call regarding complicated issues with how their products are being displayed on the website or how payments are being transferred:
A Poor Use of the Net Promoter Question
(more…)
Tags: measurement, satisfaction research, Survey Design, tracking studies
Posted in Market Research, Methods & Tools, Survey Design, Survey Tips | Comments Off
