Posts Tagged ‘data quality’
Thursday, October 11th, 2012
Figure 1. Example of a Daily Field Report (Click Image)
A mistake often made by both professional and do-it-yourself researchers is letting a survey sit in the field without actively monitoring it. Once we design a survey and put it out there for people to respond, we just wait patiently (or get busy on another project) until we have data for analysis, right? But collecting data is never straightforward. It nearly always requires daily adjustments and decisions from the most senior members of a research team.
So at Versta Research, all fieldwork we conduct or oversee requires a daily and detailed fieldwork report that gives us visibility into all kinds of technical and conceptual issues that might affect the quality and outcomes of research. Figure 1 shows an example of a report; nothing fancy, but full of crucial data. As we review these reports, we watch for several warning signs and intervene where needed:
(more…)
Tags: data, Data Collection, data quality, Online Surveys, Sampling
Posted in Data Collection, Market Research, Methods & Tools, Omnibus Surveys, Online Surveys, Survey Tips | No Comments »
Wednesday, May 30th, 2012
Cost matters when you choose a sample or panel provider for your survey because there are good panels and bad panels. Bad panels provide survey respondents at cheap prices. But they do a lousy job managing and screening their members. Not surprisingly, a good portion of the data you get from bad panels will likely be lousy.
A recent study entitled “Dirty Little Secrets of Online Panel Research” by one of our industry colleagues described and documented lousy panel management practices of some companies. Mystery shoppers joined and participated in online surveys offered by nearly all of the leading panel companies that most of us rely on. Here are some of the “worst practices” they uncovered: (more…)
Tags: data quality, Online Panels, Online Surveys, panels
Posted in Data Collection, Methods & Tools, Online Surveys, Sampling | No Comments »
Thursday, May 24th, 2012
Bad data is worse than no data. With no data, at least you have intuition and experience to rely on. Bad data, on the other hand, will result in “findings” that are likely to spoil important decisions.
With most surveys fielded online nowadays, we have lost the opportunity for phone (or in-person) interviewers to assess the quality of a respondent’s answers. On the other hand, with online surveys there are all sorts of useful indicators to help assess whether respondents are providing truthful responses. This is because almost nobody takes a survey purposefully to give wrong or misleading data. Instead, they are bored, lazy, or irritated by your less-than-optimal survey design. Or they are trying to get through a survey quickly to collect an incentive payment or sweepstakes entry.
There are tell-tale signs of when respondents are giving bad data. We look for them in almost every survey we conduct or dataset we analyze. Here are six ways we suggest to identify bad survey data: (more…)
Tags: data quality, Online Surveys
Posted in Data Analysis & Analytics, Data Collection, Market Research, Methods & Tools, Omnibus Surveys, Online Surveys | No Comments »
Wednesday, February 15th, 2012
Data quality will suffer if respondents are bored with long surveys
It is hard to resist the temptation of asking “just one more question” when you’ve got an engaged respondent answering your survey questions online or on the phone. But it is crucial to do so because plenty of research shows that longer surveys result in bad data. Survey respondents may be willing to answer just one more question, but at some point the quality of information you get from them declines. Survey respondents become inattentive and offer lazy answers, or worse, they offer quick random answers just to get the survey over with.
At Versta Research we have a few rules of thumb for survey length based on (1) academic and industry research measuring data quality, (2) conversations with colleagues and suppliers throughout the industry, and (3) our ongoing experience of what works and what does not work. The maximum survey lengths we typically recommend are: (more…)
Tags: data, Data Collection, data quality, Market Research, Online Surveys, phone surveys, social media, survey, Survey Design, survey respondents, survey technology
Posted in Data Collection, Survey Design, Survey Tips | No Comments »
Tuesday, January 24th, 2012
Election years are a perfect time to learn about (and brush up on) the fundamentals of survey research. Not only are the airwaves inundated with public opinion polling, but methodological experts are called upon to talk about developments and current best practices as new technologies and methods become central to measuring consumer and public opinion and behavior.
This week the Poynter Institute is offering a webinar of particular interest for PR professionals, whether or not you care about political polls. We also recommend it for any marketing professional because this type of polling is an exemplar of what all marketing research tries to achieve in measuring what people think and what they are likely to do. (more…)
Tags: data quality, journalism, Market Research, media, news, omnibus, public opinion, Public Polls, Public Relations
Posted in Public Polls, Public Relations, Resources and Recommendations | No Comments »
Wednesday, October 5th, 2011
In response to last week’s newsletter, Is Your Research Good Enough for The New York Times?, which discussed hurdles of getting online survey research reported by some news organizations, a customer reminded us that online surveys can be difficult to sell internally as well. Too many people have been burned by junk data from online surveys.
One problem with online panels is that some respondents (a small minority) participate only to get paid in cash or redeemable credits. If these respondents are not providing thoughtful answers, the data are suspect. All panels have the problem, though some are worse than others; reputable sample providers work hard to identify and remove fraudulent respondents from their panels.
But we should not rely on panel providers alone to ensure valid data. Buyers of panel surveys should always look at the data case by case to identify and remove suspicious cases. Here are typical indicators of potentially bad data:
(more…)
Tags: data quality, Online Surveys, open-ends, survey respondents
Posted in Data Collection, Methods & Tools, Online Surveys, Survey Tips | No Comments »
Thursday, May 12th, 2011
We are often surprised by the number of senior researchers in the market research industry who never touch raw data. Often they don’t even have the tools, since “data processing” is outsourced to lower levels or other countries. It is surprising because we almost always engage in work where getting into the data and puzzling over anomalies or hypotheses yields much deeper insight.
Here is an example of how critical it can be to look closely at your data, and in this case, very early in the data collection process. We launched an online survey last week and got reports back from our sample supplier that incidence was just one-third of what we expected, which would have serious feasibility and cost implications.
But once we looked at their report portal, we saw that for every qualified respondent completing the survey, two qualified respondents quit before finishing. That’s an unusually high ratio of “suspends” as we call them. So what was the problem? Were we just getting lousy respondents who did not want to seriously participate in a survey? Was the survey was too difficult, tedious, boring, or confusing? One source of answers (rarely examined) is to look at the data question by question to identify where in the survey people are quitting.
The story in this data: Something is wrong with your survey
(more…)
Tags: data, Data Collection, data quality, insight, research, stories, survey, survey respondents
Posted in Charts and Data Visualization, Data Analysis & Analytics, Data Collection, Methods & Tools, Survey Design | No Comments »
Saturday, April 16th, 2011
Among the many sources of potential error that can affect surveys are respondents themselves. They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.
This is not to say they are bad or fraudulent respondents. Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions. The problem with respondent error, it turns out, is poor survey design, which may involve biased or ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.
Recent research shows that grid-style questions that look like this:
or this: (more…)
Tags: bias, Data Collection, data quality, Market Research, Online Surveys, Survey Design, survey respondents
Posted in Data Collection, Market Research, Methods & Tools, Online Surveys, Survey Design, Survey Tips | No Comments »
Wednesday, March 30th, 2011
One resource that we give far too little thought in market research are the people who offer their time and thoughts about the stuff we are researching. They are truly the lifeblood of market research.
An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys. The news is mixed. While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great. The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:
Declining Satisfaction with Surveys
In addition to the chart above, key statistics outlined in the article include: (more…)
Tags: Data Collection, data quality, Market Research, public opinion, Public Polls, Survey Design, survey respondents
Posted in Data Collection, Future Trends, Market Research, Public Polls, Public Relations, Survey Design | No Comments »
Thursday, December 30th, 2010
The latest data from the CDC’s National Health Interview Survey show that one quarter (25%) of U.S. adults do not have land-line telephones in their homes. So if you conduct a traditional random-digit-dial (RDD) phone survey, you will automatically be excluding one quarter of the population. Does it matter, given that surveys rarely interview everyone anyway? Probably. If those 25% are different from the remaining 75% in important ways, then excluding them will skew your survey findings.
(more…)
Tags: bias, Data Collection, data quality, ethics, Market Research, phone surveys, public opinion, Public Polls, research, Sampling, Survey Design, survey respondents
Posted in Data Collection, Methods & Tools, Public Polls, Sampling | No Comments »
