Don’t Stop Your Straight-Liners
Some respondents answer survey questions with the same answer over and over again. They agree to everything, or they give identical numeric ratings to all items. It’s called “straight-lining,” and we often consider this a sign of poor data quality. I was intrigued by an approach to mitigating the problem I learned last week, shared by a friend who was straight-lining his way through a survey, and then got the following message:
“You responded the same several times. Please consider your answers carefully.”
This did make him slow down and think more carefully, my friend said. And so it raised the question for me: should Versta Research build this real-time intervention into our surveys as well?
We decided no, and here’s why:
- Straight-lining is often valid. Most respondents are thoughtful and attentive, and for them, a straight line of answers reflects the truth. Cautionary statements coach them to begin shifting their responses, making subtle distinctions in places they usually do not make. Sure, this will increase variation (usually a good thing) but it may decrease validity and reliability, introducing a more serious source of error.
- Straight-lining helps flag bad data. Invalid straight-lining is easy to spot and a good way to identify respondents who game surveys or who race through without reading the questions. If a person straight-lines a majority of the question sets, and completes the survey faster than 90% or 95% of other respondents, chances are good they’re giving you bad data. The last thing I want to do is coach these respondents into behavior that makes it hard to detect them.
Should researchers worry about straight-lining? Absolutely. Should we cut straight-liners from our data? Yes, if we have solid criteria for identifying them. But should we intervene with attempts to eliminate straight-lining behavior from our data collection? No. Straight-lined data are valuable data because they often reveal the truth about who our respondents are, how they behave, and oftentimes what they think.