Many of us in marketing research have been deploying web surveys for over ten years, and web surveys are, by far, the dominant mode of data collection in our industry nowadays.  But our techniques and methods are an amalgam of practices adapted from other data collection modes, learned in part through trial and error, and taught to others through channels more akin to oral traditions.  So it is helpful when our academic colleagues manage to document and codify the art and science of what we do.

A new Handbook of Web Surveys does just that.  Among other things, it reminds us that whatever the survey mode—mail surveys, phone surveys, in-person surveys, mobile surveys, or online surveys—the key to rigorous research is bringing together theory, logic, mathematics, and practicality.

The biggest challenges for web surveys are that (1) not all people have Internet access, introducing the potential for coverage bias, and (2) most web surveys rely on volunteer respondents, introducing the possibility for self-selection bias and non-response bias.

There are ways of correcting for these biases, primarily through careful adjustment of the data through weighting.  A highlight of this handbook is that it reviews the complex ways in which weighting can and should be done for web surveys, including the use of regression estimates, raking (also known as rim weighting) and propensity scores.  Indeed, as one recent reviewer of the handbook noted:

The chapter on sampling and the later chapters on self-selection (chapter 9), weighting adjustment (chapter 10) and response propensities are central to statistical analysis of Web survey data, and the concepts treated in these chapters are at the core of debates on the scientific use of Web surveys. The authors should be complemented on the accessible way they introduce and describe these topics.

If you do any kind of survey work, you need to understand these issues.  You need to understand them at a conceptual level, and you need guidelines on how to implement them at a practical level.  This handbook will help.  Versta Research can also help.  We have expertise in complex quantitative methods, including the use web surveys for scientific and market research as well as for public opinion polling.  Please feel free to give us a call.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/just-published-handbook-of-web-surveys/feed/ 0 http://www.verstaresearch.com/blog/how-to-estimate-the-length-of-a-survey/ http://www.verstaresearch.com/blog/how-to-estimate-the-length-of-a-survey/#comments Wed, 07 Dec 2011 20:50:11 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1771 In Versta Research’s Winter 2011 Newsletter, published just this week, we describe a simple method for estimating how long it will take respondents to complete surveys.

Here we offer the “Versta Digest” version as a handy reference card.  Once you get the hang of it, you don’t need the examples and explanation.  You just need to know the rules.  We recommend reading the full article first, so you know what we’re talking about when it comes to “points.”  Then, when you need a refresher or a reference source, consult these rules:

• One point for each simple question or scaled response
• One point for every two response options in a multiple choice question
• One point for each row in a grid question
• Two points for any response that requires mental calculation
• Three points for every short response to an open-ended question
• One point for every three sentences of extra text that respondents must read

Then:

• Tally up the points
• Divide by 8 for online survey length (in minutes)
• Divide by 8 then multiply by 1.5 for phone survey length (in minutes)

The system is straightforward, easy to learn, and easy to execute.  It is a method that really works and that we have validated against hundreds of different types of surveys over the past several years.

Once you know how to determine survey length, you can think more strategically about the ideal survey length to optimize the value and content of a survey within your budget.  Call us at 312-348-6089 with any additional assistance you may need.

–Joe Hopper, Ph.D.

Friends often solicit from me quick advice about conducting do-it-yourself customer satisfaction surveys.  What questions should they ask?  How many questions should they ask?  What measures and scales should they use?  And, of course, shouldn’t they be using NPS (Net Promoter Score) like everyone else?

I tell them that, by far, the most useful question they can ask is an open ended question that would be something like this:

If there were one thing you would like us to improve, what would it be?

You can tailor the wording to your specific market, service, product, or situation.  Best of all, you do not need to code and tabulate all the responses, although you could and might want to.  You just need to read through them and act on them if it makes sense.

I offer this advice with conviction based on the experience of a business colleague who implemented an ongoing web satisfaction survey ten years ago.  I suggested he include a question like this.  He has told me several times over the years it is the only data he regularly reviews and acts upon.  Almost invariably a customer will point him towards a functional glitch on his website or offer a simple suggestion that would make it easier for customers to buy.

He asks other survey questions as well, measuring satisfaction across several dimensions.  And surely a robust effort to analyze that data might tell him what drives satisfaction and how to boost his levels of satisfaction from 85% to, say, 90%.  But what he really wants to know day in and day out is what he can fix to keep improving, and whether any of his customers have a great idea he never thought of that he can easily incorporate.  He gets that with this one question.

His experience is a testament to the idea that one question can provide a great deal of useful insight.  Sometimes one question is all you need.  If so, you’re all set.  If not, give us a call.  We’ll advise you on the next nine questions you need on your survey as well.

–Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/the-one-question-you-need-on-your-survey/feed/ 0 http://www.verstaresearch.com/blog/the-problem-with-maxdiff/ http://www.verstaresearch.com/blog/the-problem-with-maxdiff/#comments Thu, 03 Nov 2011 21:38:05 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1714 MaxDiff is a powerful method and it is increasingly popular among market researchers.  But it is not always the best choice for measuring the importance of attributes, and here’s why.

Suppose you want to measure the importance of 12 attributes for a new product or service.  If you know ahead of time that consumers are going to say that all 12 are extremely important to them, then MaxDiff is an excellent method for differentiating among the attributes so you can focus on the top two or three that matter most.

But what if you don’t know that all 12 attributes are extremely important?  Maybe none of them are.  Maybe they run the gamut from unimportant to extremely important.  The problem with MaxDiff is that it only tells you the importance of attributes relative to each other, but it won’t tell you whether the attributes are important.  The MaxDiff model will assign ratio-level numbers so that you can rank and quantify the importance of each attribute vis-à-vis the others.  But it will not anchor the attributes in a meaningful way.

This week we are designing a study in which we want to differentiate among attributes, but we also want to measure the gap between satisfaction and importance for items that are truly important to our target market.  We cannot do that with data from a typical MaxDiff study.  So we are using an old fashioned importance rating scale instead.

As always, it is critical to think about the story you want to tell with your research data, and then work backwards to the design and the choice of methods.  In many cases MaxDiff is the perfect tool.  In other cases it will leave you with data that is difficult to apply to critical questions.

Feel free to give us a call if you need some help deciding among the best methods for your research, whether it be MaxDiff, other conjoint techniques, or something else entirely.  We’ll help you focus on the story you need to tell and on the research design you need to tell it.

–Joe Hopper, Ph.D.

But NPS questions do not make sense in many situations.  Here’s one we saw last week—it’s a survey sent by Amazon to sellers who call regarding complicated issues with how their products are being displayed on the website or how payments are being transferred:

A Poor Use of the Net Promoter Question

The problem with this question is that any honest respondent will almost certainly answer “very unlikely” and give a score of zero.  The support representatives are anonymous, so it is impossible to recommend them.  If you call or email Amazon, you cannot request a specific representative, making the issue moot.  And even if you could recommend a specific representative, how often would you have the opportunity to make such a recommendation?  Few sellers talk to other sellers about the display and payment problems they are having.  So all in all, how likely do you think most sellers are to recommend the specific person they talked with?  If we have any faith in the reliability and validity of survey data, it should be about zero.

The hapless support representative likely got a lousy score (there goes her bonus!) not because she did a poor job, but because whoever is in charge of this tracking survey didn’t think very hard about NPS, how to use it and when to use it.  More generally, somebody didn’t think very hard about how to do smart research.

Staying on top of recent innovations, important industry trends, and best practices is critical.  We can help you with that, with suggestions about the right ways to apply them so that you get valid insights.  Your customers and your service reps will thank you.  And the next time your manager and executive team rate you, they will likely recommend you for the next important research project or satisfaction tracking study (score 9 or 10).

—Joe Hopper, Ph.D.

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

A Poor Fit: Pie Charts and Numeric Scales

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.

This is not to say they are bad or fraudulent respondents.  Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions.  The problem with respondent error, it turns out, is poor survey design, which may involve biased or  ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.

Recent research shows that grid-style questions that look like this:

or this:

. . . hurt the reliability of answers from respondents.  The evidence for this conclusion comes from one of the leading academic authorities on survey design, Duane Alwin, at Penn State.  Here we quote from a Harvard reviewer who summarized the findings in a recent journal article:

“Of particular note is Alwin’s finding that the widespread survey practice of presenting items in batteries—sets of consecutive questions using the same response format—tends to yield less reliable responses than presenting them alone or in a series of topically related questions with differing response formats. He conjectures that “[s]imilarity of question content and response format may actually distract a respondent from giving full attention to what information is being asked” (p.180). Some might anticipate that similarity in response format would instead heighten reliability, by raising correlations among items in a battery. Alwin’s reliability estimates do not depend on within-occasion correlations between different items,however: the finding suggests that respondents vary across occasions in how they use a battery’s response format.”

In short, even if you have good respondents who are not “straight-lining” through your grids, they may be focusing more on the task of filling out the grid than on a thoughtful response to each item in the grid.  So instead, lay out your questions one at a time, like this:

And when you’re not sure what to do next, namely how to turn all of that now-reliable data into a story that you can really use, give Versta Research a call.

—Joe Hopper, Ph.D.

An article in the spring 2011 issue of Public Opinion Quarterly looks at trends over the last thirty years in Americans’ view of polling and market research surveys.   The news is mixed.  While the vast majority feel that public opinion polling is generally a good thing, fewer feel that market research surveys serve a useful purpose, and trust in the industry is not so great.  The most worrisome news (but not surprising, given the number of truly bad surveys flooding our lives nowadays) is the steady decline in people saying that participating in research is interesting and in their best interest:

Declining Satisfaction with Surveys

In addition to the chart above, key statistics outlined in the article include:

• Nearly nine out of ten (87%) feel that public opinion polls, on the whole, are a good thing

• Four out of five (79%) say that the last time they were interviewed for research, it was a pleasant experience

• Two-thirds (68%) agree that research surveys help manufacturers or other organizations produce better products and services for consumers

• About half (54%) feel that the research industry serves a useful purpose.

• Just one third (34%) trust pollsters to tell the truth, though twice as many (64%) believe that most polls give results that are “pretty nearly right”

• Just one-quarter (29%) think that survey organizations can be trusted to protect their privacy

• One-quarter (25%) feel that answering questions in polls or research surveys is a waste of time

Clearly the research industry (and our clients) can do much to improve.  Here are two quick ideas as a start.  (1)  Next time you are tempted to send yet another survey to your customers, ask whether it’s purpose is to benefit them or you—if it’s the latter, don’t send it.  (2) Next time you are tempted to send out a 45-minute survey with grid after grid of repetitive measures, work more carefully with the people who will use the research to specify which of those measures can truly be acted upon.

A more careful and thoughtful approach to research will benefit you and your customers, and it will boost the confidence that our publics have in research and polling.  Versta Research would be delighted to work with you in making this happen.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/what-people-think-of-surveys/feed/ 0 http://www.verstaresearch.com/blog/tips-for-surveys-on-smartphones/ http://www.verstaresearch.com/blog/tips-for-surveys-on-smartphones/#comments Thu, 27 Jan 2011 20:39:22 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1102 The technology to field surveys via mobile devices has been around for a while, but has not yet gained much traction (and for good reasons).  But with smartphones now proliferating at a remarkable pace, we may be in for a change.  Deloitte released their 2011 IT and technology predictions last week, arguing that smartphones are likely to account for almost half of computer spending during the year.

When it comes to surveys for market research and public polling, smartphones have bigger and better screens, better technology, and web capabilities.  But more importantly, smartphones are becoming the “device of choice” for tasks that were once the domain of our desktop computers. For example, web-based e-mail via PCs is on the decline, while email through mobile devices is growing rapidly.  And of course unless we’re doing phone surveys, e-mail is currently the primary means by which researchers recruit survey respondents.

Leora Lawton, a colleague in AAPOR (the American Association of Public Opinion Research) recently reported back with some tips on survey design for smartphones that she synthesized from a talk by Mario Callegaro of Google.  In brief, she reports:

• Avoid long surveys
• Do not use question grids
• Scales should be a maximum of five points
• Show one question per page
• Put navigation buttons on top
• Use vertical, not horizontal, buttons
• Avoid logos, progress meters, help links, and copyrights that take up space
• Avoid drop-down menus that require JavaScript

Of course technologies change quickly, and enhanced capabilities and tools for effective and sophisticated smartphone surveys are in development.  But for now these are good ideas to keep in mind.

Need help thinking about the best approach for your research, whether it be telephone, in-person, online, or smartphone-based?  Feel free to give us a call; we would be happy to help.

—Joe Hopper, Ph.D.

It is important to distinguish the two because unfortunately there is plenty of research not designed to answer important questions.  Research is frequently done because someone has a nagging feeling that they need more information to make better decisions or because doing research is considered a “best practice.”  Such research generates lots of data that marketing managers wonder what to do with.  Not surprisingly, they and their bosses start to question the value of research.

Our advice is to do market research only after formulating specific questions and information needs and only after you have a clear idea (in writing) of what you will do with the answers to those questions.  We have produced a whitepaper entitled The Art of Asking Questions (you can download it by clicking on the image) that outlines a useful process to help you formulate those questions.

The Art of Asking Questions: A White Paper from Versta Research

The key is to articulate five types of questions, including:

• The mission-critical questions
• The nice-to-know questions
• The red-herring questions
• The already-answered questions
• The look-elsewhere questions

Next, you outline at least two likely or possible answers to those questions.  And finally you describe what action you might take based on each possible answer.

Articulating specific questions and outcomes will not only vastly improve the quality of your research, it will also make it easier to bring a full executive team on board with the research.  Why?  Because it moves internal discussions away from the value of research per se, towards the importance of the questions that have been formulated and the business value of having answers.

Versta Research would be happy to help you formulate these questions and/or help you decide that research is premature.  Feel free to call us at 312-348-6089.

—Joe Hopper, Ph.D.