The authors looked at various types of biases that creep into business managers’ decisions when based on data presented to them.  They did this by conducting experiments with business school students and managers who are members of the American Marketing Association.  Some were presented with numeric data in tables, while others were presented with data in charts or graphs.  All tables, charts, and graphs were clear and well-designed.

The results?  When compared to an optimal decision based on a purely rational assessment of the data, decisions were typically biased, and “graphical formats that followed existing recommendation for the appropriate display of data did not reduce . . . biases compared with data presented in tables.”  Moreover, “neither real-world experience nor explicit training reduced these biases.”

In some ways this is surprising, because graphing data can often help us better (and more quickly) grasp its meaning.  On the other hand it is not surprising.  Graphs can be so visually compelling that they might hinder purely rational assessments.

In our view (and the authors’ as well) the research does not invalidate the need for effective data visualization.  A good chart can be a compelling piece of story.  But it is not the story, any more than a table of numbers can be the story.  A good chart is a communication tool.  So are good tables, and so are good sentences that weave together a compelling story.

At Versta Research we use chart and tables in about equal proportions (sometimes tables work better) and we always integrate them into a clear story with appropriate statistical modeling to lend support.  The solution to overcoming bias is not fancier graphics, but rather a clear presentation of information with a compelling synthesis and assessment of what it means.

—Joe Hopper, Ph.D.

One of the fascinating insights to emerge from our discussion was that social media is not only a new channel of information and data, but that it is fundamentally different from previous channels of data. As such, it opens up new areas of inquiry for our efforts. What is that fundamental difference? It is the networked nature of social media. As we code and tabulate people’s comments on social media as part of our CS&L research, potentially we have access to the networks of each person whose comments we are analyzing. We can know how many people are reading each comment, we can measure how strong and extensive the network of influence is, where it overlaps important segments of customers, and so on.

The implications of this are huge. The impact of loyalty among one’s best customers can now be defined not only in terms of how much they buy and how “sticky” they are, but also in terms of their influence among other customers and prospects. Smart CS&L research will not count everyone’s opinion the same, but will give more weight to those occupying central nodes of critical networks. There are implications for sampling as well. Is true random sampling necessary, or can “networked” sampling provided sufficient entry points that give visibility into the full population of customers?

In my view, this sort of network analysis will bring about a seismic shift in CS&L research, though not all panelists agreed. We were split about 50/50 on whether social media constitutes just one more channel of data to integrate, bringing greater precision to what we’ve always done, or whether it represents a more radical departure. A full report of our panel’s deliberations will be presented in the October 2010 issue of Marketing News, the AMA’s monthly magazine.

The stimulating discussion among panelists also highlighted the importance of remembering that good CS&L research requires ongoing thoughtfulness, intelligence, and curiosity. New tools and technologies are often easy to install, but their value is in offering new opportunities to bring deeper understanding and analysis to research.

Stay tuned – we’ll provide a closer look at some of those opportunities for better research (and reprints upon request) when the AMA publishes its report this fall.

—Joe Hopper, Ph.D.

For those of us at Versta Research, he is well known for his focus on understanding versus research:

“Basically, I’m not interested in doing research and I never have been,” he said. “I’m interested in understanding, which is quite a different thing. And often to understand something you have to work it out yourself because no one else has done it.”  (From an interview cited in the New York Times)

Why are we inspired by this?  Because even though we are interested in doing research, we do it for one reason:  To understand things.  In fact, we named our company from the social science concept (and German word) “verstehen” which means to understand.

Fundamentally, we define our work not by proprietary methods, or the specific tools that we use or develop, or the statistical procedures we implement.  We define it by our interest in and approach to understanding.  And that, of course, means using, learning, developing, and inventing whatever tools, processes, data collection techniques, and analysis that will get good answers.  In short, to achieve understanding, we do research.

Understanding is the outcome.  That’s what we really care about (and what you really care about) and what motivates our work.  Research is the means to that end.

If you have not done so, take a look at the homepage of our website.  It’s all about understanding.  We are committed to helping you understand:

• Your clients
• Your prospects
• The public
• Consumers
• Your competition
• Your customers
• Your data
• Your products
• Your image
• The world

If and when research can help you understand, we are here to help.

—Joe Hopper, Ph.D.

One critical trend that is transforming the research industry is the rapid adoption of mobile technologies, especially smartphones.  It transforms research in multiple ways.  It means that people disconnect their landlines, so we have to adjust our methods for phone polling.  But more importantly, it means people are accessible in multiple ways (text, phone, email) all the time, and wherever they are.  We can ask them for feedback instantaneously in the stores where they shop, or as they are making a decision we care about.  We can talk to them, send messages, ask them to respond to survey questions, ask them to take pictures or videos for us, and so on.  There are amazing opportunities for rich, immediate data for research that we could only dream about a few years back.

Quirk’s magazine recently ran a set of articles about trends in mobile research, focusing in particular on survey research.  We thought it is was worth repeating some of the insights highlighted in those articles:

• Surveys via mobile devices are increasingly possible, both SMS-based and web-based surveys
• Achieving representative samples for mobile surveys can be a challenge, however
• Attention spans for mobile surveys are short; ask no more than five or six simple questions
• Design surveys with small screens in mind
• Avoid long questions that need explanation
• Avoid long lists of answer options
• Avoid grid questions
• Take advantage of open-ended questions; people with mobile devices are capable (and often prefer) typing out responses on their phones
• Be sure to cover any costs that respondents might incur

Although nobody should be doing mobile research just because they can, keeping up with new tools, capabilities, and opportunities for rich data is one piece of doing smart and rigorous research.  Versta Research can advise, and help you take advantage of new technologies and methods when, and if, you need them.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/using-mobile-technologies-in-research/feed/ 0 http://www.verstaresearch.com/blog/bridging-the-quantitative-qualitative-gap/ http://www.verstaresearch.com/blog/bridging-the-quantitative-qualitative-gap/#comments Fri, 25 Jun 2010 12:56:49 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=639 The summer 2010 newsletter from Versta Research focuses on how to bridge the gap between quantitative research and qualitative research, whether it be market research or academic research.  Both methods give rich insights, and both offer compelling ways to summarize and communicate data.  But rarely does each method draw upon the strengths of the other.

How do you bring the two together?  In our view, you bring them together by investing in people who are trained in both methods and who have deep experience in both.  The newsletter article outlines:

• Why it is best to be fluent in both
• The strengths and promises of each method
• The qualities and training needed to excel at both
• The false promise of technology
• Practical ideas for incorporating the strengths of each into a research effort

Ultimately, research is all about asking questions, finding information and data, thinking about it, probing deeper, assimilating and synthesizing it, and then turning data into stories so that the research gets heard and understood.  Whether you use qualitative or quantitative methods, your data needs to tell a compelling human (business) story, and the best stories draw upon both.

Give us a call, and we would be happy to share with you examples and stories of our work that have successfully bridged the gap.  In the meantime, take a look at our June 2010 Newsletter for a deeper understanding of our approach, and what you can expect from our people.

—Joe Hopper, Ph.D.

Is this possible?  No.  It unfortunately confuses the tools of market research and public opinion polling with the interpretation and outcomes of research.  To be sure, better tools and technology help us do our work faster, smarter, and cheaper.  They bring sophisticated tools into the hands of smaller organizations who can then help businesses and media outlets with nimble and cost efficient solutions.  Market researchers are benefiting enormously from these new technologies and tools.

But ultimately these easy-to-use applications that deliver real time data and pie-charts give us … well, just data and pie-charts.  They don’t give us insights.  For insights, we need smart people who bring expertise, brainpower, and thoughtful creativity to the research effort — people who know how to design and implement studies, and then interpret and communicate information to answer critical questions.  When managers and clients see their own desktops stuffed with “auto-alerts-sent-to-key-stakeholders-enterprise-wide” from our newest suite of tools, they need people to answer challenging questions like, “So what?  Is this really true?  What does it mean?  How should I proceed?”

These are the kinds of questions we can help you with.  We can help you sort through the latest tools and can help you implement them within  your organization.  But don’t forget that tools are a means something bigger and more fundamental, like a story with a context, a puzzle, a question that needs to be answered.

—Joe Hopper, Ph.D.

Contrast this with a client for whom we were generating weekly data reports from a tracking study who said that her directive (and hence our directive) was that “nothing goes to my internal clients without insight.”  And so every week we cleared off our desks, closed the door, and examined the data for new angles and insights.  If we were unable to come up with a compelling, non-boring, challenging story, then surely our client’s client would not find it either.  And if nobody can find a compelling story in the data, why is the client spending money on it, and why are we being paid to do it?

In our view, high quality market research requires smart, thinking people who are:

• Curious about your problems
• Passionate about getting the right data to answer the right questions
• Diligent and detailed in turning that data into stories that give you insight

Nobody can do these things in their sleep.  The next time your research partner starts nodding off mid-sentence, “Why, yes, we have done this kind of work so many times we can do this project in our sleep . . .” take another look at the work they are doing for you and ask yourself whether everything they do delivers thoughtful insights you can use.  If it doesn’t, give us a call for a different perspective on how to do research.

—Joe Hopper, Ph.D.

How do we begin the research design process to ensure incisive and smart research that really helps our clients answer their critical questions?  Here are five key elements of the process for us, which we offer to you as tips for your own success when launching an internal research effort:

1.  Focus on the questions that need to be answered, not on research methods.

It is tempting to start the research process by saying “I need a survey about toothpaste preferences” or “We need focus groups about our advertising message.”  A critical first step in research design is to separate the business questions from the methods that will be used to answer them.  (Need help?  See our recent article on this topic, The Art of Asking Questions.)

2.  Consider how the research will be used.

Sometimes research findings are for internal strategic insight, while other times they are needed for communications and public release.  Whom you include in your sample, the kinds of questions you ask, and the answer scales you choose all depend on who needs to see the results, and how they will use them.

3.  Know what is known.

By this we mean that you must do your homework, because many aspects of a problem are likely known from other research, or could be answered easily with existing data.  Good research will build upon what is known, rather than replicating it.

4.  Play with opposites.

If your first impulse is to conduct a survey, consider what would happen if you did qualitative research instead.  And vice versa.  We recently conducted a powerful conjoint study that started out as an idea for focus groups, until the client realized how compelling it would be to have a mathematical model of preference.  The findings from this research informed a crucial and profitable decision for the company, and was deliberated at the highest executive levels in the company.

5.  Plan all steps.

Write a full plan for all steps of the research process, including sampling, questionnaire design, fieldwork, data collection, timing, budget, analysis, and reporting.  Believe it or not, every one of these steps will affect the questions you will want to ask in your research, and how you will ask them.

We’re serious when we say we love writing research proposals.  It’s because we love designing research, which is where brainpower and high level expertise add significant value.  Need help?  We’ll write one for you.  Or, if your current project is strictly do-it-yourself, call us anyway and we would be happy to offer ideas and tips.

—Joe Hopper, Ph.D.

Here are some great quotes we found to keep in mind as guiding principles:

A prudent question is one half of wisdom—Francis Bacon, 17^th century philosopher and scientist

If you do not ask the right questions, you do not get the right answers—Edward Hodnett, 20^th century poet and writer

The uncreative mind can spot wrong answers, but it takes a very creative mind to spot wrong questions—Sir Antony Jay, contemporary writer

Successful people ask better questions, and as a result, they get better answers—Anthony Robbins, contemporary self-help author and motivational speaker

As noted in the newsletter, one of the best ways to truly add value to the research that you do is to listen carefully to your internal clients and formulate the right questions.  Research is all about answering questions, which is, of course, all about asking questions.  Building fancy statistical models is fun and brainy and definitely worth bragging about at your next party, but few people in your organization care much about models.  They care about what answers those statistics provide to the fundamental business questions that keep them up at night.

Any questions? Give us a call. Struggling to find your questions? Take a look at our March 2010 Newsletter.  We are happy to help you find them.

—Joe Hopper, Ph.D.

Here are the steps involved in conducting a rigorous “gold standard” telephone survey of the U.S. population:

• You will need a sample of about 1,000 to 1,200 U.S. adults, which means you will need a list of at least 20,000 households to call.
• Generate your list of households using stratified random sampling (stratifying by region of the U.S.) from a list of land-line exchanges (there are currently  more than 69,000 of them).
• Add random digits to each exchange to create a complete telephone number – random digits ensure you will reach unlisted households.
• Determine a procedure to randomly select one adult in each household that you reach.  Asking to survey the adult who had the most recent birthday is one common procedure.
• Similar to the land-line sampling procedure, you will need to include a sample of randomly generated cell phone numbers.
• Try to call every number multiple times, varying the day and time of day at which you call.
• Weight the final data to adjust for sample imbalances on region, gender, age, race, ethnicity, marital status, education, number of adults in the household, number of landlines into the household, and presence of both landline(s) and cell phone(s).

For now, these are the procedures required to pass the rigor test of the most conservative methodologists, many of whom eschew Internet sampling.  But of course with telephone response rates declining dramatically and landlines disappearing quickly, the challenges of “gold standard” telephone research as outlined above may soon become insurmountable.  A recent research article in Public Opinion Quarterly concluded that sampling and coverage issues with both landlines and cell phone “call into question . . . the very future of telephone surveys.”

Whether you should launch a telephone survey or some other type of survey depends entirely on your objectives.  There are advantages and disadvantages to telephone surveys.  Knowing the steps involved and the pros and cons can help you make a smart decision.  Versta Research has deep experience conducting surveys by phone, Internet, mail, in-person, and in multiple combinations of these.  If you need help understanding and weighing your options, we would be happy to give you our best advice.

—Joe Hopper, Ph.D.