But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

–Joe Hopper, Ph.D.

• Thirty percent of U.S. households do not have a landline telephone
• An additional 16% have a landline telephone, but never or rarely use it to receive calls
• The percentage of households without landlines is increasing by about five to six percentage points each year
• Half of young adults under age 30 have no landline in their homes
• Half of adult renters have no landline in their homes
• Nearly four out of ten Hispanic adults have no landline in their homes

For anyone doing random sample surveys of consumers, the implications are huge.  Methodological purists insist that only phone-based surveys are rigorous, but clearly the biases of phone-based research can be severe, and no doubt they often are.

Putting aside the issue of research methods, however, we believe a more significant issue over the next few years will be the explosive growth in smartphone usage.  Although not captured in government surveys that track phone usage and availability, the Pew center estimates that one-third of all U.S. adults currently own a smartphone, and industry analysts predict that by the end of 2011 half of all cell phones in the U.S. will be smartphones.

Why is this is so critical for researchers?  Because more than any other technology since the widespread adoption of the Internet, smartphones are changing how consumers behave.  Consumers get up in the morning and check media apps before brushing their teeth.  Then while commuting to work or waiting for their computer to boot up, they purchase coupons for goods and services they might never have bought otherwise.  They map and track their locations, and expect product features and services that can anticipate what they need.

Smartphones are important for reasons far beyond narrow methodological concerns about how to execute surveys.  They are important because they are changing the very nature of what we research: how people behave and think, and what they buy, believe, want, and aspire to.  Indeed, just as we nearly always ask our research respondents to tell us their gender, age, and income, we now often ask whether they have smartphones.

Beyond anything else, market research is about understanding people (the verstehen in Versta), which means our focus at Versta is always on the “what, how, and why” of what matters in peoples’ lives in a rapidly changing world.  Smartphones matter in a more profound way than cell phones ever did.

—Joe Hopper, Ph.D.

]]> http://www.verstaresearch.com/blog/smartphones-matter-more-than-cell-phones/feed/ 0 http://www.verstaresearch.com/blog/how-to-measure-the-un-measurable/ http://www.verstaresearch.com/blog/how-to-measure-the-un-measurable/#comments Thu, 24 Feb 2011 19:41:05 +0000 Joe Hopper http://www.verstaresearch.com/blog/?p=1156 Yesterday nearly one hundred marketers and researchers met in Chicago to talk about new directions in marketing research.  Our topic:  Measuring the Un-Measurable.  The event was organized by the AMA’s market research group in Chicago, headed by Joe Hopper, president of Versta Research.  The event brought together professionals from companies such as Cargill, Allstate, US Cellular, Nielsen, Maritz, Versta Research, Sears, Aon Hewitt, the American Medical Association, the American Bar Association, and many others.

It was a fascinating discussion that highlighted some of the newer technologies and methods of research that can help us measure important—and hitherto unmeasurable—aspects of customer and buyer behaviors.  These include:

1.  Neurological brain responses to marketing stimuli.  Candyce Wisner and Matte Valle presented a case study of Truvia’s recent ad campaign that was tested and refined with millisecond-by-millisecond measurements of viewers’ brainwaves.  The data helped them identify images and moments in television commercials where viewers showed cognitive engagement with the product, and where they showed positive vs. negative emotions.

2.  Implicit memory that drives responsiveness to brands. Bruce Miller described techniques to discover words and imagery that stimulate brand responsiveness at unconscious levels.  He argued that customers are “wired” differently based on past experiences, and that the key to a brand’s success is finding the message that speaks to customers who are wired to love that brand.

3.  In-the-moment decision making as consumers consider which products to examine and purchase.  Kathy Doyle talked about qualitative methods of measurement made possible with mobile technologies, including “street talk” mobile interviews and “shop along” interviews.  These new techniques capture consumers’ spontaneous thoughts, actions, and reactions to products, environments, and experiences.

4.  The collective consciousness and sentiment around brands and other important issues, as manifest in social media.  Joe Colacurcio described new possibilities for measuring and understanding brand awareness, loyalty and advocacy as they are manifest in the (partially) unprompted world of forums, blogs, and other social media.

Versta Research was pleased to help lead this effort.  Every new technology and method in research provides more options for a deeper, richer, contextualized understanding of customers, prospects, and products.  Indeed, Versta Research derives its name and focus from the social science concept Verstehen, which means “to understand” in a much deeper way than traditional methods typically offer.  As previously un-measurable aspects of customer and buyer behaviors become newly measurable, we are three steps closer to that deep understanding our clients need.

—Joe Hopper, Ph.D.

The Role of Social Media in Customer Satisfaction and Loyalty Research

A partial transcript of our panel’s deliberations was just published in the October 2010 issue of Marketing News, the AMA’s monthly magazine.  Here is a quick summary of key points highlighted in the article:

1. There is no “one” social media, but many channels with different users, objectives, and purposes for which customers post their comments.  Data need to be interpreted within the context of the channel from which that data comes.
2. Most social media is used for listening and interacting with customers as part of a customer service and customer relations effort, rather than for research per se.  But whatever its purpose, social media can offer rich data for mining insights, which can inform a well-rounded CS&L research effort.
3. Social media channels are unlikely to replace traditional modes of CS&L research, especially survey-based research using carefully drawn samples.
4. Triangulating data from social media with findings from other sources, including survey research, in-bound call data, purchase history, and so on, can help pinpoint the most critical customer satisfaction issues that a company needs to address.
5. Social media data can help pinpoint the range and meaning of issues that need to be explored with traditional quantitative research.
6. Social media can be mined for quotes and richness to provide a deeper and more compelling understanding of quantitative data.

One fascinating topic we discussed that was not covered in the article was the potential for network analysis informing CS&L research.  In my view, this is where a seismic shift in how we do research will happen.  Not everyone agreed.  In fact, the final question I posed to the panel was whether they see a seismic shift happening in CS&L research as a result of social media.  Here is my tally of the answers:

• Yes it will create a seismic shift – 2 votes
• No it will not create a seismic shift – 1 vote
• Maybe not seismic, but definitely some shake-ups – 2 votes
• Too soon to tell – 1 vote

Feel free to contact us at Versta Research if you would like us to send you a copy of the published article.  And of course give us a call if you need to discuss your CS&L research needs, including how to integrate new technologies and data streams into your analysis.

—Joe Hopper, Ph.D.

One of the fascinating insights to emerge from our discussion was that social media is not only a new channel of information and data, but that it is fundamentally different from previous channels of data. As such, it opens up new areas of inquiry for our efforts. What is that fundamental difference? It is the networked nature of social media. As we code and tabulate people’s comments on social media as part of our CS&L research, potentially we have access to the networks of each person whose comments we are analyzing. We can know how many people are reading each comment, we can measure how strong and extensive the network of influence is, where it overlaps important segments of customers, and so on.

The implications of this are huge. The impact of loyalty among one’s best customers can now be defined not only in terms of how much they buy and how “sticky” they are, but also in terms of their influence among other customers and prospects. Smart CS&L research will not count everyone’s opinion the same, but will give more weight to those occupying central nodes of critical networks. There are implications for sampling as well. Is true random sampling necessary, or can “networked” sampling provided sufficient entry points that give visibility into the full population of customers?

In my view, this sort of network analysis will bring about a seismic shift in CS&L research, though not all panelists agreed. We were split about 50/50 on whether social media constitutes just one more channel of data to integrate, bringing greater precision to what we’ve always done, or whether it represents a more radical departure. A full report of our panel’s deliberations will be presented in the October 2010 issue of Marketing News, the AMA’s monthly magazine.

The stimulating discussion among panelists also highlighted the importance of remembering that good CS&L research requires ongoing thoughtfulness, intelligence, and curiosity. New tools and technologies are often easy to install, but their value is in offering new opportunities to bring deeper understanding and analysis to research.

Stay tuned – we’ll provide a closer look at some of those opportunities for better research (and reprints upon request) when the AMA publishes its report this fall.

—Joe Hopper, Ph.D.

The idea seems crazy, but as reported in Science, researchers at Carnegie Mellon University have found that certain kinds of twitter data can give them a good read on public sentiment. The looked at things like positive or negative comments about President Obama and found results that aligned with traditional polls. Comments about finances and savings aligned with consumer confidence polls. What does the polling industry think? Here’s what one colleague says: “I believe that I am now going to run to a dark corner of the house and cower in fear of what may come.”

To be sure, we are not recommending that you rely on social media to accurately measure overall public opinion. No way. But in our view, these findings may lead to new insights about how people, individually and collectively, behave and think, and how research — both academic and practical — can harness these new forms of data to measure markets and the social world.  The findings may also force us to re-think our theories of statistical inference that rely on random sampling. There is much research to be done before we will know, but possibilities for social media someday offering insights that are as statistically valid as our current methods is intriguing and nothing to cower about.

Thinking about measuring social media as part of your research plan? We would be happy to advise you. We can help you explore new options in research while ensuring that your research and findings remain rigorous and defensible.

—Joe Hopper, Ph.D.