If something works and it keeps on working but you don’t know exactly why it works, what would you do? Our view is that you should keep doing it. Not everyone agrees with us. The American Association of Public Opinion Research (AAPOR) convened a task force to study online survey panels, and released their report last month (we posted a summary of findings last week). To us, the most jarring statement in the report was this:
“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”
Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters. Why? Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works. (more…)