Arbitron's sampling methodology is FAR more importantthanMrorMrsDePetro's misdeeds

[Garrett Wollman]

<<On Sat, 23 Aug 2008 12:31:22 -0400, "Dan Billings" <billings@suscom-maine.net> said:

> The sampling errors gave a boost to Kerry, who led in all six releases of 
> national exit poll results issued on Election Day by the National Election 
> Pool (NEP), the consortium of the major TV networks and the Associated Press 
> that sponsored the massive survey project.

> http://www.washingtonpost.com/wp-dyn/articles/A64906-2004Nov20.html

This article betrays a complete failure to understand the nature of
statistical sampling.  (Which is sadly common among reporters of all
stripes, but particularly egregious in a "post mortem" story like this
one.)

All polls are published with a "margin of sampling error", which
depends on two factors: the number of people interviewed, and the
confidence level.  By convention, nearly all (honest) social-science
researchers, including pollsters, report their margin of sampling
error at 95% confidence.

Given the sample sizes reported in the article, the results discussed
would all have been well within the margin of sampling error at 95%
confidence, and therefore no conclusions could or should have been
drawn from them.

(What does 95% confidence mean?  It means that there is a one in twenty
chance that the sample does not reflect the population.  Some
engineering disciplines require stronger confidence levels, 99% or
99.9%, but it would require a huge sample size to get useful polling
results at that level.  On the other hand, sometimes dishonest
organizations will conceal or reduce the confidence level to make
their polling results seem more conclusive than they really are.)

Ask Arbitron what their 95% confidence margin of sampling error is.
You may find that the difference between first and tenth place is
entirely within the margin of error.  (And there will still be a one
in twenty chance that it's completely wrong.)

-GAWollman