The ugly truth about the fatal flaws of public polling and
political polling is starting to get some serious attention.
Prof. Cliff Zukin wrote in the June 21 New York Times Sunday
Review that the classic ideal of “probability sampling” has become more or less
impossible. Thus, it’s impossible to use statistical theory to determine the
reliability of the survey results. Thus, most survey findings are essentially
meaningless. Entertaining, perhaps, but meaningless.
Let’s be clear about the nature of the problem: it’s not
necessarily true that all survey results are horribly wrong. Rather, the sticky
point is that there’s no way to reliably measure the “margin of error” and
therefore there’s no way to determine whether a survey result—such as 49%
support Mary Poppins and 46% support Matt Damon—represents a “real” difference.
The sampling problem is caused by two things: widespread
usage of cell phones (instead of landlines) and increasing reluctance of Mr.
and Mrs. Citizen to consent to be interviewed.
Last year about 60% of adults said they exclusively or “mostly”
used a cellphone, and not a landline. That’s a problem because it’s procedurally
very difficult and quite expensive to try to get a “random sample” of cell
phone users. Many pollsters substitute cheaper polling via the Internet, which
violates strict statistical requirements for random sampling. Zukin explains the details here.
Forty years ago, in face-to-face and telephone surveys, a
response rate or completion rate of 80% and more was common and mandatory for
successful polling. That is, 80% of the people who were contacted (“sampled”) actually
completed the interview. In plain words, most folks were willing to talk to
survey interviewers. Last year, the prestigious Pew organization reported that
its typical response rate was a mere 8%.
I did survey research for 30 years. In plain words, a
response rate in the single digits means the survey is invalid. No meaningful conclusions
can be reached from a poll that collects data from only 8% of the originally
targeted sample. To add insult to injury, every polling organization cooks the
data by using computer techniques to “weight” the numbers so it looks like a
random sample of subjects was successfully surveyed.
Bottom line: you can have fun talking about the political
polls that are going to pop up as the 2016 presidential campaign unfolds, but
don’t believe anything they say is necessarily true.
Copyright © Richard
Carl Subber 2015 All rights reserved.
No comments:
Post a Comment