The ugly truth about the fatal flaws of public polling and political polling is starting to get some serious attention.
Prof. Cliff Zukin wrote in the June 21 New York Times Sunday Review that the classic ideal of “probability sampling” has become more or less impossible. Thus, it’s impossible to use statistical theory to determine the reliability of the survey results. Thus, most survey findings are essentially meaningless. Entertaining, perhaps, but meaningless.
Let’s be clear about the nature of the problem: it’s not necessarily true that all survey results are horribly wrong. Rather, the sticky point is that there’s no way to reliably measure the “margin of error” and therefore there’s no way to determine whether a survey result—such as 49% support Mary Poppins and 46% support Matt Damon—represents a “real” difference.
The sampling problem is caused by two things: widespread usage of cell phones (instead of landlines) and increasing reluctance of Mr. and Mrs. Citizen to consent to be interviewed.
Last year about 60% of adults said they exclusively or “mostly” used a cellphone, and not a landline. That’s a problem because it’s procedurally very difficult and quite expensive to try to get a “random sample” of cell phone users. Many pollsters substitute cheaper polling via the Internet, which violates strict statistical requirements for random sampling. Zukin explains the details here.
Forty years ago, in face-to-face and telephone surveys, a response rate or completion rate of 80% and more was common and mandatory for successful polling. That is, 80% of the people who were contacted (“sampled”) actually completed the interview. In plain words, most folks were willing to talk to survey interviewers. Last year, the prestigious Pew organization reported that its typical response rate was a mere 8%.
I did survey research for 30 years. In plain words, a response rate in the single digits means the survey is invalid. No meaningful conclusions can be reached from a poll that collects data from only 8% of the originally targeted sample. To add insult to injury, every polling organization cooks the data by using computer techniques to “weight” the numbers so it looks like a random sample of subjects was successfully surveyed.
Bottom line: you can have fun talking about the political polls that are going to pop up as the 2016 presidential campaign unfolds, but don’t believe anything they say is necessarily true.
Copyright © Richard Carl Subber 2015 All rights reserved.