Americans increasingly don't trust political polls-- mostly because these surveys are so often wrong. But there is a more important reason why polls shouldn't be trusted. They rarely meet scientific standards.
Yes, sometimes a bad poll can pick the winner. So can pulling one name of the two candidates out of a hat. With one pick you have a 50/50 chance of being correct. If this picking the winner technique was to pull one of the two names out of a hat 10 times it might pick Joe Biden's name out 8 times. In other words, one set of 10 picks can deviate from the 50/50 probability. If Biden then won by a large margin would that validate the pickem-out-of-the-hat method of polling? No!
It would be foolish to endorse this method based on a successful prediction. To inform serious discussion during an ongoing political race it is essential to have trusted polls that stand on sound scientific principles.
As Director of a graduate program at CUNY, Brooklyn College, I taught research methods and statistics. The first thing I want to see in a report on a poll or survey--or any research--is the methodology. How many people participated in the poll? What was the makeup of the sample--e.g. age, sex, education, political affiliation, residence, and more? In other words, I want to know how well the sample represents the population under study as a whole.
Why these factors first? They comprise the first link in the chain leading to the results. If the first link is broken or markedly deficient why look further at findings that lack scientific credibility?
News anchors like to pretend that all polls and studies are valid, so they can bring out their panels each time there is a change--even slight--in the polling numbers. It's their job to fill up air time and keep their viewers watching. They couldn't do that if they began by evaluating the methodology and concluded that a poll result was questionable or even worthless.
Let's look at an example. On October 5th The New York Times reported that the New York Times/Siena College Poll in Pennsylvania showed Biden ahead by 7% (42%-49%)--an impressive lead. The only details in this report were that 706 likely voters were polled with a sampling error of 4.1 percentage points. This bare-bones media reporting is typical for almost all political polls.
The 4.1 margin of error in the Times/Siena poll suggests that the actual gap between Biden and Trump could be smaller than 9%, but still a significant lead for Biden. However, the sample size of "706 likely voters" is woefully inadequate for assessing the Pennsylvania population of 12.8 million with over 8 million registered voters. It is highly unlikely, if not impossible, for a sample of 706 likely voters to represent the diversity in the Pennsylvania population equivalent to their representation in the general population of Pennsylvania voters--the factors I just cited. Yet Small non-representative samples--even less than 706-- are typical of political polls.
Like most political polls, the Times/Sienna poll was conducted over the phone. Unfortunately, the response rate to telephone queries is in free-fall--down to as little as 6% according to the most recent report by the Pew Research Center. We don't know how many thousands were called to get the 706 responses in the Times/Siena poll. Thousands of those called obviously didn't answer or refused to participate. How many are never reported? Why is that figure important?
Pollsters like to say they are choosing subjects using the respected method of random sampling to avoid bias--like picking every 20th name in a telephone directory. Good intention. But most who are called refuse to participate. So rather than the pollsters picking a random sample, the small number that responds is picking the pollsters.
Are the ones who are willing to cooperate representative of the entire population? Probably not. As the Pew Research Center puts it: "The key to survey research is to have a random sample so that every type of person has an equal chance of having their views captured...Polls of volunteers would violate this principle...the kinds of people who might volunteer for our polls are likely to be very different from the average American." Yet the small numbers who agree to be interviewed in almost every poll are in effect volunteers.
The way pollsters address the paucity of representation of diversity in sampling is by "weighting." Simply stated, weighting is a statistical technique of compensating for inadequate sampling by assigning a mathematical number--giving greater weight to the under-represented number of participants in a category. Each weighting, though, degrades the poll as expressed in the margin of error (also called error of measurement). The greater the margin of error the less likely the accuracy of reported results. That's why it's important to know specifically how many factors are weighted. But that too is rarely if ever reported by the media.
In the case of the Times/Siena poll, I had to go to the site of the organization that conducted the poll to find out: The Siena College Research Institute. There I learned that virtually every component in the skimpy sample was deficient and therefore had to be weighted: "The data was weighted by party, age, race/ethnicity, education, region, gender, and voter likelihood." Not surprising as tipped off by the small 706 sample size.
An analogy will explain weighting and why it degrades the validity of polls.
There is a story of a Russian peasant couple who splurge with the few rubles they recently acquired. They treat themselves to a few potatoes and are excited at the prospect of eating potato pancakes just like rich people. They prepare the potatoes but when it comes to adding the other ingredients they realize they don't have them. So they use substitutes. Don't have the proper oil? No problem--replace it with a different ingredient. Don't have the required spices? No problem--replace them with substitutes--or leave them out.
(Note: You can view every article as one long page if you sign up as an Advocate Member, or higher).
1
1
1
