In two words, sample size. Well, see y’all tomorrow…

Okay, so there’s a bit more to it. The use of what I call micropolls is now the norm. Current US population is a bit over 327,000,000 – three hundred and twenty-seven million human beings. In order to find out what they are thinking about a given topic, modern pollsters survey 1004 people. It can vary a bit but most will be just slightly over 1000.

If you try that with a question about whether they believe that the rhinoceros is a dinosaur you would NOT get accurate results. But ask who they want for president and the margin of error is usually around plus or minus three points. Why does it work fine with one and not the other? Well, it doesn’t.

What’s really happening is a process called weighting. For the record, weighting is perfectly valid and it WILL work – sometimes, kinda. Weighting is basically educated guessing and then ‘rebalancing’ the scales based on those guesses. If you know that two out of every three people always vote Republican no matter what (I am so totally making this part up – it’s an example, okay?) then when you have 600 of your 1000 folks saying that they are voting Democrat, you know you messed up somewhere and that the sample isn’t representative. But since you know that 2/3’s vote Republican, you just adjust the numbers accordingly.

This is perfectly fine if two conditions are met: first, that you have a reliable way to double check results periodically. We do – they are called elections. The second is that the population is stable in its behavior. That one is the problem child.

Pollsters use elections to check their accuracy and to adjust the numbers they use to weight their polls. In the US which is usually highly centrist and remarkably stable, that works more often than not. BUT when the population becomes unstable – when voters begin to change their behavior in significant numbers, the last election doesn’t tell you enough about the population to be able to guess how they will behave – and weighting starts to mess up the polls that weighting normally fixes.

We see voting behavior every two years – we rarely ask people their opinions on rhinoceros’. We can try to weight the first. We have no way to weight the second. But when the population becomes volatile – when lots of people start changing their minds and then their behavior – we can’t weight that, either. The polls become exactly what we see today – wildly variable. Sometimes right, sometimes wrong and sometimes just ridiculous. The pollsters are still desperately guessing – and those who happen to be most like the changing pop will guess better – but it’s still just guessing. Guessing with lots of math and numbers – but just guessing at the end of the day.

The way to do polls that aren’t guessing is to use much larger sample sizes. But large sample sizes are REALLY expensive. Pollsters don’t call just 1000 people – they call enough people to get 1000 responses. Counting automated calls, that’s at least in the tens of thousands assuming 1 in 10 pick up the phone and 1 in 10 of those answer the survey. It’s probably worse than that if we count the blocked calls.

I remember in Statistics 201 years ago being told minimum reliable sample size was 30%. But I could be remembering wrong so let’s use 10%. Ten percent of the US population is 32,700,000. Can you imagine the phone bill just trying to call that many people – assuming they all pick up and they all answer the survey? Limiting to likely voters might get it down to 7,500,000 – still extremely expensive to do. This is why modern pollsters are dependent on weighting.

So how do we fix it? Only two possibilities – wait until it straightens itself out or spend a huge amount of money doing much larger surveys. The truth is, if you’re not running a campaign, polls are just a past time. Yes, it’s an easy short hand for who’s winning – when it’s right, of course, but knowing which horse is first out of the starting gate doesn’t tell you which horse will definitely win. In truth, even at the best of times, the polls aren’t particularly useful until about ten weeks or so before the election. That’s when the undecideds are beginning to solidify and the polls start to reflect more reality than guesswork. Even then, if handicapping were perfect, no one would bet on horseraces. Same is true for elections. Not even the best polls with huge sample sizes are ever perfect predictors of the election outcome.

There’s a third option – stop relying so heavily on polls. News media love ’em – it’s a lot easier to quote poll results than to count yard signs or small donor donations. Thing is, those other two indicators were correct in 2016. The media sells a product – information. As long as we buy their polls, they will sell them to us.

Maybe we should stop buying.