Nothing. Thanks for dropping by.
Okay, okay, so there might be more to it than that. First thing to know about any poll is that it’s a snapshot of what’s going on at the minute it’s taken. It can be meaningful days afterwards if the electorate is stable – and it can be completely wrong two hours later – even if it was done perfectly (which nothing is – pollsters come in the same size human as the rest of us). But for the most part, it’s a solid glimpse into what’s going on at that moment which we assume is usually going to continue to be about the same for a few days afterwards.
Assuming you did it right, asked the right questions, asked enough people, got enough responses, didn’t get too many jerks trying to muck it up and actually did the math correctly. That last one’s tough not because the formulas are particularly difficult – they aren’t – but because statistics is like a woodworker’s shop – oodles and oodles of spiffy tools that are totally useless if you don’t know what you’re doing with them. No, you can’t do scrollwork with a hammer – and averages are the rough equivalent of that adage ‘when you have a hammer, every problem looks like a nail’.
What this means is a market saturated with polls – like the media has been for the past thirty years – is giving bad information more often than not. Polls aren’t news – heck, they may not even be representative of actual opinions of the actual group they are measuring. They’re just hammers in the tool bag of political analysis. Great when you need to drive a nail; pretty awful when you want to cut a board.
Let’s look at the RCP Average.
Polling Data
| Poll | Date | Sample | Approve | Disapprove | Spread |
|---|---|---|---|---|---|
| RCP Average | 8/14 – 8/25 | — | 46.9 | 49.1 | -2.2 |
| Rasmussen Reports | 8/23 – 8/25 | 1500 LV | 44 | 55 | -11 |
| Economist/YouGov | 8/21 – 8/24 | 1242 RV | 48 | 46 | +2 |
| Politico/Morning Consult | 8/21 – 8/24 | 1996 RV | 50 | 48 | +2 |
| USA Today/Suffolk | 8/19 – 8/23 | 1000 RV | 41 | 55 | -14 |
| The Hill/HarrisX | 8/20 – 8/22 | 2846 RV | 49 | 43 | +6 |
| Reuters/Ipsos | 8/18 – 8/19 | 1002 A | 46 | 49 | -3 |
| NBC News/Wall St. Jrnl | 8/14 – 8/17 | 790 RV | 50 | 48 | +2 |
Notice the — under Sample in the RCP Average line? It means they didn’t take a sample – they are just averaging the totals of the polls below it. It’s just about the most useless thing I can think of – there’s no way that all those polls have the same margin of error – assuming they all use the margin of error. The average of a grape, a zucchini, a kumquat, a bell pepper and an apple is not a fruit salad. For the average to be meaningful the polls all have to be substantively the same – and they aren’t.
So why do pollsters do it? To be able to do now what we used to do all the time in the 1970’s – give a single number as the answer. But the reason we could do that in the Seventies is that there was only one polling organization that anyone bothered to cite – Gallup. By the late 1990’s, news media were using their own polling data – Gallup wasn’t cheap. But quickie phone polls were not only cheap, they were great for ‘finding’ the ‘right’ results.
People did catch on – and there were several big scandals about news media polling. Mostly because they were so shoddy that they were seldom proven right when the election rolled around. Once the media wiped off all the egg, they started paying for better done – if not always well done – polling.
And where there’s money to be made there are plenty of folks willing to fill the void. Competition is a great thing – but most people haven’t a clue how polls actually work (hence even commentators citing the RCP average). So we get a hodge-podge of polls varying from usually well done (Reuters, Rasmussen) to the usually not (USA Today). If you don’t know what to look for under the hood, it’s hard to know which to trust – if any.
And even bad polling organizations get it it right occasionally. I’m inclined to think that USA Today is actually closest to the truth in the numbers listed above. That may very well be my own bias but it seems unlikely that Politico and the Economist are getting such high numbers on those dates.
Which brings up the other monkey in the room – remember what I said about ‘snapshots’? NBC News poll is nearly a full week ahead of the others. In a slow news cycle, that is still a problem but only for us polling geeks. In a fast moving news cycle like the Afghanistan debacle? NBC’s poll has nothing to do with the polls done by Rasmussen – they are simply too far apart to be measuring the same thing.
That monkey’s cousin is that all the polls are done over a span of several days. Just like a camera can do cool stuff by lengthening the exposure time, so can polls. Mostly, for polling it means getting enough people to actually answer the poll (maybe having one a week isn’t a great idea with umpteen polling organizations doing umpteen polls each week). But like a long exposure on a photograph, long durations can distort a poll – and yes, that’s especially true in a volatile news cycle.
So what do all the fancy numbers tell us? The average is just a benchmark – it’s not a great way to measure things under the best of circumstances (remember a C is supposedly an ‘average’ grade – but would most parents think that most kids should be getting C’s?). What about the median? That’s basically the number in the middle (the average isn’t the middle – yeah, I know. I blame public education and the letter C).
What’s the median of the above? Let’s take a look. Since we have an odd number of results, we can do this the easy way. From lowest to highest the results are: 41, 44, 46, 48, 49, 50, 50. The median is the middle number – 48. (Who said all math has to be hard?)
Er, what else have we got, Statistics? Well, we have this little Mode thing hardly anyone ever uses. Conveniently, the data set is the same (tiny, odd data sets are fun!): 41, 44, 46, 48, 49, 50, 50. The mode is simply the number or numbers that appear the most often so the mode is 50.
The mean (average), the median and the mode – the three sisters of Statistics. And what they tell us is nothing. The Fall of Kabul and the heart wrenching pleas on social media occur within the time span of these polls. USA Today gets a radically different value from Politico and the Hill – yet they are the closest in timeframe to USA Today. But the closest in value is Rasmussen which is done later and Reuters which is done over a shorter timespan. The spread of the seven polls in this average is nine points – that’s pathetic.
So take a bunch of polls done in slightly different timeframes and in different time spans (from 2 to 3 days), take the average and tell people that is represents presidential approval ratings for a span of 11 days. Never mind they have a wild spread (seriously, if they are measuring the same thing, how can they be getting such different results?) and aren’t even done simultaneously – what the hey, toss in the lemon and let’s call it lunch!
Makes about as much sense as the grape, zucchini, kumquat, bell pepper, apple and lemon fruit salad – y’all can have my share, thanks.
