BUT HARDLY ANYONE UNDERSTANDS WHAT THEY SAY

Public opinion polls are not infallible. At best they give a snapshot of what people are thinking at the moment the polling agency spoke to them. At worst, the snapshot is poorly focussed and the results unreliable. Polling results are presented as being accurate within a certain range (±1.5% is about as good as it gets), 19 times our of 20. It is always possible that this poll is the one time out of twenty the sample was not representative and therefore the results are not to be trusted.

A provincial election took place here in Saskatchewan on Monday. A month ago, shortly after the election was called, a public opinion poll reported that support for the incumbent party was 27% higher than support for the main opposition party. Shortly before the election two other polls showed a difference of 18%, leading the opposition party to rejoice that it was on the verge of major gains. Now that the votes are in and counted we see that the incumbent party received 31% more votes than the opposition party (62% for the party in power, 31% for the main opposition party, the remainder split between an assortment of small parties).

What happened? I can’t answer for this particular case, but a lot of things can go wrong in gathering and interpreting statistics. For the results to be trustworthy one needs a sample that is representative and random and questions carefully designed to obtain a clear answer. The results need to be intelligently explained, something most media outlets don’t have the expertise to do.

There is a group of major polling companies in Canada whose results are generally reliable. They are Ipsos-Reid, Léger, Environics, EKOS and perhaps one or two others. The first Saskatchewan poll was produced by one of these companies. The two later polls were produced by smaller companies that do not have a history of producing remarkably reliable results. The stated confidence levels for these polls was ±3.9% and ±4.4%. Even with that generous margin of error, their predictions were far off the mark.

During a Saskatchewan election campaign 20 years ago, results of an opinion poll weree widely reported in the media shortly before the election. It was a poorly executed and poorly interpreted poll, but I believe it influenced the election results. One major flaw was that the results showed something like 35% of voters were undecided. This was compounded when those interpreting the results ignored that number, assuming that those people weren’t going to vote. This resulted in a definite edge for one party. But when you subtract a vary large group of respondents from the results, the margin of error balloons from ±1.5% to some stratospheric number.

That poll was an amateur effort by a small consulting company and should never have been published. In publishing those results the newspapers were showing that either they didn’t have a clue what they were doing, or they were deliberately interpreting the results to favour one party. Take your choice, I don’t know which it was.