How Opinion Polling Works

Which of these is most representative of public opinion at large? A heavily gamed voluntary poll of 160,000-odd people done on the behest of This Morning that shows a commanding lead enjoyed by Jeremy Corbyn over Theresa May; or any of the recent spate of polls by professional polling companies who show very much the opposite consistently on the basis of samples between 1,000 and 1,800 people. I have to ask because lots of people have been pushing ITV's poll as more representative than anything YouGov can come up with. After all, it covers more people. The latter? Pah. It was founded by a couple of Tories and provides findings politically convenient for Jeremy Corbyn's opponents. If they were free and fair it would show more support for Labour because I know loads of people who support Labour.

If you happen to share these views, you're wrong. The methodology of opinion polling has been refined over decades of research, and why pollsters and other researchers (including distinctly un-Tory sociologists like me) can make confident generalisations from seemingly small pools of people. This operation has two dimensions to it, but both kinds of test deal with probabilities.

Before anything, we need to start with the ‘null hypothesis’. This is the assumption that when we approach two social phenomena there is no relationship. The maths underpinning statistics are set up to confirm or refute this hypothesis. In the case of polling, tests of statistical significance show the likelihood that claims of no relationship between the cases under study and the results can be rejected. Hence when a poll is compiled, characteristics reflective of the population at large are selected for. For a typical poll, the sample group of, say 1,000 respondents, represents in miniature the population at large or the segment of the population the operation wishes to survey. If we don't do this, then a huge validity question mark hangs over the subsequent claims made. A sample should approximate as much as possible the age, income, gender, ethnicity, etc. profiles of the group or sub-group to be studied. If 30% of the population are over the age of 60, then that should be the case with the sample. If 10% are from a non-white ethnic background, it needs to be reflected. I’m sure you get the picture. Selection then is never completely random but is within the parameters set by the research design. If you’ve never been contacted by a polling company, don’t take it personally!

We have our pool of demographically representative respondents then, but how can we surmise that the views of the sample are equally as representative? This is where tests of statistical significance come in. These are mathematical procedures designed to establish the likelihood that observed characteristics – in this case political opinions – are random (i.e. the null hypothesis is true) or infer a pattern of views that really exist “out there” in wider society. All surveys compute statistical significance tests, which you can usually find by burrowing into the data sets polling companies release along with their results. These tests ask a simple question: if a hundred representative samples were taken, what number of the observed results could be put down to chance alone? If the computed figure returns 0.6, then 60% of cases can be put down to randomness, for example. If it’s 0.05, then five per cent of the sample cases are likely to be random, and so on. The lower the level of significance, the more confident researchers can be that observed data reflects real proportions existing in real populations. When it comes to statements about samples, researchers typically use either 0.05 or 0.01 depending on sample sizes (large for the latter, small for the former). i.e. We are 95% or 99% certain that observed patterns really do exist and are not an artefact of the maths.

This isn’t the only test of statistical significance available. Instead, one can produce an ‘interval estimate’ which, instead of identifying the probability of sample patterns mirroring those of general patterns, looks at errors in sampling. For instance, if 48% of our sample say they’re going to vote Conservative, and such polls have done the rounds recently, how close to the real figure is this finding? This can be inferred by computing a standard error statistic. This means multiplying the Tory figure (48) by the non-Tory figure (52). This gives us 2,496, which is then divided by the sample size. Assuming a sample of 1,000, this equals 2.496. We then apply a square root, which gives us 1.578. This is all very well, but why? This standard error can be used to suggest the real number will be circa 1.6% above or below the polling figure. We have already seen that >0.05 (or 95%) is taken as an acceptable level of certainty in our previous significance test providing, of course, the sample is representative. If it is, we can say with 95% confidence that the numbers of people planning to vote Conservative will be 48%, +/- 1.6%. For example, this is why pollsters in the lead up to the first round of the French presidential election found it very difficult to call because the four front runners were, at times, all within the margin of error of one another.

Sometimes pollsters weight their samples in a particular direction. For example, rather than going for an accurate snapshot of the general population, they sometimes ensure older people are over represented and younger people underrepresented because, as we know, the old are much more likely to vote than the young. Likewise, people from low income backgrounds, have lower levels of formal qualifications, and so on might be scaled down for exactly the same reason.

There you have a very basic overview of polling. There are criticisms of significance testing, and in this age of Big Data a growing clamour suggesting that sampling of this sort may have had its day now huge data sets are available (though, it has to be said, most of these are under the lock and key of public bureaucracies and private business). There are specific criticisms one can make of polling companies. YouGov, for example, is reliant on a database of voluntary sign-ups. There are about 800,000 who’ve joined their UK panel, so while they are likely to not reflect the general population the company has enough data about their demographic characteristics and preferences to construct representative samples out of them. However, they have got into murky waters when they’ve tried polling members of organisations. For one, they have no hard data on the characteristics of their wider membership and so have difficulties generating representative samples. And also, they sometimes have very low numbers of people belonging to certain organisations. I can remember them conducting a poll on Jeremy Corbyn’s support among trade union panel members, and arrived at the CWU’s result after asking just 50-odd people. The union has around 190,000 dues payers.

As a mathematical discipline, statistics have two centuries of scholarship behind it. Polling might get it wrong occasionally, but again that's because it deals with probabilities. Researchers and pollsters can learn from these mistakes, methods can be refined, techniques can be calibrated, improved. Unfortunately, rejection of polling because a leading firm is owned by Tories, because they are used for self-serving political reasons, and because they show Labour plumbing the depths doesn't mean they're wrong. To pretend they have to be because they contradict your experience and views is naive cynicism. The problem is this gets us nowhere. Clinging to illusions is only setting yourself up for a fall when reality crashes in.

If we want to change the world, we have to ask questions, analyse, think, and explain. If things aren't going our way, why? And on that basis, what are we going to do about it? That's the route to making things better because it's the only way.