Those Opinion Polls: A Reality Check

This is a guest post from Stephen Duncan. I don’t usually publish polls discussion on this site as James Kelly of Scot goes Pop usually has this covered but Stephen has put a lot of work into this and is clearly less excited than most pro Indy Commentators. Here he explains why.

Those Opinion Polls: A Reality Check

People seem to be working themselves into a lather of excitement over the recent spate of opinion polls on Scottish Independence, if the comments by regular columnists in, and myriad readers on, various articles in The National are anything to go by (see, for example, Scottish independence support at 56%, Ipsos Mori poll finds | The National). Whilst there have now been 4 consecutive polls showing a lead for YES, and 3 of them since the UK Supreme Court ruling on 23^rd November that the Scottish Parliament did not have the right to legislate for an Independence referendum, it is wise to bear in mind that these results form a relatively small proportion of a much longer time-series recorded since the Independence referendum of 18^th September 2014.

Polling Analysis

Statistics and numbers can be turn off for many but they are used to justify arguments and defend positions so it’s important to have as a full an understanding as possible of results published so that ‘the narrative’ and claims being made can be properly scrutinised and assessed. So in the interests of perspective, context and, indeed, accuracy I have looked at this in some detail and in different ways. Here’s what I found:

Batch analysis of survey results registered post 2014 using a simple average across all pollsters:

This approach combines tranches of survey results from different pollsters and calculates the arithmetic average. The drawback to this is that averaging across polling firms can mask CHANGE in support picked up by each organisation due to inherent differences in methodologies employed by the different pollsters. On the other hand the statistical biases in the LEVEL of support recorded may be balanced out in the averaging process as long as no single firm dominates the series of surveys.

Conclusion:

There is no discernible change in rounded terms over the longer 12 month comparison where the YES/NO split is roughly 50:50. However, an increase in YES support is depicted in cohorts comprising fewer poll results with the earliest to most recent 3 survey comparison depicting an increase from 50% to 54%. This is clearly as a consequence of the most recently published surveys which been favourable to YES. This is also shown when a comparison is made between the 12, 9, 6 and 3 most recent polls with YES increasing from 50% to 54%, the most recent 3 surveys accounting for half of that rise.

Batch analysis of survey results registered post 2014 using a weighted by sample volume average across all pollsters:

This differs from the first calculation in that all percentages are weighted by the (different) sample sizes involved in the various surveys and is technically a more accurate measure. However, the results are pretty much identical due to sample sizes being fairly standard – it’s usually just over 1000, yielding the normal +/- 3% margin error – which has the effect of rendering the weighting by volume effect fairly neutral.

Conclusion:

Similar to the unweighted assessment with a small and fairly insubstantial +1% uplift over the earliest to most recent comparisons containing 12 polls and a +1% rising to +4% increase for the cohorts comprising smaller number of poll results.

[Note: In all calculations, and as standard, I have excluded those that Don’t Know, Won’t Say and Won’t Vote).

One of the reasons why it is problematic to infer at this stage a ‘step increase’ in YES support based on these most recent polls is that two of the three recent surveys were conducted by polling firms with limited experience in conducting polls on support for Scottish Independence. Both Redfield & Wilton Strategies and Find Out Now who published results after the UK Supreme Court ruling recording YES at 52% and 54%, respectively, have only ever published one other poll on the subject. This means we have very limited data points for these firms with which to establish a benchmark and from which to discern any trend.

However, this is not the case for Ipsos who conducted the recent poll showing YES on 56% – and have carried out 17 surveys since the referendum – nor for other major and experienced polling firms Panelbase, Survation and YouGov who have published 50, 45 and 39 polls, respectively. These are now considered.

Looking at First 12 surveys post referendum versus most recent 12 and computing a weighted by sample volume average for each of the 4 firms that have polled across the period since September 2014:

Analysing by polling organisation has the advantage of comparing ‘apples with apples’ although differences  in methodologies can yield different levels of support from firm to firm in the results. However, it is probably the most accurate way, in my opinion, to gauge change in support across the time period.

Conclusion: 

Three out of the four polling firms show some limited increase which, taken in the round, could point to an uptick of around +1%. (Taking an arithmetic average of these is slightly distorting but given that sample sizes are similar in the polls this should only be in the neighbourhood of percentage decimal points).

The problem here is that neither Panelbase, Survation nor YouGov have posted any poll results since the UK Supreme Court pronouncement so no inference can be drawn as to what effect the ruling may have had on support for Independence.

Assessing YES support over the long term by each of the 4 firms that have polled across the period:

On this basis three of the pollsters depict a reduction whilst one indicates that support has flat-lined. The drawback to this method is that the earliest result is based on a single poll while the latter is an average across the number of surveys conducted in the series per polling firm. (For example, Ipsos recorded 54.92% in its first post-referendum survey but the weighted by sample size average across 17 polls since 18th September 2014 is 52.00%). However, this does serve to highlight the danger of using a single snapshot as being the true measure of YES support. 

There have, of course, been significant fluctuations at different points across the period and these, together with the underlying long term level of YES are best illustrated graphically:

Conclusion

This points to a reduction in support from the immediate post-referendum boost. It is noted, however, that an uptick in YES can be detected from Ipsos, Survation and YouGov prior to the UKSC ruling perhaps indicative of the cost of living crises positively impacting YES sentiment.

[Note: In all calculations, and as standard, I have excluded those that Don’t Know, Won’t Say and Won’t Vote).

Further Observations

So, however you look at it, the picture over the longer term is a fairly static one, albeit YES has increased versus the baseline referendum actual result (44.70%). But the increase post-referendum happened immediately after the 18th September 2014, down probably to EVEL being announced the next day and ‘buyers regret’ on behalf of some NO voters.

So for The National, its columnists and commenters, the SNP and pro-SNP bloggers to refer variously to ‘surging’, ‘soaring’, ‘shooting’ and ‘sky-rocketing’ support for Independence is somewhat hyperbolic. And perhaps a bit desperate.

The reason why it’s desperate is that there is a continuing vacuum of leadership in the SNP on the constitutional question facing Scotland.

In addition we have been here before. There have been upward spikes measured at different points in the past, as the time-series for each pollster shows.

Some have said that the UK Supreme Court ruling is a ‘game changer’ and that historical polls are, therefore, not relevant. Of course historical polls do become less relevant over time and as circumstances change. However, you cannot describe something as a game changer on the basis of a 3 data points. You need many more to confirm a step change in the level and trend of support. Remember, even the run of 20 consecutive opinion polls in 2020-21 came to an end with support evaporating as the pandemic eased.

At this stage, therefore, we should be cautious especially when it is recalled that we were told by prominent political figures and The National’s authors that the Brexit vote, then the perceived better handling of the pandemic in Scotland and finally the Brexit implementation were all watersheds and that tidal wave in favour of YES would follow.

That may have happened if the momentum so generated by events had been captured by the SNP and used to drive support with a suitably aggressive and nuanced campaign. It wasn’t.

It remains to be seen if the leadership will ‘catch the wind’ this time or whether any momentum – generated by cost of living crises given added impetus by the UK Supreme Court ruling – will be lost due to prevarication, timidity and unpreparedness on behalf of the movement’s political leadership.

What we do know is that opinion polls are passing.

And Independence is permanent.

Sources

Wikipedia: https://en.wikipedia.org/wiki/Opinion_polling_on_Scottish_independence

Ipsos UK: https://www.ipsos.com/en-uk/news-and-events/news

Panelbase: https://drg.global/our-work/political-polls/

Survation: https://www.survation.com/

YouGov: https://yougov.co.uk/topics/overview