Methodology

Polling

Ensuring that polls are representative raises a number of difficult issues. Some of the major issues are discussed here. Populus champions openness and transparency in polling and so is always willing to discuss its polling methodologies.

Omnibus polls

Omnibus polls are conducted among the general public on a regular basis. The Populus Telephone Omnibus runs twice a week and the Online survey weekly. Omnibus polls combine questions on different topics for different clients. The costs of the poll are shared across the clients who share the demographic data but only receive the polling data for the questions they pay for.

Telephone Polling

Everyone in the country with a landline telephone has an equal random chance of being polled in a Populus telephone survey. Within each region a random sample of telephone numbers is drawn from the entire BT database of domestic telephone numbers. Each number selected has its last digit randomised so as to provide a sample including both listed and unlisted numbers.

Using this technique – known as random probability sampling – gives us a robust statistical basis for national representative polls, accurate to a small margin of error.

Face-to-Face Polling

Face-to-face polling is rarely used for large-scale polls as nationally representative samples can be achieved much more reliably via the telephone or internet. However, it is useful for projects such as surveys of children (which are carried out in strict accordance with the Market Research Society’s Code of Practice), or questionnaires involving sensitive or personal issues, allowing respondents to fill in a questionnaire anonymously before putting it in an envelope and handing it back to the interviewer. Populus uses a national network of trained interviewers, enabling us to ensure quality control wherever surveys are conducted throughout Britain.

Internet Panel Polling

More and more research is being undertaken using the internet. Populus has access to a number of very large panels of the general public; both in the UK and around the world. We also maintain and build specialist panels.

Polling on the internet raises particular challenges since internet penetration is uneven across different demographic groups, making it hard to reach some socio-economic types. Populus will always advise clients whether it is appropriate to conduct research online.

Ensuring polls are demographically representative

Data from national polls are weighted to the profile of all adults aged 18+ (including non telephone owning households). This is the accepted best practice for making sure that polls are as representative as possible.

At its most basic level, data weighting compensates for unrepresentative data gathering. So, as an example, 51% of the UK population are female. If only 40 people out of 100 polled are female this doesn’t accurately reflect the population – it is “unrepresentative”. To overcome this, the answers of the female respondents would be given slightly more “weight” so that they have a representative impact on the final findings.

Populus polls are checked against a range of demographic criteria to make sure they are representative and where necessary data are weighted by sex, age, social class, household tenure, work status, number of cars in the household and whether or not respondent has taken a foreign holiday in the last 3 years. Targets for the weighted data are derived from the National Readership survey, a random probability survey comprising 34,000 random face-to-face interviews conducted annually.

Ensuring polls are politically representative

Following the 1992 General Election, when published opinion polls were wrong by an average of 7.5% (much more than their theoretical margin of error) – and all erred in the same direction, when the statistics of opinion polling mean that polls should scatter either side of a mean -, an enquiry by the Market Research Society concluded that a sample that was demographically representation of the population could no longer be assumed to be politically representative. In response some polling organisations added a measure of political representativeness to the battery of demographic data by which polls are weighted. The most commonly used gauge of how politically representative a sample is, is how that sample remembers voting at the last general election. The recalled past votes of each sample is weighted so that each party’s share is close to the actual result of the last election (but not exactly the same because some people misremember or misstate who they voted for and the weighting system used takes account of this).

The companies which introduced this change have been proven much more accurate at subsequent elections. Populus, learning from this experience, applied similar methods when starting polling in 2003. At the 2005 General Election Populus’s final poll of the campaign was virtually spot-on, putting Labour on 36% (exactly what they got), Conservatives on 32% (1% below their actual share) and the Lib Dems on 23% (exactly what they got). To view the tables for this poll, click here.

Populus derives voting intention from 4 questions. First of all respondents are asked how likely it is that they will actually vote at the next general election. Those who say they will vote are asked to say which party they would support if there were a general tomorrow. Next respondents are asked whether they voted at the last general election and, if so, for which party.

The voting intention figures that are published each month in The Times are calculated after Populus has excluded those who say they will not vote, who refuse to answer the question or who don’t know which party they would vote for. The figures are then adjusted for turnout on the basis of respondents’ declared likelihood of voting (based on where on a ten point scale of likelihood to vote each person interviewed has placed themselves).

Populus then weights the whole sample on the basis of its ‘past vote’ - adding the most recent poll data to its previous 10 most recent polls (so as to avoid the random volatility that can appear in comparing any two individual samples) and calculates the past vote weighting from the average recalled past vote in this data.

An additional step is then taken to address the tendency for ‘spirals of silence’ among supporters of unpopular parties causing an inadvertent bias in voting polls.

Spirals of silence

A ‘spiral of silence’ is the tendency of some voters not to want to reveal a party preference they perceive to be unpopular or unfashionable. In past elections failure to take account of this phenomenon has been a significant factor in polling error, especially so in 1992. When spirals of silence are present disproportionate numbers of former voters for one or other party will say that they don’t know how they are going to vote next time. The methodological response to this phenomenon is therefore to put some of these don’t knows back into the column of the party they voted for at the previous election.

Following the 2005 General Election, Populus drew data from a very large poll (5,000 + sample) conducted over the three days immediately after the election. This data – confirming previous callback polls and academic studies – found that there was a probability of about 50% that someone who voted Labour or Conservative in 2001 but then said they didn’t know how they would vote in future ending up voting the same way again; the equivalent percentage for Liberal Democrat voters was around 30%. Based on this evidence, Populus applies a spiral of silence weighting to its political polls by taking all those who will say how they voted at the last election but not how they’ll vote at the next one and adding them onto the current support for each party at a depreciated value: 0.5 for past Labour and Conservative voters and 0.3 for past Lib Dem voters.

.

Back to
Methodology

Contact me about
this item