[an error occurred while processing this directive]
BBC News
watch One-Minute World News
Last Updated: Tuesday, 19 October, 2004, 16:38 GMT 17:38 UK
Treat polls cautiously, experts warn
By Richard Allen Greene
BBC News Online

John Kerry won the third and final presidential debate by a convincing margin, bringing him even with George W Bush in the race for the White House, opinion polls suggest.

Woman talking on the telephone (generic)
People increasingly only take the calls they want
But can we trust these polls?

Experts warn that surveys should be taken with a grain of salt - particularly in a race as close as this one.

Michael Traugott, a senior researcher at the University of Michigan's Center for Political Studies, urges poll watchers to remember that surveys have a margin of error.

"A standard pre-election poll is of 700 to 1,000 people, which produces a margin of error of three to four points," he said.

That means the poll can only reliably detect a lead of six to eight points, he said.

Neither Mr Bush nor Mr Kerry has consistently pulled that far ahead in the race, which means all that polls can show is that the race is close - not who is leading.


And even when races are not this tight, changes in technology are raising problems for pollsters who rely on random telephone surveys.

"The big concern is that people have got quite clever at avoiding calls they don't want to take - caller ID, mobile phones, call forwarding," said Arthur Lupia, a professor of political science at the University of Michigan.

"Response rates on phone interviews are dive-bombing. Ten years ago, if you got 70% response, that was not a very good effort. Now people would jump for joy if they got 70%."

Frank Newport, the editor-in-chief of the Gallup Poll conceded the point but said it did not affect the reliability of surveys.

"To date there is no evidence that a lowered response rate is introducing bias," Mr Newport said.

Anna Greenberg of Greenberg Quinlan Rosner Research said pollsters rang numbers repeatedly to ensure they got enough responses.

"Most pollsters call at least three times [to reach a number], and academics and government researchers will call up to 10 times," she said.

"Every person in the population that you care about has to have a probability of being pulled into the survey," she said.

Man talking on a mobile phone (generic)
Pollsters are not allowed to ring mobile phones in the US
Telephone polls generally under-represent some portions of the population, such as lower socio-economic levels, Mr Traugott said.

The rise of mobile phones also raises concerns, since it is illegal in the US to phone mobiles for surveys, he added.

But Ms Greenberg says only 3% of Americans solely use mobiles.

"At the moment people are still pretty linked to their land lines," she said.

Weighting data

Once pollsters have completed their telephone surveys, they weight the data - trying to interpret the responses so they represent the population as a whole.

Each pollster uses different formulas, which are kept secret.

Some people may be calling themselves Republican one week and Democrat the next
Frank Newport, Gallup
Experts agreed that weighting data was an art, not a science.

There is a fierce debate in the field about whether to try to take party identification into account when weighting data.

The Gallup Poll does not, its editor Mr Newport said, because party identification "is not stable like race or colour of hair.

"Some people may be calling themselves Republican one week and Democrat the next," he said.

The Zogby polling agency, on the other hand, does try to control for party identification - and was one of very few to predict correctly that Al Gore would win the popular vote in 2000.

The head of Zogby International, John Zogby was unavailable for comment.

Polling experts say Gallup Poll results tend to lean Republican, while Zogby results lean Democrat.

Anna Greenberg said both approaches had merit.

"Gallup is right that party identification is an attitude, but Zogby is also right in that preference is stable over time. The real answer is somewhere in the middle."

Who votes?

Ms Greenberg raised another potential source of error: Pollsters tend to assume that voters in each election are like the ones that came before.

The danger is that when newspapers and television sponsor polls, they have to write a story based on the poll
Anna Greenberg
But there is controversy over how to identify likely voters.

Gallup, for example, asks respondents to answer seven questions, including whether they typically vote and if they voted in the last election.

But experts caution that trying too hard to identify likely voters could mean excluding first-time voters.

"If you are too strict about who you let into your survey, you miss out on some voters," Ms Greenberg said.

That could skew results in a year - like this one - when both parties are engaged in massive voter-registration drives.

While the experts admit that polling is an imperfect science, they also suggest the media may make surveys look less reliable than they are.

"The danger is that when newspapers and television sponsor polls, they have to write a story based on the poll," Ms Greenberg said.

All the experts agreed it was safer to rely on a series of polls than on any individual headline result.

"There is no question that polls are fairly reliable in interpreting changes in public opinion over time," said Vince Price of the Annenberg School for Communication at the University of Pennsylvania.

"Given the wide number of polls that are conducted, reading trends across polls is likely to lead to a conclusion. Some people complain about the large number of polls, but there is actually an advantage there."

Of course, as the politicians say, ultimately there is only one poll that matters. It takes place on Election Day.

The BBC is not responsible for the content of external internet sites


News Front Page | Africa | Americas | Asia-Pacific | Europe | Middle East | South Asia
UK | Business | Entertainment | Science/Nature | Technology | Health
Have Your Say | In Pictures | Week at a Glance | Country Profiles | In Depth | Programmes
Americas Africa Europe Middle East South Asia Asia Pacific