• News Feeds
Page last updated at 08:30 GMT, Thursday, 18 November 2010
BBC Weather Test: Setting course

Clouds over Somerset

By Roger Harrabin
BBC Environment Analyst

Tim Palmer, Royal Meteorological Society president, shook his head and lowered his eyes to the brown marble floor of the Royal Society entrance hall.

"I'm not sure it can be done," he said. "Not at all sure."

This was an unwelcome opinion about the BBC Weather Test we're organising to see which forecasters the public can most trust.

BBC WEATHER PROJECT
Who can we trust to forecast the weather? BBC News is running a project to study the weather predictions of a number of forecasters who use different forecasting methods to try find out which is most accurate.

Professor Palmer, one of Europe's leading meteorologists, was offering his view before a meeting of the steering group we've formed to set the rules for testing weather forecasters. It includes the Royal Meterological Society, the Royal Statistical Society and the Royal Astronomical Society.

But the steering meeting went rather well. And the public meeting which followed it was one of the most constructive open meetings I have ever attended.

At the end the professor was looking less lugubrious. "It could have gone worse," he admitted.

Indeed, it could have gone much worse, although the possibility of failure of the project overall still exists.

As Professor Palmer told the open meeting: "It seems a very reasonable question to ask us to compare weather forecasters. But sometimes reasonable questions can't be reasonably answered."

Forecast Officer George Cowling of the Air Ministry Meteorological Office, 1954
Weather forecasting has moved on a great deal in the modern age

And indeed, we had spent most of the afternoon trying to thrash out preliminary agreement on the parameters of what you might have thought would be a simple issue. First there was a long debate about what sort of weather forecasts we were able to test.

Daily weather forecasts rely on human interpretation of computer analysis of Newton's laws of thermodynamics and are very reliable over the short term (we'll discuss later exactly how reliable because some people might want to argue).

Seasonal forecasts, on the other hand, rely on studying the great ocean currents, particularly El Nino and La Nina. The UK is the other side of the world from these currents so they affect our weather only faintly and sometimes it is hard to discern the Pacific signal in our weather from the general noise of the semi-chaotic weather system.

My preference was to test everything from daily to seasonal forecasting because that's what the public are exposed to. But that raised many questions.

First, for daily forecasts, would the forecasters refuse to co-operate with supplying all the data needed for daily forecasts in multiple sites? And would Leeds University, who will crunch the numbers for us, be able to cope with a daily forecast over three years from maybe 10 different providers?

Then, what sites should be chosen for daily forecasts and how many of them? Eventually the steering group plumped for simplicity - the weather station at Heathrow Airport is the most prominent site in the UK because it's used for gauging the spot price of the gas market.

Winter weather in London
Being properly prepared is essential if bad weather is on the way

We would attempt to maximise co-operation from providers by asking them to present information on just Heathrow (a decision that was later savaged in the open meeting).

Then what forecast information should our contestants present? Should it be daily maximum and minimum temperatures? If so, over what 24-hour period? Ending at midnight doesn't make sense because that cuts the night in half.

What about rainfall? Is it adequate to forecast total rainfall throughout the day? Or should the forecast demand specifics on when exactly it would rain and how much?

What about wind speed? There was a consensus that wind speed is measured adequately across many sites? What about sunshine hours? This was controversial because when weather forecasters measure sunshine they are actually measuring radiation - and the public might perceive that very differently.

And the forecasts - should they be deterministic - "It's going to be hot tomorrow". Or should they be probabilistic - "There's a 70% chance of a hotter than average October day tomorrow."

Family erecting a windbreak on a windy Margate beach
Bad weather forcasts won't stop everyone going to the beach

The BBC's forecasters offer deterministic forecasts because the BBC does not believe the public is ready for percentage likelihoods on getting drenched. But round our committee table there was a conviction that probabilistic forecasts were a proper test of skill.

Here's how the probabilistic forecast test would work for temperature, for instance: forecasters would be given 100 notional forecasting "tokens" for every forecast.

They would divide those tokens into three "bins" labelled (notionally of course) below average, average and above average. That allows a certain amount of hedging of bets because sometimes the weather is very easy to forecast - in a spell of high pressure for instance - and sometimes it is very hard. If a forecaster really doesn't feel confident to make a prediction on the next day he has to spread his tokens evenly across all three bins.

It was agreed that a forecaster could offer a deterministic forecast if he or she wanted - but this strategy would risk winning or losing everything by dumping all the tokens into one bin.

Then there's the question of who would provide the verification data for the forecasts.

The independent Philip Eden had volunteered, and I liked this because Philip is regarded as genuinely independent.

But then others objected that the Met Office is the official provider of World Meteorological Organisation statistics and it would be ludicrous to reject their highly sophisticated statistics smoothed by computer models just because the Met Office forecasters were amongst the contestants in the Weather Test.

This question was left unresolved, with the two journalists present fearing the possibility of a cynical public reaction to the Met Office being considered judge and jury and the scientists considering this fear rather silly.

Then it was on to the monthly forecasts. It was generally agreed that these are of relatively limited value (though the public isn't told this). Monthly forecasts use a combination of the methods of daily and seasonal forecasting.

It wasn't reasonable to ask forecasters to give a regional forecast - only a UK forecast. A Northern Europe forecast would be more fair. And when it comes down to seasonal forecasting … well that is even more of a black art, it seems. The Met Office have stopped publicising their seasonal forecasts after taking too much stick over their blundering barbecue summer forecast.

No matter that the barbecue summer slogan was coined by the Met Office press office and the forecasters had made a more sensible probabilistic forecast. Seasonal was considered to be such an experimental area that it was agreed that no-one would want to be judged in that category.

And so it was left…. until the public meeting later that day.


If you want to get involved, get in touch using the form below.

Name
Your E-mail address
Town & Country
Phone number (optional):
Comments

The BBC may edit your comments and not all emails will be published. Your comments may be published on any BBC media worldwide.





FEATURES AND COMMENT
Ajibola Lewis (right) with her daughter Police custody 'scandal'
A charity calls for a public inquiry into the number of people who die while being held by police.

Christmas tree Mass Observing the season
The spirits of Christmases past, as seen by the British people

Children selling low-value goods at the roadside are a familiar sight in Liberia Catch-22
Evan Davis examines Liberia's attempt to rebuild its economy following the recent civil war.

AUDIO SLIDESHOWS
RECENT INTERVIEWS

SEE ALSO
BBC Weather Test: Starting out
Friday, 24 September 2010, 06:20 GMT |  Today

BBC navigation

BBC © 2013 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.

Americas Africa Europe Middle East South Asia Asia Pacific