By Roger Harrabin
BBC Environment Analyst
Tim Palmer, Royal Meteorological Society president, shook his head and lowered his eyes to the brown marble floor of the Royal Society entrance hall.
"I'm not sure it can be done," he said. "Not at all sure."
This was an unwelcome opinion about the BBC Weather Test we're organising to see which forecasters the public can most trust.
BBC WEATHER PROJECT
Who can we trust to forecast the weather? BBC News is running a project to study the weather predictions of a number of forecasters who use different forecasting methods to try find out which is most accurate.
Professor Palmer, one of Europe's leading meteorologists, was offering his view before a meeting of the steering group we've formed to set the rules for testing weather forecasters. It includes the Royal Meterological Society, the Royal Statistical Society and the Royal Astronomical Society.
But the steering meeting went rather well. And the public meeting which followed it was one of the most constructive open meetings I have ever attended.
At the end the professor was looking less lugubrious. "It could have gone worse," he admitted.
Indeed, it could have gone much worse, although the possibility of failure of the project overall still exists.
As Professor Palmer told the open meeting: "It seems a very reasonable question to ask us to compare weather forecasters. But sometimes reasonable questions can't be reasonably answered."
Weather forecasting has moved on a great deal in the modern age
And indeed, we had spent most of the afternoon trying to thrash out preliminary agreement on the parameters of what you might have thought would be a simple issue. First there was a long debate about what sort of weather forecasts we were able to test.
Daily weather forecasts rely on human interpretation of computer analysis of Newton's laws of thermodynamics and are very reliable over the short term (we'll discuss later exactly how reliable because some people might want to argue).
Seasonal forecasts, on the other hand, rely on studying the great ocean currents, particularly El Nino and La Nina. The UK is the other side of the world from these currents so they affect our weather only faintly and sometimes it is hard to discern the Pacific signal in our weather from the general noise of the semi-chaotic weather system.
My preference was to test everything from daily to seasonal forecasting because that's what the public are exposed to. But that raised many questions.
First, for daily forecasts, would the forecasters refuse to co-operate with supplying all the data needed for daily forecasts in multiple sites? And would Leeds University, who will crunch the numbers for us, be able to cope with a daily forecast over three years from maybe 10 different providers?
Then, what sites should be chosen for daily forecasts and how many of them? Eventually the steering group plumped for simplicity - the weather station at Heathrow Airport is the most prominent site in the UK because it's used for gauging the spot price of the gas market.
Being properly prepared is essential if bad weather is on the way
We would attempt to maximise co-operation from providers by asking them to present information on just Heathrow (a decision that was later savaged in the open meeting).
Then what forecast information should our contestants present? Should it be daily maximum and minimum temperatures? If so, over what 24-hour period? Ending at midnight doesn't make sense because that cuts the night in half.
What about rainfall? Is it adequate to forecast total rainfall throughout the day? Or should the forecast demand specifics on when exactly it would rain and how much?
What about wind speed? There was a consensus that wind speed is measured adequately across many sites? What about sunshine hours? This was controversial because when weather forecasters measure sunshine they are actually measuring radiation - and the public might perceive that very differently.
And the forecasts - should they be deterministic - "It's going to be hot tomorrow". Or should they be probabilistic - "There's a 70% chance of a hotter than average October day tomorrow."
Bad weather forcasts won't stop everyone going to the beach
The BBC's forecasters offer deterministic forecasts because the BBC does not believe the public is ready for percentage likelihoods on getting drenched. But round our committee table there was a conviction that probabilistic forecasts were a proper test of skill.
Here's how the probabilistic forecast test would work for temperature, for instance: forecasters would be given 100 notional forecasting "tokens" for every forecast.
They would divide those tokens into three "bins" labelled (notionally of course) below average, average and above average. That allows a certain amount of hedging of bets because sometimes the weather is very easy to forecast - in a spell of high pressure for instance - and sometimes it is very hard. If a forecaster really doesn't feel confident to make a prediction on the next day he has to spread his tokens evenly across all three bins.
It was agreed that a forecaster could offer a deterministic forecast if he or she wanted - but this strategy would risk winning or losing everything by dumping all the tokens into one bin.
Then there's the question of who would provide the verification data for the forecasts.
The independent Philip Eden had volunteered, and I liked this because Philip is regarded as genuinely independent.
But then others objected that the Met Office is the official provider of World Meteorological Organisation statistics and it would be ludicrous to reject their highly sophisticated statistics smoothed by computer models just because the Met Office forecasters were amongst the contestants in the Weather Test.
This question was left unresolved, with the two journalists present fearing the possibility of a cynical public reaction to the Met Office being considered judge and jury and the scientists considering this fear rather silly.
Then it was on to the monthly forecasts. It was generally agreed that these are of relatively limited value (though the public isn't told this). Monthly forecasts use a combination of the methods of daily and seasonal forecasting.
It wasn't reasonable to ask forecasters to give a regional forecast - only a UK forecast. A Northern Europe forecast would be more fair. And when it comes down to seasonal forecasting
well that is even more of a black art, it seems. The Met Office have stopped publicising their seasonal forecasts after taking too much stick over their blundering barbecue summer forecast.
No matter that the barbecue summer slogan was coined by the Met Office press office and the forecasters had made a more sensible probabilistic forecast. Seasonal was considered to be such an experimental area that it was agreed that no-one would want to be judged in that category.
And so it was left
. until the public meeting later that day.
If you want to get involved, get in touch using the form below.
The BBC may edit your comments and not all emails will be published. Your comments may be published on any BBC media worldwide.