BBC NEWS Americas Africa Europe Middle East South Asia Asia Pacific

BBC News World Edition
 You are in: Health  
News Front Page
Middle East
South Asia
Medical notes
Talking Point
Country Profiles
In Depth
BBC Sport
BBC Weather
Number-crunching medicine
Statistics have been tough to compile
The problems faced by experts trying to unravel the facts at Bristol show starkly how hard it is to check the performance of doctors.

Child heart surgery is one area of medicine in which the measures of success and failure are more obvious than most.

It comes down to the number of operations, versus the number of deaths - divide one into another and you are fairly close to a success rate for any particular surgeon.

However, the number-crunchers trying to prove whether Bristol's results were good, bad, or indifferent have struggled mightily.

Wisheart operating
When do the figures suggest he should have stopped?
And it means that, unless a mammoth effort is made in the coming years, giving patients an idea of how good their doctor compared to the best of the rest could be a nigh-on impossible task.

Statistics expert Dr Jan Poleniecki from St George's Hospital in south London feels that until now, attempts to compare the performance of doctors have failed.

He also thinks that the statistical evidence used to "convict" Bristol and its surgeons is deeply flawed.

The initial figures compiled by anaesthetist Steven Bolsin certainly screamed out a problem at Bristol.

'Appalling' data

His analysis of death rates, even when corrected of one major calculating error, showed death rates far in excess of the numbers said to be the "average" at that time.

And it is the word "average" which is problematic.

The quality of data is appalling

Dr Jan Poloniecki, statistician
Dr Poloniecki told BBC News Online: "The quality of data is appalling and one of the main problems is the centralised database.

"It is desperately difficult to produce meaningful figures, even for heart surgery."

A large group of statistical experts have used a variety of techniques to try and obtain a more accurate measure of how Bristol surgery compared with the rest of the country during the 80s and 90s.

In the mid 1980s, heart surgery was streets ahead of other specialty in compiling and issuing national mortality figures for each unit in the country.

A look at this register suggests a trend - of other units getting better at operations, and losing fewer patients, with Bristol's overall mortality in babies under one year old staying roughly the same.

Between 1984 and 1987 a quarter of these tiny patients undergoing surgery at Bristol did not survive. Between 1988 and 1990 it was up to 29%, but in 1991 to 1995 - really the critical period for the scandal - it was back down to below a quarter.

In the early 80s, Bristol's performance is reasonable compared to other centres. At least three appear to be worse performers.

One, Birmingham, appears to have an amazing 39% overall mortality.

By the late 80s, Bristol is edging towards the outer margin - there is only one centre that is worse overall.

And during the early 90s, the statistics suggest that Bristol is the poorest performing unit by a clear margin.

But even this is not necessarily enough to pass a clear-cut verdict on the doctors and managers running the service.

If you have 13 centres, someone has to be first, and someone last, and if there is an average, it stands to reason that some centres will fall below that mark.

How bad must it be?

How big does the discrepancy between Bristol and next worst unit have to be before it can be damned?

After months of work, the statistical experts employed by the inquiry believe that there is enough information around to draw this conclusion.

They conclude that, between 1991 to 1995, between 30 and 35 too many children died at Bristol. This, they said, meant that Bristol had roughly double the "average" mortality of the rest of the UK.

Inquiry chairman Professor Ian Kennedy has statistical advisers
They suggested it would be possible for doctors pick up the divergence from "average" standards by 1990, with simple analysis, but conceded that overall data quality was poor.

In future, say the statisticians, the data quality must improve so that minor problems - and not just major crises such as this, can be picked up.

Some of those doctors closely involved with the Bristol unit are adamant that the cardiothoracic society's register - the principle source of information about heart surgery in the rest of the UK - is not robust enough to use to make career-affecting decisions about medics.

Among other flaws, they say, many units did not report their results accurately to the register, skewing the results. In addition, they say, the register took no account of particularly complex and difficult cases needing the same operation.

The Bristol surgeons both claimed they had far higher than normal numbers of these "high risk" operations.

Dr Poloniecki agrees to some extent - saying that both were later told by the General Medical Council that they should have stopped operating - even though the GMC had never issued firm criteria suggesting what was acceptable and what unacceptable.

He said: "Dhasmana has certainly suffered a very severe penalty - because I simply don't see by what criteria he is supposed to have made that decision."

Spreading the blame

The inquiry's statistical experts, however, rejected all this, saying that despite these acknowledged weaknesses, the information available was enough to at least say there was a significant problem at Bristol.

What those figures fail to do, however, is apportion blame to individual doctors.

The inquiry did try to use a small sample of cases to look for patterns of weakness in any particular member of staff's work.

Any they found that, while many children appeared to have in retrospect received substandard care at the hospital, responsibility for this was spread fairly evenly around.

In some cases, the poor care happened during the operation itself, while in others it happened pre-operatively, or post-operatively.

The analysis of Bristol's problems has taken many months, many more man-hours and huge amounts of money.

Even after this, because of the poverty of data which could be used as a comparison, only the unit as a whole can firmly be said to be operating significantly below par.

The problem facing the medical profession - which itself faces an increasingly demanding public - is how both to measure the performance of individual surgeons and other doctors, and how to present those to questioning parents.

Less money

They will have much less time and money available to compile their own figures.

And although there will always be a centre with the lowest mortality in the land, only a limited number of children can go there.

Parents will have to accept second, third, or fourth best for their child - or be even further down the league table.

At what point does the tail-end of the league table legitimately become a no-go area?

Full coverage of the Bristol heart babies inquiry report

Government response

Key stories

Key figures

Parents' stories

Background briefing


Bristol year by year
Internet links:

The BBC is not responsible for the content of external internet sites

Links to more Health stories are at the foot of the page.

E-mail this story to a friend

Links to more Health stories

© BBC ^^ Back to top

News Front Page | Africa | Americas | Asia-Pacific | Europe | Middle East |
South Asia | UK | Business | Entertainment | Science/Nature |
Technology | Health | Talking Point | Country Profiles | In Depth |