"This was not a cash problem in the hospital, it was not a capacity problem in the hospital, it was a management problem in the hospital.
Speaking in the Commons, Mr Milburn rejected accusations that the incident was an inevitable outcome of government policy.
"Patients and their families have a right to be treated with dignity and respect at all times in all parts of the NHS. In this case they were not."
Mr Milburn said an initial investigation had been carried out - and found that a failure of management was to blame.
There is to be a full investigation, which will be made public.
Mr Milburn said guidelines issued by the government twice last year, which say temporary mortuary facilities must meet minimum standards to respect patients' dignity, had not been followed by the hospital.
He added that an extra mortuary, built at a cost of £20,000 was at no stage full. A minor fault with the doors of the mortuary had been reported, but not tackled for several days.
"This was not a cash problem in the hospital, it was not a capacity problem in the hospital, it was a management problem in the hospital.
"The issue should, and could have been sorted out on the spot. Instead deceased patients were denied their dignity."
The Health Secretary said people who said what happened at Bedford was a reflection of what happens in other NHS hospitals were "wrong".
His comments follow the revelation this week that up to eight bodies were stored on the floor of the chapel of rest, wrapped only in sheets.
After a public outcry, the chief executive of the hospital, Ken Williams resigned.
'Scapegoat
But consultants have said Mr Williams was made a scapegoat for what they described as "the chronic under funding of the pathology services".
"We are concerned that any pressure is being put on him because at the end of the day he is not really responsible for the under funding that has led to this situation," he said.
"Although I hear the regional chief executive say that it is unacceptable I am not entirely clear what alternative Bedford Hospital, or indeed many other hospitals might have when they are faced with an overflow situation."
Mr Rawlins said hospital managers were being "scapegoated" out of "political expediency".
He said managers had been forced to spend available cash on cutting waiting lists - rather than in areas in which they wanted to invest.
"We see pathology as a priority but Mr Milburn [the Health Secretary] would rather it was spent on treating non-urgent cases within 18 months.
Shadow health secretary Dr Liam Fox said the blame lay with Mr Milburn.
"This horrendous episode has blown the lid on the government's incestuous and corrupting running of the NHS and this is one occasion where the Secretary of State cannot blame the previous government or funding, nor can he blame management.
"They are his priorities, by his people, on his watch - and it's his responsibility."
Dr Fox said an independent inquiry should be carried out.
The Health and Safety Executive is to investigate the hospital next week.
This will be followed up by a visit from the Commission for Health Improvement.
Jeremy Saunders, medical director at Bedford Hospital, told the BBC: "Under-funding is at the root of the problem.
"The department of histopathology is too small, it is underfunded, and the mortuary is far too small for the needs of the local area".