Charles Clarke has said he does "not agree" with Tony Blair's decision to sack him as home secretary.
Mr Clarke told the BBC he could have "carried through" the reforms needed to the Home Office following the furore over foreign criminals in the UK.
He said he had been offered "serious" alternative jobs within government by Mr Blair but had turned them down.
The prime minister had to make "hard judgements", Mr Clarke said, adding that he remained a supporter.
John Reid, who takes over as home secretary, remained "a good friend and an outstanding political colleague", Mr Clarke said in an interview with BBC political editor Nick Robinson.
He said: "I have been sacked from this job as home secretary, not sacked from government as a whole, but sacked from this job as home secretary.
"And I regret that because I think I could have carried it through but the prime minister not only has the right he has the duty to make those kinds of judgements.
"They're very hard judgements to make and that's the judgement he has made."
Mr Clarke, who replaced David Blunkett as home secretary in 2004, returns to the Labour back benches.
Shami Chakrabarti, director of the human rights group Liberty, said: "Mr Clarke may feel harshly judged today but for his anti-free speech and ID card laws and for instituting punishment without trial, our children may judge him even more harshly tomorrow.
"His finest moment was hours after the London bombings last July.
"As frightened people hung on his every word he made vital distinctions between distasteful speech and cold-blooded murder and admitted that ID cards would not have prevented the atrocity.
"He forgot party politics and demonstrated what a home secretary could and should be.
"It is a great shame that such promise was never fulfilled."