It is "grotesque" that outgoing ministers can take severance pay even if they return to government months later, says a Lib Dem frontbencher.
Mr Baker says the payments are abuse of taxpayers' money
Norman Baker says the payments are an abuse of taxpayers' money and ex-ministers given new jobs should at least have to repay the cash.
Cabinet ministers are entitled to one quarter of their £74,902 salaries when they leave office.
The system began in 1991 and is to prevent sudden drops in income.
All severance payments below £30,000 are tax-free.
Mr Baker pointed to how David Blunkett had resigned as home secretary last December but rejoined the Cabinet after last month's election.
Mr Blunkett would have been entitled to about £18,000 in severance pay.
Other ministers have also left office only to return to a different job.
Alan Milburn was entitled to a pay-off both when he quit as health secretary in 2003 and, after returning to office last September, when he left again last month.
Among lower-paid ministers outside the Cabinet, Beverley Hughes resigned as immigration minister last April but was appointed children's minister last month.
The rules were introduced when the Conservatives were in government and officials stress there has been no change in the way they are applied.
Outgoing ministers are not entitled to the severance pay if they take up a paid government job within three weeks or if they are retiring at age 65.
Mr Baker said the rules should be changed.
He said: "It is dubious at the best of times that ministers should get a pay-out when they leave office... It is particularly objectionable if they are re-employed in similar jobs."
He said the payments should end or ex-ministers should have to pay them back if they re-enter government.
Alternatively, there could be a year's gap before they received severance pay.
A Cabinet Office spokesman said he could not comment on individual cases or say whether specific ministers had taken the severance pay.
He pointed to how the Top Salaries Review Board had in 1988 recommended severance pay to take account of how ministers could face an immediate drop in income.
"It takes time for former ministers to pick up the threads of their previous career," said the spokesman.
A spokesman for Mr Blunkett said: "These arrangements were introduced under the last Conservative government and recognise the need for a modest period of readjustment.
"That is why the Labour government introduced the redundancy payment scheme 30 years ago, which has protected the interests of millions of workers since that time.
"I presume Norman Baker will now pledge to forego his severance pay should he resign or lose his seat at the next general election."
Other paid public office-holders, such as the House of Commons speaker or the leader of the Opposition, are also entitled to severance pay.