Thousands of civil service jobs have already been cut or moved out of London as part of a major cost-cutting drive.
Mr Brown plans to axe 104,000 civil service jobs
Chancellor Gordon Brown said 12,500 jobs had gone while 7,800 were being moved out of the South East.
He plans to axe 104,000 jobs to free up money for education, health, defence, housing and overseas aid.
Unions oppose the plans but Mr Brown said £2bn savings had already been made and more jobs had been cut than had been expected at this stage.
A further 200 jobs at the Department of the Environment, Food and Rural Affairs have been earmarked to be cut.
At the Department for Work and Pensions, 30,000 jobs are to go, 560 will be lost by the end of the month at the Department of Trade and Industry and 400 are to go at the Inland Revenue and Customs.
In his Budget statement, the chancellor said the first 12,500 civil service jobs had been cut, on target.
About 4,300 civil servants will leave London and the South East by the end of March 2005 and there are plans to relocate another 3,500.
Of those 300 Department of Health posts will go to Yorkshire while hundreds at the Department of Culture will move to Birmingham and Newcastle.
'Bonfire of the quangos'
Mr Brown also announced plans to merge 35 agencies into nine - described by one civil service union as a "bonfire of the quangos".
Mark Serwotka, the leader of the Public and Commercial Services Union said: "In last year's budget we had the day of the long knifes as the chancellor kicked off the crude game of who could cut the most civil service jobs between the government and the Tories.
"There was a time when the only worry thousands of hard working civil and public servants had on budget day was whether petrol or taxes would go up, nowadays the worry is whether they will have a job by the end of it."
He said Mr Brown had made welcome announcements on closing tax loopholes and extending the New Deal, while cutting "the very people who deliver them."
The Treasury also announced plans to reduce the number of public sector workers on sick leave with a new system of checks and tougher measures against those suspected of abusing the system.