"I don't believe in balancing the budget on the backs of the poorest," he told the party faithful in Manchester.
His proposed public sector pay freeze - which exempts the frontline military and anyone earning less than £18,000 a year - goes further than the one announced by Labour on Monday, which will be restricted to GPs, judges and other high earners.
Mr Osborne said the measures contained in his speech would, by the end of the next Parliament, be saving £7bn a year in government spending.
He said he could not think of abolishing Labour's new 50% tax rate on the rich "while at the same time I am asking many of our public sector workers to accept a pay freeze to protect their jobs".
In a series of measures likely to form the backbone of the Conservative election manifesto, Mr Osborne proposed a freeze on the pay of government ministers and curbs on "excessive salaries at the top of Whitehall".
Any new public employee with a salary greater than the level of the prime minister would need special permission from the Treasury.
And he said the time had come to "find ways to impose a £50,000 annual cap on the size of public sector pension payouts".
He also vowed to cut the cost of Whitehall - proposing to slash departmental budgets by a third during the lifetime of Parliament, a move he said would save £3bn.
Tax credits for families earning more than £50,000 a year would be ended and the Child Trust Fund would be limited to the poorest third in society.
But child benefit will be preserved as a universal benefit and the party would not axe the winter fuel payment or free TV licences for pensioners, he said.
Mr Osborne also confirmed plans revealed on Monday to bring forward a rise in the age at which men can claim a state pension to 66 from 2016, up to ten years earlier than planned.
He also said that in the longer term he would also seek to reverse the "pensions tax raid" imposed by Gordon Brown as Chancellor when Labour first came to power.
Explaining the need for the tough package of measures, Mr Osborne said: "These are the honest choices in the world in which we live and we have made them today.
"Anyone who tells you these choices can be avoided in not telling you the truth. We are all in this together."
The government has come under fire from public sector unions for its plans to freeze the pay of senior civil servants such as judges and GPs.
Chancellor Alistair Darling has written to salary review bodies calling on them to freeze the pay of 40,000 senior public servants in 2010/11.
He also recommended that about 700,000 middle-ranking public servants, including doctors, dentists and prison officers, get a rise of between 0 and 1%.
Mr Osborne attacked Mr Darling for "sneaking out" the announcement during the Tory conference saying "he didn't have the guts to announce it to the Labour Party conference".
But Liam Byrne, chief secretary to the Treasury, said: "George Osborne appears intent on talking Britain down. We were told his speech would tell us 'everything' we needed to know about how the Tories would get the deficit down. But he lost his nerve.
"How can George Osborne say 'we're in this together' and then recommit his party to a tax giveaway to the wealthiest 3,000 estates?"
Liberal Democrat treasury spokesman Vince Cable said he called the proposals "Lib Dem Lite".
"The sum total amounts to nothing more than a drop in the ocean and will not deal with the structural deficit. What is more, Osborne has in effect guaranteed that these policies will merely pave the way for a return to traditional Tory politics - hitting the public sector now to pay for tax cuts for millionaires later," he said.
Public sector workers warned of strike action over the pay freeze plans.
Public and Commercial Services union leader Mark Serwotka told the BBC: "Over 24 hours we have seen Alistair Darling and George Osborne playing politics with people's lives, asking low paid public workers to pay for a crisis they did not create.
"If they intend to freeze the pay of hard-pressed public sector workers, many of whom are on low wages, I think there's a very strong likelihood of industrial action."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.