More than 100,000 families earning £50,000 or more receive means-tested tax credits designed for the less well-off, official figures show.
Tax credits are designed to help working families
The figures also show 7,000 recipients have incomes of more than £57,500.
Tory spokesman George Osborne said they showed the "distorted priorities" of Gordon Brown's tax credit system.
But Paymaster General Dawn Primarolo said tax credits aimed to benefit nine out of 10 families to reflect the costs of bringing up children.
"The Tories have now admitted they would remove this support from at least 112,000 middle income families, and they will not stop there," the paymaster general said.
"If they believe that middle income households should receive no support from the state, they must say how many families they will remove from child benefit, and how many pensioners they will remove from the State Pension, the Winter Fuel Allowance, and the Council Tax Rebate."
Facing the music?
Figures provided by Ms Primarolo show in 2003 to 2004 112,000 families with incomes of more than £50,000 were awarded tax credits.
Of those 72,000 had incomes of more than £52,500; 36,000 of more than £55,000 and 7,000 of more than £57,500.
Mr Osborne said: "We really should be asking ourselves whether taxpayers on average incomes of £15,000 should be providing means-tested benefits to people earning up to £66,000 a year.
"It is time Gordon Brown stopped getting his junior ministers to answer for the failing system and stood up to face the music himself."
Earlier this month the Lib Dems attacked the rising costs of managing tax credits which had gone from £36.1m to £403m in four years.
The system has also come under fire recently for problems with its payments.