First Minister Rhodri Morgan says all Welsh assembly politicians face "disgrace and humiliation" if they fail to agree a budget.
Rhodri Morgan's government has been hit by defections
He sounded his warning after opposition Assembly Members united to force Labour to re-think its financial plans.
They want the minority Labour assembly government to put more of the £13.8bn into education, council tax and rail.
Mr Morgan told BBC Wales' Dragon's Eye programme that the reputation of all 60 AMs was at stake.
Labour holds only 29 seats in Cardiff Bay since Blaenau Gwent AM Peter Law left the party to fight last May's general election as an independent.
It means the assembly government regularly loses votes. Its business statement has been lost twice in three weeks and Labour was forced to do a deal before the summer on university top-up fees.
Last week Plaid Cymru, Conservatives, Liberal Democrats, John Marek of Forward Wales and Mr Law joined forces to demand that the budget is re-drafted.
Mr Morgan told Dragon's Eye they were "opportunistic", adding: "The opposition parties have the votes and we have the coherence".
The budget has to be set by 10 December and if it is not, Mr Morgan said the task would fall to the assembly's top civil servant.
The final draft of the budget will have to be agreed by 10 December
"In the end if the assembly politicians can't set the budget, and this is the ultimate disgrace and humiliation that falls on all of us if we can't, the permanent secretary has to.
"And what's the public in Wales going to think of that?
"And what are they going to say in the House of Commons when we are asking for more legislative powers if we've had to sub-contract the process of setting the budget because the politicians can't make a fist of it?"
But Mr Morgan said Labour would not give up assembly power because of the long-term damage to the party.
"It's a struggle but they (the public) still expect us to deliver government.
"If we walked away the Labour Party would take decades to recover because it would be seen as an ultimate act of irresponsibility.
"The people of Wales expect Labour to be in government. Even if we weren't in government we would still get the blame for everything that happened because people think Labour is in government even when they are not."