Incoming German Chancellor Angela Merkel has pledged to push on to create a coalition government, despite turmoil over the withdrawal of two key figures.
Mr Stoiber's relations with Angela Merkel have been tense
Efforts to forge a left-right cabinet suffered a setback when Social Democrat (SPD) chief Franz Muentefering said he would step aside as party leader.
On Tuesday, Christian Social Union (CSU) leader Edmund Stoiber pulled out of the government as economy minister.
Mrs Merkel said that she would push ahead with negotiations.
But the Christian Democrat (CDU) leader admitted the task had become difficult.
"It goes without saying that difficult discussions remain ahead of us, but we will conduct them with optimism, determination and a bit of joy that voters have chosen us for this job," Mrs Merkel said.
Meanwhile, SPD leaders are expected to nominate on Wednesday the governor of Brandenburg state, Matthias Platzeck, to replace Mr Muentefering, Reuters news agency reports.
The BBC's Ray Furlong in Berlin says that without Mr Muentefering and Mr Stoiber it is going to be much harder to form a viable cabinet - and it will be much weaker if it does come into being.
On Monday, Mr Muentefering said he would not seek re-election to his post next month after losing a key vote within the party.
He is one of the key architects of the bi-partisan coalition government with the CDU/CSU.
His position as employment minister and vice-chancellor in the emerging government is now in doubt too.
1. Christian Democrats/Christian Social Union (CDU/CSU): 226
2. Social Democrats (SPD): 222
3. Free Democrats (FDP): 61
4. Left Party: 54
5. Greens: 51
He made his announcement after losing a vote in the leadership over who would take the number two job in the party.
His favoured candidate was defeated by a leading left-winger - a slap in the face for Mr Muentefering.
Mr Stoiber had second thoughts about joining the cabinet after Mr Muentefering's announcement, conservative sources said.
The coalition talks follow inconclusive elections in September which left neither the SPD nor the CDU/CSU with a majority in the Bundestag, the lower house of Germany's parliament.
Wolfgang Bosbach, a senior conservative and Merkel ally, told Germany's N-TV television: "Angela Merkel is not to be envied at this time.
"It was one of the key ideas of the potential partners that the three party chairs would sit together at the cabinet table."
Other leading Christian Democrats have now started openly questioning whether the new government will be formed at all.
Mrs Merkel has not commented yet, but at the very least the timetable she set for being elected Germany's first woman chancellor on 22 November has been cast into doubt, our correspondent says.