Michael Howard is now a "lame duck" Conservative leader and should quit immediately, one of the party's leading financial backers has said.
Michael Howard is expected to complete his reshuffle on Wednesday
Dixons president Lord Kalms said Mr Howard had done a terrific job but "once you're gone, you're gone".
Other donors disagree and Mr Howard says he will stay on until leadership election rules are reviewed.
Tory MP John Bercow has criticised the Tory election campaign, suggesting its focus on immigration was "repellent".
Mr Howard is expected to reshuffle his shadow cabinet.
No Tory MP has declared himself as a leadership challenger but party co-chairman Liam Fox is making a speech about fundamental Conservative values on Tuesday.
Dr Fox has made clear, however, that he will not use the speech to enter the fledgling leadership contest.
Other figures tipped as possible candidates include David Davis, Sir Malcolm Rifkind, David Cameron and George Osborne.
Former Conservative treasurer Lord Kalms wants the leadership race to start now.
He told BBC News: "There's nothing worse than a lame duck leader.
"I mean Michael's done a terrific job and he's tried very hard and he's had inexhaustible energy but once you're gone, you're gone and the quicker you go, the better. Every day is now a wasted day."
Lord Kalms was also critical of the Tory election campaign, calling it a technocratic agenda which had made voters' eyes glaze over.
The party needed to be more aspirational, he said, with more emphasis on its opposition to the European constitution and its championing of measures for wealth creation and cutting business regulation.
John Bercow has raised worries about treatment of immigration
Mr Bercow said the manifesto had been "embarrassingly thin" and calling Tony Blair a liar had played into Lib Dem hands by giving the Iraq war more prominence.
Repeatedly talking about immigration "seemed at best obsessive and at worst repellent", he told BBC News.
He said the party must be reformed as effectively as Tony Blair reformed Labour so the Conservatives could fight elections on the centre ground.
The Conservatives' chairman of treasurers, Jonathan Marland, argued such public criticisms were unhelpful.
"I am frankly rather tired personally as a member of the party of everyone airing their dirty linen in public," he said.
"I wish they would shut up and we can debate behind closed doors."
Oliver Letwin could move from being shadow chancellor
Mr Marland said there was an overwhelming desire among donors, MPs and Tory members for Mr Howard to stay as leader as long as possible.
Another major party donor, Sir Stuart Wheeler, said he wished Mr Howard had not said he would go at all.
There should be "considered debate" before choosing his successor, said Sir Stuart.
Front bench departures
There is still no timetable for Mr Howard's departure and his most pressing task is to rebuild his frontbench team.
Sources close to the shadow chancellor Oliver Letwin have dismissed reports that he is to stand down from the shadow cabinet.
But they have not rejected suggestions that he is to move to another post in the shadow cabinet.
Newspaper reports suggest Mr Letwin wants to move out of his Treasury role so he can resume work as a director of the NM Rothchild merchant bank.
Nicholas Soames and Tim Yeo quit the shadow cabinet on Monday. Education spokesman Tim Collins also lost his seat in the election.
Mr Yeo wants to leave his environment brief so he can focus on climate change but has refused to rule himself out of a bid for the Tory leadership.
Mr Soames, who was shadow defence secretary, says Europe and planning are also important.
He says suggestions he wants the leadership are "stupid" and has also brushed off suggestions that he wants to be chairman of the 1922 committee of Tory backbenchers.