Tony Blair's decision to delay his holiday has not done enough to answer criticism within his own party over his handling of the Middle East crisis.
Tony Blair believes the next couple of days are crucial
Glasgow MP Mohammad Sarwar said the prime minister's policy had "lost credibility" in the country.
He joined other Labour MPs in calling for the recall of Parliament.
But Work and Pensions Secretary John Hutton said the "sharp argument" in the Labour Party was not over whether to restore peace, but how to achieve it.
The prime minister has put back his three-week summer break by a few days to concentrate on securing a UN resolution for a ceasefire in the Middle East.
He has been criticised for refusing to call for an immediate ceasefire.
But Mr Blair has argued it is more important to secure a UN resolution so words could be backed by action.
A Foreign Office spokeswoman also confirmed that Foreign Secretary Margaret Beckett has left on her caravan holiday to France, but said she is prepared to travel to New York should the UN Security Council agree on a resolution.
Mr Sarwar said foreign diplomats, religious leaders and members of the public had voiced their concerns that Mr Blair was not doing enough.
''We believe that Israeli military actions are excessive, aggressive and disproportionate and our prime minister has failed to represent the county's feelings during this conflict," he told the BBC Radio 4 Today programme.
"He has refused to endorse Jack Straw's remarks that Israel's response is disproportionate... he has refused to call for an immediate ceasefire."
On Saturday an open letter calling for a ceasefire and signed by 40,000 people, including 110 Labour MPs, is being presented to Downing Street, the Independent newspaper reports.
Meanwhile, Birmingham Perry Barr MP Khalid Mahmood, parliamentary private secretary to Home Office minister Tony McNulty, has hinted members of the government may be considering their position.
"There are people in the Cabinet, there are other ministers that are looking at that.
"I think people have been very concerned about the lack of real progress in trying to seek a ceasefire in Lebanon at the moment.
"I think currently what people feel in the Middle East and in Lebanon is that we're not treating them on a level footing."
'Failure of nerve'
John McDonnell, the Hayes and Harlington MP who has said he will challenge for the party leadership when Mr Blair stands down, has also called for Parliament to be recalled from summer recess so Mr Blair could be "held accountable for his failure to work with others to secure peace in Lebanon".
BBC political correspondent Robin Brandt said that the prime minister was standing firm.
"Tony Blair thinks he is right, that is why he wants to push ahead and continue his commitment to what he thinks is sealing a UN resolution which he thinks is key to addressing the whole spectrum of problems in the Middle East."
However he added that Downing Street was "concerned".
"There are critics in his own party. There is pressure," he said.
Mr Hutton acknowledged that there was a "sharp argument" within the Labour Party over the current crisis.
But he insisted that the differences were not about the objective of restoring peace, but about the means of achieving it.
And he added: "What I do believe very, very strongly indeed, having spoken to the prime minister and knowing how he is approaching this issue, is that he is trying to bring this process of violence and death and destruction to an end as quickly as possible, but do it in a way that isn't just about words."