Culture secretary Tessa Jowell says Tony Blair's plans to step down as leader were creating "uncertainty" within the Labour party.
Mr Blair will step down as leader before the next general election
But she told the BBC speculation about a change of leadership was just part of the Westminster "soap opera".
Ms Jowell was speaking as the prime minister returned to Downing Street after his holiday in Barbados.
The prime minister is also facing renewed criticism about his handling of the conflict in Lebanon.
The Liberal Democrats said 37 Labour party members in the Derby South constituency of Foreign Secretary Margaret Beckett have defected to them in protest at his policy on the situation.
Ms Jowell said: "There is undoubtedly a sense of uncertainty and a sense of focus on this issue of the leadership.
"This is very much fed within the Westminster village but it is by no means the overriding issue in the Labour Party."
Mr Blair is spending his first weekend back after his holiday working on a speech he will give to the TUC annual conference next month.
In it he will address the challenges of globalisation, said a Downing Street spokesman.
"Concerns around security, immigration and community cohesion are issues that the public demand politicians put to the top of their in-trays," he said.
But Mr Blair is facing pressure to outline his departure plans at or before the Labour Party's annual conference in Manchester, in September.
He is expected to deflect questions about his future by revealing a number of policy initiatives over the coming weeks.
These will include moves to tackle social exclusion, improve health treatment and raise school standards.
Europe minister Geoff Hoon told BBC Radio 4's Today programme voters were starting to see Conservative leader David Cameron as a serious alternative to Tony Blair.
He said it was essential for Labour to demonstrate it could still come forward with new ideas and policies in the run-up to the next General Election.
Mr Hoon acknowledged the government had been going through a difficult period and that, after nine years in government, people were thinking there might be something better than Labour.
"There is no doubt in mid-term every government in history - apart actually from the last two Labour governments - has had the same kind of problems that we have seen," he said.
"There are a whole range of issues that lead people to think that, perhaps after a long period in government, there might be an alternative.
"There is no doubt that the Conservative Party looks very different than the one, say, Michael Howard led."