Tony Blair has again refused to name a date for his departure, saying he will not use the party conference to say when he is stepping down.
The prime minister returned from his holiday to face intense speculation about his future plans as leader.
In an interview with the Times newspaper he called for people to "let me get on with the job".
Mr Blair announced in September 2004 he would serve a "full third term" before stepping down.
With the Labour Party conference beginning on 24 September, key people have openly discussed Mr Blair's role during the summer.
His close aide Lord Falconer said it was not the time to change leadership, while Culture Secretary Tessa Jowell said Mr Blair's plans to step down before the next election were causing uncertainty.
But Mr Blair told the Times: "It is absurd for the people who say we must stop this continual speculation about the leadership to continue to speculate about it.
"I'm not the one who keeps raising this issue.
"I have done what no other prime minister has done before me. I've said I'm not going to go on and on and on and said I'll leave ample time for my successor.
"Now at some point I think people have to accept that as a reasonable proposition and let me get on with the job," he said.
He called on people to "stop speculating".
"If what they are really worried about is timing I think most of you can look at what I have said and draw conclusions about that."
Labour MP Derek Wyatt said the leadership issue had to be resolved before next year's local elections.
"Everywhere you go, wherever you go, everyone is talking about when is the day, and who will take over and so on," he said.
BBC political correspondent Laura Kuenssberg said many people at Westminster expected Mr Blair to decide on a departure date during his holiday, announce it before or during the conference and hopefully silence further speculation.
She said there had been growing clamour from supporters of Gordon Brown for an annoucement, who were unlikely to be satisfied with the latest Blair comments.