The prime minister has gone to Chequers for a quiet weekend after hospital treatment for an irregular heartbeat.
Mr Blair is 'tackling paperwork'
Tony Blair, who underwent a procedure on Friday, sparked speculation about his future after he said he intends to serve a full third term if elected.
BBC political correspondent Ian Watson says Gordon Brown's allies feel "betrayed" by this announcement.
But on Saturday the Chancellor issued a call for party unity, saying people's prosperity was the priority.
In Washington for annual IMF and World Bank meetings, he warned the government must not become distracted by speculation over the prime minister's future.
"The priority is to get on with the job of ensuring prosperity for the British people," he told the BBC.
"We have got to have the strength to take long-term decisions. We must never be distracted from that."
Earlier, former Cabinet minister Jack Cunningham reacted to leadership speculation saying people with "their own interests" in the leadership should "shut up".
The prime minister has announced his intention to serve a full third term in office if elected, but not a fourth. Mr Brown was in Washington when the news broke.
Mr Blair told BBC political editor Andrew Marr he felt he could still make a "real contribution" to the country.
"If I am elected I would serve a full third term. I do not want to serve a fourth," he said.
'Like a coup'
The chancellor's allies believe a deal was brokered for Mr Brown to take over from Mr Blair, according to BBC political correspondent Ian Watson.
He told Radio 4's Today programme that they point to Mr Blair's recent house purchase and the timing of his operation as evidence he had planned to hand over the reins of power at the Labour party conference.
A Brown aide is quoted in The Guardian as saying of Mr Blair's third term announcement: "I can't believe he's done this."
"It's like an African coup. They waited until he (Brown) was out of the country."
But Dr Cunningham, the former Cabinet "enforcer", said there is not a shred of evidence to suggest a deal had been struck between the two men, dismissing it as "gossip".
Speaking on Today, he said: "Frankly it would help if others who have an idea about their own interests in the succession would just shut up and help him get on with governing the country."
"I just wish people in the party would focus more on the challenges that are coming up in the next general election and less on the speculation, media gossip and tittle-tattle about the prime minister."
As Mr Blair left Downing Street for his official country residence on Saturday he smiled broadly and said he felt "excellent".
Doctors have said the chances of Mr Blair's heart flutter returning are very low and he plans to go ahead with a trip to Africa next week.
He has no formal meetings over the weekend, Downing Street said, but he would be tackling some paperwork.