Voters in the May local elections will be passing a verdict on "all of us", not just Tony Blair, says Gordon Brown.
Mr Brown is likely to face a leadership challenge
The chancellor made the comments on ITV's Sunday Edition when asked whether the electorate would be judging 10 years of him and the prime minister.
But Mr Brown said people would also vote on local issues, and said that Labour's apparent poor opinion poll ratings were "a phase".
Tory leader David Cameron said Mr Brown was part of a failed government.
Mr Brown is the favourite to succeed Mr Blair when the prime minister stands down.
Polls predict that the party will fare badly at the ballot boxes on Thursday in elections for local councils in England as well as the Scottish Parliament and Welsh Assembly.
Asked if they would be a referendum on Tony Blair, Mr Brown said: "Well, people will be voting on all of us."
He added: "I think primarily people will be looking also at the powers that the assembly and the parliament has, and what local government's doing, and voting on these things as well."
When questioned as to whether Tony Blair had "hung around too long and damaged you in the process", he replied: "Yes - well, that, you have... you have to establish."
But asked about poor opinion poll ratings he said: "You go through phases", adding that the only result that matters in the end is when it actually comes to a general election".
Tory leader David Cameron has repeated his call for an early election if Mr Brown becomes prime minister this summer. But the chancellor rejected the idea, saying many before him had arrived there mid-term.
"I didn't hear Mr Cameron or the Conservatives calling for a general election the minute that John Major took over in 1990.
He said David Lloyd George, HH Asquith and Winston Churchill had not faced such calls either.
"I think you've got to put this in its proper historical perspective, that the British people elect a House of Commons, and out of the House of Commons the leadership is chosen."
Lib Dem leader Sir Menzies Campbell said he did not think Mr Brown was in a position to hold a snap election anyway.
"First of all Labour has no money, and secondly the opinion polls are very damaging to him," he said.
Mr Brown looks increasingly likely to face a Labour leadership challenge after it emerged there will only be one left-wing candidate.
Labour MPs Michael Meacher and John McDonnell - who had been planning separate leadership bids - have agreed one of them will withdraw based on how much support they have when Mr Blair resigns.
The Sunday Times has reported that Mr Blair will back Mr Brown as his successor after Environment Secretary David Miliband indicated he would not run.
On Sunday Health Secretary Patricia Hewitt said she believed Mr Blair would publicly endorse a successor "fairly soon".
But the Conservative said Mr Brown was responsible for NHS targets and taking away responsibility from teachers and police officers.
"Gordon Brown to me is part of the government that has failed, is part of the government that has given us top-down state control."