"We clearly have lost ground, it would be almost absurd to think otherwise," he said.
He is expected to contact his lawyers on Monday as he prepares evidence for the inquiry into the Hinduja passport affair, which led to his downfall.
Mr Mandelson has promised he will not attempt to damage the government or Labour Party as he tries to prove he did not mislead Downing Street.
Mr Hinduja was part of the Hinduja Foundation, which donated £1m towards the Millennium Dome's Faith Zone, while Mr Mandelson was minister responsible for the Greenwich project.
His return to the UK coincides with renewed pressure on Europe Minister Keith Vaz with Tories calling for his resignation and government ministers offering their support.
Mr Vaz has faced a barrage of allegations over his role in the Hinduja affair and links to business.
Voters in Mr Mandelson's Hartlepool constituency are not convinced of their MP's innocence and do not want him to stand at the next election, according to local newspaper reports.
Party faithful
The regional Sunday Sun newspaper said a poll of readers showed they did not want him to stay.
The Hartlepool Labour Group has until now offered their support to Mr Mandelson.
Leader Russell Hart said the paper's poll results were disturbing, but dismissed suggestions their candidate would stand down.
"We clearly have lost ground, it would be almost absurd to think otherwise," he said.
"I wish the last two weeks had never happened and Peter Mandelson wishes the same.
"We have to move on and pick ourselves up."
Mr Mandelson, expected to visit Hartlepool later this week, has insisted he did not mislead people and wants to establish that he did not lie.
Part of his defence at the inquiry led by Sir Anthony Hammond QC is likely to be a memo from his former personal secretary which he claims proves he did not lie about his involvement in the Hinduja affair.
The Independent on Sunday quoted the memo as saying: "I am told you raised the issue with Mike O'Brien in a telephone call or personal note. Mike O'Brien doesn't remember how you raised it."