Home Secretary David Blunkett has said he just wants to get on with his job amid Tory claims he may have to resign.
The Home Office says it is wrong to speculate on Mr Blunkett's future
An inquiry is investigating whether Mr Blunkett helped fast-track a visa application for his ex-lover's nanny.
Shadow home secretary David Davis says he should quit if he is found to have influenced the process even indirectly.
Mr Blunkett says people are probably sick of hearing about him while No 10 said Tony Blair's backing had not changed "one iota" since last week.
On Sunday it emerged that Mr Blunkett's former lover, Kimberly Quinn, had offered to give evidence to the inquiry, headed by Sir Alan Budd.
Mr Blunkett, who launched a video to teach teenagers about personal safety and combat guns and knives at a London school, denies intervening to help her nanny, Leoncia Casalme, from the Philippines.
Leaving home in his Sheffield constituency on Monday, Mr Blunkett said: "If people are sick of hearing and reading about David Blunkett, I have every sympathy with you because I am as well.
"I hope I can now just get on with the job, the job that I have been appointed to do by the prime minister and do it to the best of my ability."
A Home Office spokesman said: "The Opposition should wait until the inquiry has reported before they decide that David should resign.
"Like anyone else, he is entitled to the presumption of innocence."
The Home Office spokesman dismissed as "complete speculation" a newspaper report that Mr Blunkett would admit Ms Casalme's residency application was fast-tracked, but claim an "overzealous" junior staffer was to blame.
Another report claimed Mr Blunkett chaired a meeting to discuss delays in the system after he learned of Ms Casalme's wait.
The Home Office spokesman said it would be up to Sir Alan's inquiry to decide if any such meeting was relevant.
Mr Davis told the BBC on Sunday he believed the allegations facing the home secretary, if proven, would be enough to force Mr Blunkett to quit.
The allegations centre around Mrs Quinn's nanny
"If he took the papers in, and if that led to influence, even if he didn't give the instruction in writing, as I'm sure he didn't, that I'm afraid, is a very unwise thing for him to do and I actually do think this is a resigning issue."
But it was too early to say whether Mr Blunkett was guilty of the alleged offences, Mr Davis said.
Shadow secretary of state for the family Theresa May also called on Mr Blunkett to "consider his position".
BBC political correspondent Vicky Young said those comments signalled a "change of tack" from the Conservatives, preparing for the possibility that the Budd inquiry finds no evidence of direct intervention by Mr Blunkett.
On Monday, Lord Chancellor Lord Falconer strongly defended Mr Blunkett, arguing that newspapers were publishing details about his private life which had nothing to do with his public role.
"David Blunkett is showing that he is performing his job as home secretary 100%," said Lord Falconer.
Mrs Quinn's husband Stephen has confirmed that she will offer evidence to the inquiry.
"She has written (to Sir Alan) and she will testify and she should testify, but it's her decision," he told Channel 4 News.
"She can testify in written submission, I think she should be written to. I don't think this should be some sort of whitewash."
He refused to join calls for Mr Blunkett to step down if the inquiry found against him.
"I'm only concerned about the health and welfare of my wife and child," he added.