Tony McNulty has been replaced as immigration minister by police minister Liam Byrne in a home office reshuffle.
There are no official estimates of illegal immigrants in the UK
A home office spokesman rejected claims the move was a result of recent immigration scandals.
Mr McNulty will take over the police brief from Mr Byrne, who had been in the job for just two weeks.
David Davis, for the Tories, said if Mr McNulty "is not good enough to run the immigration department, how will he be capable of running the police force?"
But the home office insisted Mr McNulty was the best person to push through police reforms, including the controversial merger of regional forces, because he was a former government Whip.
Mr Byrne, on the other hand, was an expert at management reform and therefore best placed to sort out the Immigration, Nationality Directorate (IND), the spokesman said.
"The right people have been given the right jobs - we've looked at their skills and decided who was best for the job," a spokesman told the BBC.
He said the moves were part of Home Secretary John Reid's strategy to "sort out the Home Office".
Mr Reid has also said he will boost public protection following cases where released prisoners have gone on to commit serious crimes, including murder.
He has announced plans to "re-balance" the justice system in favour of the victims of crime.
He told the Parole Board of England and Wales that victims or their representatives would in future have a greater say on the release of offenders back into the community.
Mr Byrne was handed the policing role on 5 May in Tony Blair's cabinet reshuffle, which saw Charles Clarke sacked as home secretary following a row over the deportation of foreign prisoners.
But a home office spokesman said the moves had been "provisional".
According to Downing Street, Tony Blair wrote to Mr Reid on 15 May setting out John Reid's priorities at the home office.
Mr Blair wrote: "As a first step, I would be grateful if you come back to me by the end of May on how you have allocated responsibility for key issues and objectives among your Ministerial team.
"I am keen that we make the very best use of our excellent team in government".
In the past week, Mr McNulty has found himself in the firing line again after the official in charge of removals told the home affairs committee he had no idea how many illegal immigrants there were in the UK.
On Tuesday, Mr Reid faces a grilling on immigration by the same committee.
Home office officials had been due to face the MPs, but Mr Reid has stepped in to their shoes in an attempt to seize back the initiative and prevent further damaging revelations.
He will also issue MPs with an update on the hunt for more than 1,000 foreign criminals mistakenly released without facing deportation.
Meanwhile, in separate developments, an immigration official has been suspended over allegations that he offered to help a teenage asylum seeker with her application in return for sex.
It has also been disclosed that 232 foreign nationals arrested in counter-terrorism operations were allowed to remain in Britain as asylum seekers.
And an investigation is underway into claims that five illegal immigrants arrested after being sent to work at the IND's central London offices were employed on the service's premises for years.
On top of that, the Criminal Records Bureau (CRB) was found to have wrongly labelled around 2,700 ordinary people as criminals.