Defence Minister Bob Ainsworth says security in Basra is "far from ideal" after the withdrawal of UK troops, but the "final story isn't written yet".
UK ministers say Britain will continue to help Iraqis in Basra.
He was speaking as British forces transferred control of the province to the Iraqi authorities, four-and-a-half years after the invasion.
Tory Liam Fox says it "remains to be seen" whether Iraqi forces will cope.
Al-Qaeda's Ayman al-Zawahiri says the UK's decision to "flee" Basra shows Iraqi insurgents are gaining strength.
Iraq took formal responsibility for security in Basra province on Sunday.
The 4,500 British troops still in Iraq will now take a back seat role, focusing on training local security forces.
Mr Ainsworth denied UK forces had been forced to retreat because they "realised the Iraqi army was riddled by militia".
"We have not been forced to retreat," he told BBC Radio 4's Today programme.
"We have pulled back from Basra in co-operation with the Iraqis themselves and with the agreement of the Iraqi government and with the agreement of the coalition."
But he admitted that the presence of British troops had been making the situation worse.
"That is true and nobody has denied that we became the target over time," he said.
"We made progress more quickly in Basra in terms of our relationship with the population. Over time that deteriorated - nobody has denied that.
"There is rarely perfection in circumstances like these. We have never denied that the situation in Basra is far from the ideal and nobody is trying to suggest that it is.
"We would like the situation to be far better than it is, but nonetheless... we believe that they are up to the job of running that province and putting it in the right direction so there's hope for the future.
"The final story isn't written yet. We will continue to assist, support and to train and we will get more into the civilian areas of assisting with economic regeneration which is surely the future for the people of Basra."
'No clear mission'
But Colonel Tim Collins, who was the commander of the 1st Battalion of the Royal Irish Regiment at the time of the Iraqi invasion in 2003, said the withdrawal from Basra had "badly damaged" Britain's military reputation.
"I am not aware of any aim or objective and that was part of the problem," he told Today.
"There was no clear mission. I think the whole enterprise has been muddled thinking and characterised by a lack of planning and over optimism."
And shadow defence secretary Liam Fox questioned what role British troops would now have in Iraq.
"The prime minister has said that the force will be able to redeploy as a combat force," he told Today.
"Now, first of all, is having 2,500 troops big enough to allow that to happen? And secondly, under what circumstances would such a force redeploy and who would make those decisions?" he said.
"We really do need to know today the answers to those questions."
Timetable for return
Acting Liberal Democrat Leader Vince Cable said that if power was being handed back to Iraqis "there is no justification" for the continuing presence of 4,500 troops in the country.
"The government should set a timetable to bring all British troops home from Iraq," he said.
A BBC Panorama investigation is to reveal growing evidence that dozens of Iraqi civilians are still being tortured and murdered.
It shows that nearly five years on from the invasion of Iraq, Basra is now a city where powerful militias and criminal gangs operate freely and where being the wrong religion can lead to kidnappings, attacks and abuse and where women, in particularly face daily threats to their safety.
Panorama has seen police records of 47 women who have been killed because of their clothing or behaviour.