Gordon Brown says plans to build new huge so-called "Titan" prisons "will go ahead... after consultation", rejecting suggestions in the Commons of a u-turn.
Cutting prison budgets is fraught with risk, warns the chief inspector.
Justice Secretary Jack Straw had earlier raised doubts over plans for new 2,500-place prisons in England.
"We haven't got planning permission for these places. We are not definitely going ahead with them," Mr Straw said.
Chief Inspector of Prisons Anne Owers criticised the plans. The Tories accused the government of a "u-turn".
Mr Straw announced plans for more prisons, including three "Titan" super jails, housing 2,500 inmates each, in December.
'Get a grip'
However, he cast doubt on the scheme during an interview with BBC Radio 4's Today programme on Wednesday when he talked about the lack of planning permission.
He added: "We're not going to have large warehouses as they have in the United States and indeed France."
But just hours later at prime minister's question time, Mr Brown appeared to counter that suggestion by saying the scheme "will go ahead" following consultation.
Tory shadow justice secretary Nick Herbert said the government had "u-turned twice on Titan prisons within the space of five hours".
"The public remains in the dark about whether Titan prisons will go ahead or how much they will cost," he said.
"The government shows a complete lack of grip on criminal justice policy."
Ms Owers, who warned the prisons system was at breaking point, said she did not think "prison building should be driving our prison policy".
Resources needed to be freed up to support people after jail and to "prevent people getting there in the first place", she said.
"We also need effective prisons and if we look across the Channel we see the French who built one of these kinds of prisons in the 1980s and have never done so again.
"We do need to learn from evidence."
She said prisons were struggling to cope with population pressures and were at a "crossroads".
During 2007 the prison population hit a record high of more than 81,000, and that figure is predicted to carry on growing.
'Feeling the strain'
While Ms Owers' report praises work in healthcare, education and managing offenders, she said these gains were at risk because of the overcrowding across England and Wales.
She said figures showed there had been 40% more self-inflicted deaths over the year - particularly at the critical stage of reception.
The government's use of police cells to house some prisoners was undermining work to cut suicides, she said.
"During the reporting year, the prison population went from one all-time high to another, staving off disaster only by a series of short-term, often expensive, emergency measures, together with the crisis management skills of those working within the prison system," she said.
"Training prisons, as well as local prisons, felt the strain, with more suicides, poorer resettlement outcomes and in many cases insufficient activity.
"The management and use of indeterminate sentences continued to strand those prisoners in inappropriate prisons, and drive up the population.
"Even male juvenile prisons - the best-resourced male establishments - performed less well as a whole than those we inspected the previous year."
Ms Owers warned a planned 3% cut in the prisons budget from April would "effectively close prisons down" from Friday lunchtime to Monday morning.
"Our prison system is at a crossroads," she said. "There are signs of a more effective and measured approach to policy and strategy, some new initiatives, and plenty of good operational practice to build on.
"But, on the other hand, the risk is that we will move towards large-scale penal containment, spending more to accomplish less, losing hard-won gains and stifling innovation."
However, Mr Straw said under Labour crime has fallen and the number of prison places provided has risen by 20,000.
There are currently 1,200 spare places, with an additional 1,000 coming on stream in April and 2,600 during 2008.
Mr Straw said overcrowding was "a problem", but he denied this would automatically lead to the early release of prisoners
However, he added: "You can never ever completely rule these things out."
Liberal Democrat leader Nick Clegg said: "This breathtaking report shows Britain's prisons are on the brink of disaster.
"This isn't a problem we can simply build our way out of...Instead we must get people with mental health or drug problems out of prison and into treatment."