Council chiefs and residents have reacted angrily to speculation that a new prison could be built in Cwmbran.
The home secretary has pledged to provide a further 8,000 places
It is thought the Home Office is considering building a jail to help combat overcrowding, on the site of a former police training college.
The leader of Torfaen Council said the site was in the wrong place as it was in the middle of a housing estate.
The Home Office has refused to comment on which sites it is considering for new prisons.
Last year, the home secretary pledged to provide a further 8,000 prison places, but the chief inspector of prisons has warned the system is in "serious crisis".
All four Welsh jails in Swansea, Cardiff, Bridgend and Usk are overcrowded, and one solution could be to build a new prison.
But Colin Davies, the chair of the residents' association for St Dials said the former training college in Cwmbran was not a suitable site.
"We think its the wrong area because of the built up area of houses and the children playing," he said. "I don't think there's a need to go in the middle of a housing estate on a size of this level.
"We have enough problems in this area without creating more problems."
The college is owned by the UK Government which will have to apply through the normal planning system to turn the site into a prison.
Torfaen Council leader Bob Wellington said he would oppose any plans for a prison as it would jeopardise multi-million pound plans to regenerate the nearby town centre.
"We have plans which we've been working on for years which we've spent a considerable amount of money on, which would really transform the nature of Cwmbran town centre," he said.
"If this goes ahead, I'm advised by my officers that that will jeopardise those plans."
Torfaen MP Paul Murphy said it would be "daft " to build a prison on the site as it was "wholly inappropriate".
"There are literally tens of thousands of people living around a potential prison site," he said.
Mr Murphy said it would be sensible to put a new prison in an area of high unemployment.
"There are parts of Wales that would inevitably benefit from a prison going there in terms of jobs," he added.
A Home Office spokeswoman said the promised 8,000 additional places would be created by a mixture of expansions at existing prisons and by building a number of new prisons.
"Decisions on location and function of the new places will be determined by greatest need.
"Where possible new capacity will be provided in areas of highest demand. Specific sites are still subject to acquisition and planning consent," she added.