The prisons minister has agreed to meet Usk councillors to discuss concerns about inmates escaping from Prescoed open prison.
Several prisoners have absconded in recent months
The talks with Gerry Sutcliffe follow months of campaigning from locals.
Seven prisoners have absconded from the Monmouthshire jail since July, including child rapists and a murderer.
The Home Office said all prisoners moved to open prisons like Prescoed were carefully vetted beforehand to ensure they were no danger.
But people in the town are worried the government will use open prisons for more serious offenders as jails across the UK reach full capacity.
Welsh Social Justice Minister Edwina Hart and former Home Office Minister Alun Michael have expressed concerns over the amount of prisoners who have gone on the run.
The seven absconders since July have included child sex offenders John Elms and Martin Aspinall, and murderer Lee Dewhurst.
All three were caught and returned to prison. Lee Dewhurst was recently found hanged in his cell in Bristol prison.
Usk nursery group chair Kate Parish said parents in the town are terrified every time they hear a police helicopter.
"When the helicopter starts circling overhead, which has been a regular occurrence over the summer, suddenly there's phone calls from mother to mother, father to father... and it is panic," she said.
"I won't go on local walks now with the pushchair or the children on my own, simply for the fear that somebody has escaped from Prescoed."
Usk town councillor Tony Kear said he welcomed the prison minister's statement this weekend that violent or sexual offenders should not be moved from category C to category D, or open, prisons.
He said councillors had been arguing this point for more than two years.
"I think what we've got here is an opportunity to press home the real fears that are in the community. Kate Parish accurately sums up what's going on here at the moment.
"There is a crisis in confidence in the prison in Prescoed," Mr Kear said.
"We don't have an issue at all about it being an open prison. It's been there since 1939 and to our minds it is about rehabilitation.
"It has a place in society. But not for convicted sex offenders and particularly child rapists who are the ones who have absconded."
The Home Office said some of the absconders from Prescoed involved prisoners out on licence who simply did not return.