Families face a higher rent for the land or having it fenced off
Families who lease part of their rear garden from Network Rail face seeing their payments soar up to six-fold.
Householders along Lansdowne Road in Canton, Cardiff, have been told they must pay almost £300 per year or see the bottom of their garden fenced off.
Network Rail said it wanted a "fair and sensible" solution for everyone and a "boundary line for the public".
It said there were inconsistencies and a third of the 155 properties involved had no tenancy agreement.
People in the Canton area, whose homes back on to the rail line to Cardiff Central station, have been able to rent the few yards of land leading to the bank which carries the track.
However, not everyone rents or uses the land, so some well-tended gardens are surrounded by overgrown shrub land.
The rail regulator held three residents' meetings this month to explain its proposals.
It said it was trying to simplify the system after piecemeal agreements made with householders by British Rail and Railtrack had resulted in people paying a range of rents, while others paid none at all.
One of the options available to householders is to buy the gardens, provided a minimum of 10 adjoining properties were involved.
Network Rail say the current system is inconsistent and unfair to some residents
If they want to rent the land, many have been told the lease will cost £250 per year plus VAT, a total of £293.75.
Brenda Brookman said: "It's not as if they they can use it, they're just trying to squeeze money out of us."
Mrs Brookman, 69, said she paid £15 a year to British Rail for a number of years. But after the rail firm was privatised she said this rent became index-linked and she now paid more than £40 a year.
She said: "We've looked after it so well after all these years that you don't want to see it anything other than tidy.
"I don't want to buy it at my age. They're not going to maintain it."
Her daughter-in-law, Jenny Brookman, said she was happy to pay the £123 per year Network Rail was asking for but felt the firm had handled its proposals badly.
She said: "We're quite devastated. Now they're saying it can go up to £250 recurring yearly. My husband's self-employed and I'm a part-time working mum, I just couldn't afford to pay that money."
Her neighbour Janet McCarthy said Network Rail should show more loyalty to people like her, who have rented the land for years.
"Whenever I have made any improvements, I have always had the [rail] inspectors in to see that I haven't encroached on their land," she said.
A spokesperson for Network Rail said 56 of the 155 properties in the Lansdowne Road area occupied its land without legal tenancy agreement.
She said: "We understand that the situation with each property at Lansdowne is very varied and we are also sensitive to their different needs, therefore we had organised a forum with the residents to hear their views.
"The open dialogue with them has and will give us a better understanding on their interests and then help us find an optimal solution.
"We are working towards achieving a solution that is fair and sensible for everyone, while at the same time enable us to create a more effective and safer boundary line for the public."
Network Rail believes the current situation is hugely inconsistent with some land occupied illegally.
Some residents pay for their tenancy, while some do not, while there are also problems of fly-tipping and access to rail land.
The market rent rate is £500 and Network Rail says it has have halved the individual tenancy rent rate to £250, or £20 a month, it has recommended a joint tenancy that can potentially cut the rate further by another 50%.