Lack of land, "Nimbyism", and planning laws are causing rural social housing shortages and increased homelessness, housing associations are warning.
House prices are driven up by second home owners
The National Housing Federation has told the government's Affordable Rural Housing Commission that the situation is now critical.
Rural house prices are rising but there has been a fall in the number of new affordable homes being built.
The federation wants changes to planning to prioritise social housing.
In its submission to the commission, the federation says that from 1999-2003 the proportion of homeless households in rural areas increased by 24%.
In the South West - where housing shortages, second homes and low wages make it the most expensive area after London - the minimum target for new affordable housing per year was missed by over a third.
It says planning processes that can be slow and cumbersome, and "unthinking" opposition from people who take a "not-in-my-back-yard" attitude to developments is exacerbating the problem.
The federation wants a national rural housing strategy to be drawn up, VAT to be reduced to 5% for refurbishment of empty rural houses and for surplus government land to be used for social housing.
It is also recommending that more rural people be given key worker status and for the right to buy social housing to be restricted.
David Orr, chief executive of the National Housing Federation, said: "The acute shortage of affordable housing is a threat to the prosperity and existence of rural communities and market towns.
"A lack of land, a slow and unresponsive planning system and unthinking Nimby opposition is preventing not-for-profit housing associations from building the new homes needed to keep communities alive.
"We're not talking about huge numbers here either. Often just half a dozen new affordable homes can turn a declining village into a vibrant community with a real future - keeping open local services, such as the village school, which might otherwise be forced to close."
One rural resident in Thirlmere, Cumbria, said she was likely to be forced out of the area by rising rents and house prices.
Judith Birchall, who has two children, said the house her family rented had recently been sold at auction, with the local housing association unable to meet the asking price.
The new owners wanted to take possession of the house, but with Mrs Birchall earning low wages as a part-time telemarketer, and her husband working for the Forestry Commission, she fears they will not be able to afford anywhere else.
"I've been affected hugely. If they do get us out, where do we go? We can't afford to stay around here. There really aren't any affordable properties, even to rent.
"Well-paid jobs aren't here. And on these sort of wages we are not compatible with local house prices.
"It is definitely going to drive out local people. Houses tend to be bought by people from out of the area either as a second home or for holiday purposes. It has a knock-on effect on the schools.
"Unless more funding is made available for housing associations in a few years there won't be communities there will just be empty holiday houses and no local people."