A councillor is calling for a ban on people buying second homes on Portland in Dorset.
Weymouth and Portland will host the Olympic sailing events in 2012
Independent councillor Jacqueline Redfern says people in the area are being priced out of the housing market by wealthy outsiders.
The isle has become increasingly popular after it was announced it will be the host of the sailing events in the 2012 Olympics.
It was also made part of the recently designated Jurassic Coast.
Ms Redfern, who represents Tophill East, said wealthy investors from other parts of the country "are buying up anything that goes on the market and the local people aren't really given a look-in, especially as the wages are so low".
She said people must have the impression that because house prices are high, people must be wealthy down here.
"If anything, it is the complete opposite," Ms Redfern added.
"We actually need help down here just the same as the outskirts of London. It is just as high, our rents down here, as London properties against wages.
She said something like the Olympics is bound to bring problems to the area, and one of those problems is the surge in house prices.
"There's going to be a lot of people looking for rented accommodation in the area running up to the Olympics, which will mean even less available for local people," she said.
"We have lots and lots of people queuing up to buy affordable houses and no housing to provide it and very little land to build new homes."
Weymouth and Portland Borough Council said the current stocks of affordable housing are insufficient to meet local needs and 78% of households cannot afford to purchase a home in the lowest price range.