About two million homes will have to be built on greenfield sites to meet the prime minister's plans to tackle the housing shortage, a think tank warns.
The need for affordable homes has been put top of Mr Brown's agenda
Gordon Brown has pledged three million homes will be built by 2020, mainly on previously developed brownfield sites.
But a Social Market Foundation study claims two million homes would have to be built on undeveloped countryside or green belt around cities and towns.
The government reiterated that it plans "robust protection" of the green belt.
Mr Brown has put housing at the top of his agenda since he became prime minister and announced plans to increase the rate of new development.
He told MPs last month: "Putting affordable housing within the reach not just of the few but the many is vital both to meeting individual aspirations and to securing a better future for the country."
But Mr Brown also pledged the government would "continue to protect robustly the land designated as green belt".
Some 60% of the proposed new homes would be built on brownfield sites under government plans.
But the SMF study suggests even if the new homes were built on a density equivalent to London only 2.1 million would be on brownfield land and this would mean some parks and gardens being paved over.
The report found that on a more realistic housing density, "almost two million homes would need to be built on non-previously developed land".
The SMF concluded: "It will not be possible, even if those living in towns and cities accept the loss of their gardens and parks, to meet the UK's housing needs through previously-developed land alone."
It also suggested that the target of three million homes was "likely to be the minimum needed" as supply was failing to meet demand in a "fundamentally unbalanced" UK housing market.
The SMF also added that the green belt, which was planned to prevent urban sprawl, contains ex-industrial sites and scrubland and "was not as green as people believe".
The think tank suggests there may be a case for reconsidering the future of the green belt which often protects "neither wildlife nor areas of outstanding beauty".
The SMF's director Ann Rossiter said the UK faced "tough choices" in meeting its housing need and had to tackle the "not in my backyard" mentality.
She told BBC Radio Five Live: "We have to face the fact that if we want our kids and our friends' kids to have somewhere to live that's of a decent standard, those homes are going to have to go somewhere.
"And maybe they have to go in the field next to our house, and maybe they have to go near the view that we've always loved - but that's the reality of the situation."
But Richard Bate, from planning consultancy Green Balance, said the green belt served a number of crucial purposes.
These included serving as a distinction between town and country, preventing parts of towns and cities becoming derelict and stopping nearby towns and cities merging into each other.
"Simply letting the market rip in areas where it would like to go - very often in green belt areas - won't necessarily put development in the places that will do the most good for everybody in town and country alike," he told Radio 4's Today programme.
Housing and Planning Minister Baroness Andrews said the government believed it was possible to build the homes needed by future generations while protecting the environment and green spaces.
"Our clear priority for development will remain brownfield land - already 74% of new housing is being built on brownfield land, up from 57% in 1997," she said.