The government should spend three times as much on social housing, homeless charity Shelter says.
Much accommodation is squalid or unsuitable
The group's call follows a major investigation which heard accounts of suicidal depression and self-harm amongst children in bad housing.
Shelter wants a new government department to be set up, dedicated to housing and communities.
Squalid conditions are blamed for worsening health problems such as skin diseases and respiratory disorders.
Shelter's report carries stories of "buggy babies" - infants left in their prams because of lack of space or poor conditions, whose heads are permanently deformed as a result.
According to the government, the number of homeless households living in temporary accommodation has more than doubled since 1997 to reach an all-time high of more than 100,000, including 116,581 children.
And Shelter says there are thousands more, either not regarded as vulnerable enough to be housed or "sofa surfers" living in friends flats.
The investigation - entitled Generation Squalor - found "housing wealth" in the best-off 10% of areas had increased by 20 times more than the worst-off 10% of areas.
This meant that, while 20 years ago the money spent on an average house in Kensington and Chelsea could buy you two or three average houses in Fife, the figure had now risen to 24.
The investigation was carried out by a panel of 10 experts from a number of fields outside housing, led by journalist Fiona Millar, partner of former Downing Street communications chief Alastair Campbell.
The panel included fashion designer Wayne Hemingway, Lindsay Nicholson, editor-in-chief of Good Housekeeping magazine, broadcaster Tony Wilson and female rapper Shystie.
It was completed by experts from architecture, public health and rural affairs.
The panel heard stories of overcrowded, damp flats, full of mould and peeling wallpaper during visits to families and interviews with those whose work touched on the field of housing.
Almost one in 10 children are now living in overcrowded conditions, Shelter said.
And contrary to many people's assumption that homelessness was solely an urban issue, the problem is growing faster in the countryside.
Shelter director Adam Sampson told BBC News the current figure of 25,000 to 30,000 social houses a year needed to be trebled.
"No matter what cunning wheezes you come up with you are not going to do anything about the housing crisis unless you build for those in greatest housing need.
"Successive governments have not been investing in building affordable houses.
"A lot of people now recognise that their children may now be priced out of the home ownership market.
"Poor housing shortens life expectancy and correlates with asthma and TB."
Panellist Shystie said in the report: "I grew up in Hackney. There were six of us living in a two-bedroom flat. It was meant to be temporary. They put us on the waiting list for five years. How long is temporary?"