Shelter has welcomed Gordon Brown's commitment to build more affordable housing for people to buy and rent.
Gordon Brown said three million new homes will be built by 2020
The government is expected to unveil plans for thousands of new council homes after 10 years of Labour opposition to such housing.
Adam Sampson, chief executive of the housing charity, said more social housing was needed as fast as possible.
But Defend Council Housing blamed government policy of "dis-investing" in council homes for a housing shortage.
Speaking at a Labour Policy Forum on Saturday, Gordon Brown said he had seen young children suffering because of sub-standard accommodation and young couples unable to buy their first home.
He said local authorities and housing associations would have a larger role and more money would be put into rented accommodation.
Mr Sampson said: "Whether or not it has to be council is neither here nor there. The key thing is that the government bring on as quickly as possible as much socially rented homes as we can create.
"But it's also important that those homes are built in the right place and we don't go back to the council ghettos that we had in the past."
He added that the major shortage in social housing was a consequence of selling off a third of the six million council homes under the right-to-buy scheme.
Last week, Gordon Brown announced that three million new homes would be built by 2020 - up 250,000 from the previous plan.
Alan Walter, chairman of the Defend Housing campaign, told BBC News 24 it had been "fighting for more than 10 years" to stop government privatising.
"They've been blackmailing council tenants to have a change of landlord in return for improvements, and that's not on," he said.
"Part of the reason we've got a housing crisis is that the private sector just hasn't ever delivered the decent, affordable, secure homes that people need.
"And that's why people fought for council housing in the first place."