Councils and housing associations should provide homes in areas where there are higher earners, a government-commissioned review says.
Social housing tenants are more likely to be unemployed
Professor John Hills also proposes spare land on council estates should be used to build private homes.
This would provide a better social mix - at the moment half of social housing is in the poorest 20% places, he said.
But he said ending security of tenure would be "very unhelpful" and tenants should be "incentivised" to move on,
The government has said there are "no plans" to end the role of social housing as a "safety net".
But it expects pressures on social housing to grow, with more single people needing homes and, as the gap between house prices and incomes grows, fewer tenants buying their own homes.
The type of council tenant has changed, the report found, with more likely to be on very low incomes, or not in work at all.
People of working age in social housing were twice as likely to be unemployed than the average person - partly due to the "neighbourhood effects" of living in deprived areas, such as welfare dependency and the difficulty of moving house.
Professor Hills, from the London School of Economics, said social housing played a "crucial role" in the lives of 4m households.
"But the evidence is that we are not realising its full potential," he said.
Social housing had been intended to avoid having "rich people on one side of the tracks and poorer people on the other side of the tracks", he said.
But communities had become polarised over the decades.
He argued that private estates had gained from social landlords moving in and renovating run-down properties.
"This isn't a one-way thing," he said. "And it's healthy - in the way society works - for there to be a mix."
And he said in some cases, there was no alternative but to rebuild the worst estates, although it was costly and caused much upheaval.
He also called for more choice for social housing tenants, including more help for them to buy their own homes and regular reviews of their circumstances, to see if they were able to buy a stake in their homes.
But he said tenants needed to be encouraged, not forced to move on.
"A threat to the security of tenure of existing tenants who have taken it for granted would be controversial, to say the least," he said.
"It would also have some side effects which could reinforce some of the key problems."
The government says it wants to expand the opportunity of home ownership to more social housing tenants.
Responding to the report, Communities Secretary Ruth Kelly said building more social housing was a priority.
But she added: "The Hills review confronts us with other challenges: how can social housing be even better at achieving the twin goals of providing support to those who most need it while also equipping people to progress and get on with their lives".
The Conservatives say the current system for selling council homes to tenants was "needlessly bureaucratic".
For the Liberal Democrats communities spokesman Andrew Stunell said: "The government has taken 500,000 homes out of the social housing sector, and now, surprise, surprise, there are an extra 500,000 families on council waiting lists.
"Changing security of tenure is simply rearranging the deckchairs whilst the ship sinks. This government must build more social homes now."