Peers have taken part in a debate called by Liberal Democrat Baroness Doocey on the legacy of the London 2012 Olympic and Paralympic Games.
In September, the government published a 10-point plan for securing a lasting legacy from the Olympic Games. Elite sports funding will continue up until 2018, there will be a youth sports strategy and £1.5m invested in disability sport.
Sports Minister Hugh Robertson said momentum needed to be maintained if the commitment to "inspire a generation" was to be achieved.
But Lady Doocey told peers during the debate on 8 November 2012 that the legacy was "hanging in the balance".
She said the Games had been a "huge success" but there were "serious doubts" about whether the promise of benefits to neighbouring east London communities would be kept.
"There has been some very good progress to date but much of the legacy still hangs in the balance and urgent and sustained action is necessary to ensure that London does not fail," she told peers.
In particular, she called for more effort to encourage disabled sport and grassroots sport, and to provide more affordable homes and jobs for people living around the Olympic Park.
Crossbench peer and Paralympic medal winner Baroness Grey-Thompson said the London Games had a "significant impact" on the perception of disability.
"We must not forget this summer of sport. Respect for disabled people and celebrating what disabled people can do, and inclusion, isn't just for the Paralympics."