The report highlights that too few prisoners are learning basic skills
An "unacceptably" high number of people in England cannot read, write and count properly, MPs have warned.
The Public Accounts Committee said in 2007 51,000 pupils left school without a GCSE of at least D-G in maths and 39,000 left without this in English.
The report into adult literacy and numeracy also warned that only one in five offenders with poor basic skills had enrolled on a course to help them.
Ministers said no other government had invested so much in basic skills.
The committee of MPs said a lack of up-to-date information about skills meant the government could not be sure its schemes to improve basic skills were working.
Chairman of the PAC Edward Leigh said anyone who believed the government could meet its target of 95% adult literacy and numeracy by 2020 was "living in cloud cuckooland".
"Whilst they have made some progress, I don't think there's the remotest chance they will reach that," he told the BBC.
Even doing so would only bring England to the level currently achieved by the top 25% of the Organisation for Economic Co-operation and Development (OECD) countries, the PAC added.
Tackling poor literacy and numeracy skills was essential if more people were to realise their full potential and England was to remain competitive in the global economy, the report concluded.
The committee called for better efforts to recruit numeracy teachers and urged the government to do more to encourage public services, such as health and housing, to promote training opportunities.
The Prison Service could play a vital role in helping offenders, the committee said.
Mr Leigh said: "This is a dismal picture, both for the many who face diminished prospects in what they can achieve in life and for the competitiveness of our country in the world economy.
"It's down to teaching. As a country, we've got to accept that since the 1960s we have performed woefully in international league tables.
"We've got to accept there's something wrong with our teaching."
Further education minister Sion Simon said no government had done more to further the nation's literacy and numeracy skills.
"We have revolutionised the way in which basic skills are dealt with, and through the Skills for Life strategy, have helped more than 5.7 million adults to improve their numeracy and literacy skills.
'Too little known'
"Last year we launched a national high-profile numeracy ad campaign, 'Get On', to increase demand, and we are recruiting more high quality numeracy teachers. We have committed a further £3.9bn to Skills for Life through to 2011."
He said that since 2001, more than 2.8 million people had achieved Skills for Life qualifications.
A new survey of that scheme would be commissioned next year, he added.
Barry Sheerman, chairman of the Commons' schools select committee, criticised the PAC report saying it was a "thin piece of work", based on little evidence.
The Labour MP said: "To make sweeping generalisations about adult literacy and numeracy does a disservice to everyone, learners and teachers across the country".
The new survey was welcomed by Professor Greg Brooks from Sheffield University's School of Education.
Professor Brooks, who has conducted research into approaches to adult learning, said too little was known about what works to improve adult skills.
"We need to have more precisely focused investigation of the specific teaching methods which hold out the promise of better progress for learners," he said.