Many pupils who left primary school in 2008 failed to grasp basics in maths
More than 30,000 pupils finish primary school in England at 11 with the maths ability of a seven-year-old, a report by the National Audit Office says.
So they are four years behind the expected level in mathematics as they start in secondary school education.
The NAO said the government had spent £2.3bn - an average of £572 per pupil - on primary maths teaching in 2006-07.
But it said the target of 78% of pupils reaching the expected level in maths and English by 2011 may not be met.
The NAO's investigation into primary maths teaching found almost a quarter of pupils who left primary school in the summer 2008 had failed to grasp the basics in maths.
Out of 588,000 children who sat the national curriculum test last year, 135,000 did not achieve the expected Level 4 at the end of Key Stage 2 of their education.
And 34,000 of these were only at the level expected at the end of Key Stage 1 (seven-year-olds).
Maths specialists in primaries
In June, the government announced 13,000 maths specialists would eventually be sent into England's primary schools to help improve maths teaching.
Ministers have committed £187m to train the specialists over the next 10 years.
The move followed a government-commissioned review of maths, led by Sir Peter Williams, which criticised the amount of training teachers received.
The UK was one of the few developed nations where it was acceptable to say you were "useless at maths", Sir Peter said.
Ability and ethnicity
The NAO study found variations in pupils' performance according to their ethnicity, with Chinese and Indian children doing consistently better than white children.
Pupils from black African, black Caribbean, Pakistani and Bangladeshi ethnic groups did "significantly less well".
The performance gap between disadvantaged groups and their peers was "very large".
"At Key Stage 2 the difference is currently 20 percentage points, with only a small narrowing of the gap over the past three years."
Girls were also found to be falling behind boys in national curriculum maths tests.
The NAO said able pupils also needed more support to do well in the subject.
Its report said that last year, 66,000 pupils did not make the nationally expected level of progress by the end of primary school, even though their earlier attainment suggested they could have done.
Chairman of the committee of public accounts Edward Leigh said: "The bottom line is that improvements in mathematics results since 2000 have been unimpressive.
"And, most worrying of all, nearly a quarter of all primary school children are not reaching the levels they need to in basic maths before moving on to secondary school."
Tim Burr, head of the National Audit Office, said: "The rate of improvement in primary mathematics has slowed and almost a quarter of pupils are still not equipped with the understanding of mathematics they need to study the subject further or to tackle subjects such as science once they start secondary school.
"The department needs to improve how teachers assess pupils' progress throughout the primary years, so that they can support every child to do their best in mathematics."
Schools minister Sarah McCarthy-Fry said maths was a "key priority".
"It's true that in recent years the rate of improvement has slowed - we're impatient to do even better and that's why we are taking further decisive action to help all children reach their potential," she said.
Shadow Schools Minister Nick Gibb said: "Too many children are leaving primary school without the basic skills in arithmetic to equip them for the secondary maths and science curriculum."