David Cameron says it is "delusional" to talk about bringing in more grammar schools, as Conservative rebels are doing in calling for a policy U-turn.
He told a Westminster press conference there was a "fantasy element" to the debate, saying they had not built new grammars during 18 years in power.
The old Tory policy had been "a chain around our necks", Mr Cameron added.
His comments came amid unhappiness from some Tory MPs and supporters over the end of support for academic selection.
Mr Cameron has said his party is against the creation of new grammar schools but will not push to close existing ones.
At a meeting of the influential 1922 committee of backbench Tory MPs last week, Mr Cameron's policy was denounced as "ridiculous" and "absurd".
But the Conservative leader told a press conference on Monday: "There's a fantasy element to the debate.
"It's delusional to talk about these things in the future when we didn't do them in the past."
He added: "We need to clear this out of the way. It's been a chain around our necks."
Last week he told the Evening Standard the party would "never be taken seriously by parents" while it backed selection.
There are 164 grammar schools in England - choosing pupils by academic ability at the age of 11 - with 10 local education authorities considered to be fully selective.
A YouGov poll for the Daily Telegraph suggested that 70% of Tory voters support selective education.
But shadow education secretary David Willetts said his party backed the expansion of Labour's city academy system instead.
Academies are non-fee paying, non-selective state schools, which operate outside the control of local education authorities and have private sponsors.
Mr Willetts said last week that academic selection does not help social mobility.
He argued that middle class parents could coach a less gifted child to do better in an exam at aged 11 than a bright child from a less well-off background.