Ken Clarke - 59 votes
Iain Duncan Smith - 54 votes
Michael Portillo - 53 votes
Mr Clarke and Mr Duncan Smith now go forward to a ballot of 300,000 party members with the result due to be announced on 12 September.
The MPs final verdict overturns most pre-race expectations when Mr Portillo, once the darling of the Tory right and an all but anointed future leader, was the clear favourite.
In an exclusive interview with the BBC, Mr Portillo said: "I don't intend ever to be on the frontbench again and for the avoidance of doubt I'm not interested in the leadership."
Wishing the other candidates well, he went on: "I really don't think it's an option for me to serve in a shadow administration.
I think really the time has come for me to look for other things to do
"Apart from anything else I would just get in their way and make their life more complicated.
"I think the time has come for me to seek other things to do."
However, he added that he intended to stay in the Commons as MP for Kensington and Chelsea.
Ballot winner Mr Clarke said: "I am confident, but I have got to justify that confidence by explaining to the members up and down the country why I should be the next leader."
He said he was "very, very disappointed" by Mr Portillo's decision to return to the backbenchers.
Describing the single-vote margin of his rival's defeat as "rather a cruel blow", he expressed hope Mr Portillo would think again.
Mr Duncan Smith put his success over Mr Portillo down to his determination to "stick to the issues" in what had been a "very tight" race.
"I'm relieved and pleased that I'm in the last round," the second placed contender told Sky News.
The fact he had been tipped as the frontrunner in the lead-up to the final ballot was all "media concoction" and had never come from his camp's figures, he insisted
Mr Duncan Smith, who called on Tories to unite once the race was over, said he was also disappointed by Mr Portillo's decision to leave frontline politics.
"He would certainly grace any frontbench - either myself or Ken's," he added.
According to Portillo supporter Alan Duncan, Mr Portillo resolved to stand back from the frontbench if he lost "a few weeks ago".
Michael Portillo's campaign had come under mounting pressure
"I think probably his reasoning is to try and leave the decks clear for those left in the ballot," the Tory MP added.
Clarke supporter John Maples called the result "absolutely wonderful" while Eric Forth, who backed Iain Duncan Smith, said it gave the party membership "a very clear choice".
Mr Clarke's victory confounded most predictions as he piled on 20 votes to his total in last week's MPs' ballot.
It sets the stage for a fight to the finish between Mr Clarke's euro-friendly brand of Conservatism and Mr Duncan Smith's impeccable right-wing credentials.
Despite Mr Portillo's earlier placing as the front-runner, his campaign was recently hit by a series of blows.
There were newspaper claims he had failed to declare donations to his constituency party, although he denied any wrongdoing.
That was followed by speculation over his attitude towards Tory policy currently opposed to legalisation of cannabis, same-sex marriages and abolishing Section 28 banning the "promotion" of gay lifestyles in schools.
And over the weekend William Hague's ex-press secretary, Amanda Platell, alleged Mr Portillo's supporters undermined the Conservative general election campaign - a claim he denied.