Bernard Jenkin has been axed from his role in charge of the controversial A-list of Conservative candidates.
Bernard Jenkin declined the offer of other front bench roles
John Maples, seen as a moderniser, takes over his role of deputy chairman with responsibility for candidates.
Mr Jenkin said the A-list had been a "huge and controversial challenge" but nominations of women, black and Asian candidates were increasing.
David Cameron announced the change in a reshuffle which also saw eight MPs elected in 2005 promoted.
Those joining the top team are Shailesh Vara, Ed Vaizey, Greg Clark, David Jones, Anne Milton, James Brokenshire, Mark Lancaster and Robert Goodwill.
Announcing the changes, Mr Cameron said Mr Jenkin had done a "tremendous job" and that the A-list of candidates had been a success.
The idea of the A-list is to try and help ensure that more women and people from ethnic minorities are selected as Conservative candidates in winnable seats.
NEW SHADOW MINISTERS
Anne Milton: Tourism
Greg Clark: Charities
Shailesh Vara: Deputy Leader of the House
James Brokenshire: Home Affairs
Ed Vaizey: Arts
David Jones: Deputy to Wales Minister
Mark Lancaster: Whip
Robert Goodwill: Whip
Maria Miller: Family Welfare
John Maples: Deputy Chairman
Henry Bellingham: Constitutional Affairs
Andrew Selous: Work and Pensions
Anne McIntosh: Education
The party says 35% of selected candidates so far are women and 10% from an ethnic minority.
But Ali Miraj, an Asian Tory hopeful who failed to be selected for a seat in Essex, said Mr Jenkin had wished him luck but said he would be shocked if a white male candidate was not selected.
"That to me was not a malicious comment in any way, he was merely reflecting the reality on the ground, and he wasn't the only one who said that to me," he told BBC Radio 4's The World At One programme.
He said he would have a better chance of becoming an MP in the Labour Party.
'Rather an insult'
But Mr Jenkin said he did not recall making those comments and told BBC Radio 4's World at One: "I think it's a bit sad that Ali's made this remark. There are lots of candidates who may be coloured or may be white who get disappointed in selections.
"I don't think it does you any good when you start throwing your toys out of the pram. To imply that the selection committee are racist because they didn't select him I think is rather an insult to them."
Mr Cameron said ex-Treasury minister Mr Maples had taken over the role because he felt someone else was needed to "take it forward".
Mr Jenkin, who has held a wide range of shadow ministerial posts including as shadow defence secretary, turned down an alternative front bench role and returns to the backbenches.
A spokesman for the Labour Party said the Conservatives' A-list had failed and was "clear evidence of how little the Tory party has truly changed".
Other Conservative MPs returning to the backbenches are Malcolm Moss, Mark Field, Tim Boswell and Andrew Turner.
The changes see Maria Miller move to become shadow minister for family welfare, Henry Bellingham appointed a shadow Department of Constitutional Affairs minister and Andrew Selous become a work and pensions shadow minister. Anne McIntosh becomes a shadow minister for education.