Ten of the top seats targeted by Tories at the next election have so far chosen a candidate - eight chose men and two women.
But party sources say attempts to judge the success of efforts to get more women elected are premature.
The Conservative leadership, which wants the party to better reflect British society, has so far ruled out directly forcing shortlists on local party associations.
Enlarging talent pool
Tory members in most of the seats which have so far chosen candidates have chosen the same person who lost for them last time.
Five of those 10 target constituencies have, however, been fast-tracked because the candidates performed well at the last polls.
That means the figures are skewed to show that past candidates were being chosen rather than new faces, party sources say.
John Strafford, chairman of the Campaign for Conservative Democracy, said it was "no good blaming local associations" for the shortage of women candidates.
Seventy of the 83 women on the Tory candidates list at the last general election were chosen by local associations, Mr Strafford.
"The problem comes from getting women to put themselves forward for the candidates list," he argued.
That was because most candidates came from the Conservatives' declining membership, which could be boosted by more party democracy, said Mr Strafford.
In an interview on Thursday, Mr Duncan Smith suggests the "left-wing" media have managed to put his party into a box "marked: nasty, extreme, strange".
'Decent' party
Mr Duncan Smith told the Daily Mail: "I have to break us out of that."
The Tory leader adds: "I want people to feel easier about us; to feel that being a Conservative is not something you hide... because those who are publicly Conservative are decent people who are rather like them.
"I personally believe that the party I lead is a decent party full of decent people.
"And if there are people who are indecent and beyond the pale, then I have demonstrated that I will act very hard and very quickly on those people."
As the debate about the party's future direction continues, the UK's highest profile black Conservative, Lord Taylor of Warwick, has been accused of failing to do enough to change their image on race.
'Stop whingeing'
Shailesh Vara, the first Asian Tory vice-chairman, said Lord Taylor's complaints were doing the party "no favours".
Lord Taylor should stop "whingeing and carping" about the party leadership, Mr Vara told the Daily Telegraph.
"It is time for him to roll up his sleeves, to recognise the enormous positive changes that have happened in the party and are continuing to happen under Iain Duncan Smith," he said.
Former Conservative shadow cabinet minister Francis Maude has said the party needs to move faster towards the radical reform to attract voters.
Mr Maude says the party needs to lure in more women and ethnic minority candidates.
It should shy away from the more typical stereotype of straight, white males from traditional Conservative backgrounds, he said.