Mr Duncan Smith delivered his speech at the Tories' Harrogate spring conference only hours before Fellowes won an Academy Award on Sunday night for his screenplay for Gosford Park.
The Oscar for original screenplay was the only award that Robert Altman's film picked up at the ceremony.
Senior Conservative party officials said Mr Fellowes was one of a number of people who had been asked to submit ideas for the speech, along themes set out by Mr Duncan Smith.
A senior source said: "Julian submitted ideas in accordance with the brief Iain had given him.
"The speech was very heavily worked on over quite a number of drafts but some of his ideas - some of the language particularly - got in."
It is common for political speeches to have several authors.
A Conservative Central Office spokeswoman told BBC News Online: "As usual, Iain Duncan Smith gave the brief to a number of contributors and then he and his team worked over it until they were satisfied."
Mr Duncan Smith and Mr Fellowes - who is said to have considered standing as a Conservative candidate in the past - first met at a charity ball.
At a later meeting Mr Duncan Smith told the scriptwriter that he had a major speech coming up and asked if would contribute.
Actor
The speech delivered in Harrogate focused on Mr Duncan Smith's visit to the rundown Easterhouse estate in Glasgow - a very different world from the 1930s English country house portrayed in Gosford Park.
Mr Duncan Smith was said to have been impressed by the work of Mr Fellowes, and the scriptwriter is expected to contribute to future speeches.
Fellowes, 52, who is also an actor, was approached by director Robert Altman to write the screenplay for the film after adapting the children's classic Little Lord Fauntleroy for BBC TV.
As well as Sunday's Oscar, he has had a Golden Globe nomination and a New York Film Critics Circle award.
He is perhaps better known to UK audiences from his numerous supporting roles in TV and film.
He was a government minister in Bond movie Tomorrow Never Dies and was a Scottish laird in BBC TV series Monarch of the Glen.