The peer made his comments in a House of Lords debate
A Tory frontbencher has apologised after using a racist phrase during a House of Lords debate.
Lord Dixon Smith retracted the remark in the chamber, saying: "I apologise, my Lords. I left my brains behind. I apologise to the House."
The 73-year-old said later the phrase "nigger in the woodpile" was in common use when he was younger.
But Labour MPs called for him to be sacked, with one saying the remark was "deeply offensive".
Lord Dixon Smith made the comments - which were recorded in Hansard - in a debate on Monday on the Housing and Regeneration Bill, as he was setting out his objections to the creation of a new Homes and Communities Agency.
He said: "The Homes and Communities Agency is not a body to which we object in principle.
"As the minister has explained, it is an amalgamation of the Housing Corporation and English Partnerships.
"Of course, the nigger in the woodpile, as the noble Baroness, Lady Hamwee, has already pointed out, is that it still incorporates what I call the hangover of the new towns legislation.
"If it were not for that, we would have little difficulty with the foundation of this agency."
Another Conservative peer, Lord Brooke of Sutton Mandeville, suggested that Lord Dixon-Smith, a shadow local government and communities spokesman, reconsider his choice of words.
"My Lords, before my noble friend sits down, he used a phrase about a woodpile," Lord Brooke said.
"If your Lordships' House were happy, I think it would perhaps be helpful if the wording of the phrase were revised."
Lord Dixon-Smith replied: "I apologise, my Lords. I left my brains behind. I apologise to the House."
Lord Dixon Smith later apologised to Lord Strathclyde, the Conservative leader in the Lords, over the remark.
He told The Times it had "slipped out without my thinking," adding it had been "in common parlance when I was younger".
A spokeswoman for Conservative leader David Cameron said: "This was not an appropriate thing to say and it was absolutely right that he apologised to the House."
She said he would not be sacked from the front bench over the comments.
Keith Vaz, chairman of the home affairs committee, said: "I regard this remark as racist, because it's deeply offensive.
"It shows a lack of understanding and sensitivity to the ethnic community and seems to come from a throwback age when people used that kind of phrase as if it was normal."