Israeli Prime Minister Ariel Sharon's Likud party has voted against including the opposition Labour party in the government coalition.
Sharon has faced bitter opposition from his party to the withdrawal plans
A Likud official said that about 58% of party central committee members rejected Mr Sharon's proposal.
Mr Sharon had been seeking the coalition to advance his plan to pull out of Israeli settlements in Gaza.
However, the vote is not binding and Mr Sharon has said he will press ahead with his withdrawal plans regardless.
Labour strongly favours Mr Sharon's Gaza and West Bank disengagement plans, which the Likud party rejected in May.
Mr Sharon lost his parliamentary majority in June.
Speaking earlier at the Likud party convention in Tel Aviv, Mr Sharon said there was no alternative but to widen the governing coalition and proceed with his plan to withdraw from Gaza.
The meeting was called by critics of the prime minister, in an effort to rule out bringing the Labour party into government.
But Mr Sharon insisted he would "not disqualify or boycott anyone".
Pull-out from all 21 settlements in Gaza and 4 in West Bank
Preparation period to end by March 2005
Four-stage evacuation to be completed by end of 2005
Cabinet votes on each stage
"The Likud will conduct negotiations with all Zionist parties for expanding the coalition," the prime minister said before the vote.
Mr Sharon told Likud members that "difficult decisions" needed to be made, which would have consequences for the Likud and for Israel as a whole.
The Likud leadership had been expected to reject any coalition with Labour.
Mr Sharon was even jeered at by Likud central committee members shortly before the ballot.
The result of the vote will limit Mr Sharon's room for manoeuvre as he attempts to proceed with his plans for Gaza, and many believe it will mean early elections, says the BBC's Richard Miron in Tel Aviv.
There had been intense lobbying ahead of the Tel Aviv meeting.
On Tuesday the government announced plans to build more homes in settlements in the West Bank - a move interpreted by many as an attempt to buy over the critics.