Labour leader Amram Mitzna turned down Mr Sharon's request in his first meeting with the right-wing leader since Likud triumphed in last week's election.
Mr Sharon has made it clear that he favours a coalition with Labour to one with other right-wing parties and religious groups.
But Mr Mitzna had insisted throughout the election campaign that his centre-left party would not join a government with the hawkish Likud party.
"Sharon's positions are non-starters," said Avraham Shohat, a Labour MP. "We can't be in a government where everything that we told our voters will not be fulfilled."
Negotiations
In contrast to Mr Sharon's hardline stance, Mr Mitzna favours unconditionally resuming peace talks with the Palestinians and advocates dismantling Jewish settlements in the West Bank and Gaza Strip.
Meeting on Monday, Mr Sharon and Mr Mitzna spoke on security and political issues, the prime minister's office said.
"The prime minister (told Mr Mitzna) that a unity government, as broad a government as possible, was vital to the people of Israel, especially in the face of challenges on our doorstep and the hopes which the future holds in store for us," said a statement from Mr Sharon's office.
Likud won 38 seats in the 120-member Knesset (parliament), replacing Labour, which dropped from 26 seats to 19, as Israel's biggest party. It was Labour's worst ever election result.
Israeli President Moshe Katsav will this week formally ask Mr Sharon to form a government and he will then have up to 42 days to do so. He will need to reach a majority of 61 seats.
Mr Sharon's party has been in coalition talks with the secular centre-right Shinui Party, which finished third in the election.
However, Shinhui is reluctant to sit in government with two ultra-Orthodox Jewish parties that together make up 16 seats.