Mrs Livni said her Kadima party would form an opposition
Israel's prime minister-designate Benjamin Netanyahu says talks to include the centrist Kadima party in a coalition have ended without agreement.
The Likud leader said he had faced "categorical rejection" in 90-minute talks with Kadima leader Tzipi Livni.
Minutes earlier, Mrs Livni said the meeting "concluded without agreement on key issues".
Mr Netanyahu has been tasked with forming a government in the wake of elections earlier this month.
In other regional developments, Israel's military attacked alleged smuggling tunnels in Gaza on Thursday, in what it said was retaliation for earlier rocket launches.
EU foreign policy chief Javier Solana has been touring Gaza and expressed his support for its reconstruction, as the European Commission said it was ready to pledge 436m euros (£389m) in aid.
It was not clear if further talks would take place, but Mrs Livni said Kadima would be "a responsible opposition".
"I have done everything possible to achieve unity... but to my great regret, I faced categorical rejection from Mrs Livni," Mr Netanyahu said after the pair met in Tel Aviv.
Analysts say Mr Netanyahu is seeking a broad-based coalition including Kadima and the centre-left Labour party which might find more favour with Israel's international backers.
Javier Solana says reconstruction work in Gaza should start soon
A government of right-wingers and Orthodox Jews who champion settlement-building and oppose land-for-peace deals might jeopardise prospects for peace, they say.
Mrs Livni said the meeting ended without agreement on the matters "which to my mind are fundamental in order for us to join any government".
Kadima supports the formation of a Palestinian state in the West Bank and Gaza, which is opposed by Likud.
With 27 of the Knesset's 120 seats, Likud actually won one seat less than Kadima in the recent general elections.
But President Shimon Peres tasked Mr Netanyahu with forming the next cabinet as he stands a better chance of forming a coalition by the 3 April deadline.
Israeli planes attacked targets in Rafah in southern Gaza Strip on Thursday night.
The Israeli military said it had targeted smuggling tunnels. No injuries were reported.
It followed two rockets fired into southern Israel from inside the Gaza Strip, the army said.
Meanwhile, EU Foreign Policy Chief Javier Solana toured Gaza on Friday, visiting sites hit by Israeli warplanes during the recent offensive.
"When you look at what has happened, the damage and the destruction, it will take time to put it back in place. That is why it is so important to start very soon," he said.
As he spoke, the European Commission - the EU's executive - said it would pledge 436m euros of aid to Gaza.
The money - to be pledged at next week's international donors conference at Sharm el-Sheikh, Egypt - will be dedicated to humanitarian aid and early recovery of the Gaza Strip, the commission said in a statement.
Funds will also go to supporting the Palestinian Authority in implementing reforms and refugee programmes.
"Our priority today is to adequately respond to the disastrous humanitarian situation in Gaza," said EU External Relations Commissioner Benita Ferrero-Waldner.