Lebanon's Prime Minister Fouad Siniora has denounced Israel for what he called a border violation, which led to the first clashes since last year's war.
Israeli troops regularly patrol the border area
He condemned the "new Israeli aggression on Lebanon's sovereignty".
Israel says its troops returned fire after being shot at while searching an area where explosives were found.
No casualties were reported in the incident, which is being investigated by the United Nations peacekeeping force (Unifil) in southern Lebanon.
Unifil, which described the incident as serious, said it was assessing whether Israeli troops had indeed crossed the Blue Line, and not just a security fence.
Crossing the Blue Line constitutes a border violation, whereas going over the fence does not.
It is the first such armed incident on the border since Israel's 34-day campaign against Hezbollah guerrillas ended in August last year.
Since that conflict, the border between Israel and Lebanon has been policed by a combination of Lebanese government soldiers and UN troops.
According to Israeli reports, Lebanese troops fired when an Israeli patrol crossed a security fence near the Israeli village of Avivim.
"We called for them to stop firing, they shot at us and we returned fire," an Israeli source told the Reuters news agency.
Israel insisted that the patrol had not actually crossed into Lebanese territory.
Lebanese officials said the Israelis were across the border.
"An Israeli bulldozer crossed into south Lebanon tonight. Our forces opened fire at it. It pulled back and there was a brief exchange of fire," an army official told Reuters.
Israel said the bombs discovered earlier in the week were recently planted by Hezbollah, in violation of the terms of the ongoing ceasefire.
Hezbollah has denied the charge, saying the bombs had been there since the conflict six months ago.
A spokesman for the UN Interim Force in Lebanon, Milos Strugar, described the clash as a "serious incident".