The incident happened not far from the Macedonian village of Tanusevci - the focus of cross-border unrest involving ethnic Albanian rebels. Macedonian sources say the gunmen were trying to reinforce rebel positions in the village.
The BBC's Paul Wood says the clash is a very serious development for K-For, and raises the possibility that Nato troops could be sucked into combating the ethnic Albanian insurgency in Macedonia.
The growing insecurity in the region was also highlighted by a landmine explosion on the edge of the buffer zone between Kosovo and Serbia, which killed two Yugoslav soldiers and seriously injured two others.
Provocation
The shooting came only hours after Nato revealed that Yugoslav troops might be asked to help police the demilitarised zone around Kosovo, to deter attacks on Macedonia.
Nato's proposal has provoked a strong reaction from Kosovo Albanian leaders, who said it could spark fresh violence in the Balkans.
Kole Berisha, vice-president of the Democratic League of Kosovo Party, said they could not trust an army which until recently committed massacres in Kosovo.
He said installing the Serbian-led Yugoslav army would be "a provocation, making possible an open conflict that would include the entire region."
But Yugoslavia's ambassador to London, Vladeta Yankovic, told the BBC the aim of the Yugoslav forces would be to protect existing borders.
"We are all facing [the rebels], Macedonians, Yugoslavs, Nato together," he said.
Military statement
The US troops involved in Wednesday's shooting were on patrol with Kosovo's Nato-led peacekeeping force, K-For, when they were threatened by armed men and opened fire, said a US military statement.
The incident happened only a few miles from Tanusevci, near the Kosovo village of Mijak.
Some of the gunmen fled across the border into Macedonia but others were held, including one injured man, said a US spokesman.
Fighters from the rebel National Liberation Army have told the BBC that if any of those arrested were to be extradited to Macedonia, the organisation would consider American K-For troops to be legitimate targets.
The border has grown increasingly tense amid warnings from Macedonia that ethnic Albanians pose a threat to the stability of the entire Balkans region.
Rebel goal
Three Macedonian soldiers have been killed in rebel attacks in the past week, and there have been reports of rebels digging in along the border.
Macedonia fears that it could be afflicted by the same kind of ethnic conflict that has devastated other former Yugoslav republics.
The rebels, who want to unite parts of Serbia and Macedonia where ethnic Albanians live, have attracted strong international condemnation.
"We are looking very closely now at the possible decision to allow Yugoslav forces into the ground safety zone," Nato Secretary General, Lord Robertson, said in New York.
He said Nato was determined that the area "will not be used as a safe haven for extremists to launch attacks".
The zone, five kilometres (three miles) wide and 300 km (180 miles) long, is particularly sensitive at its southern end near Macedonia, where areas around the Presevo Valley have seen some of the highest levels of activity by ethnic Albanians.