British soldiers backed by American air power were sent to the eastern Paktia province, in an operation codenamed "condor".
US gunships and attack helicopters are now bombarding the area.
In recent days the US-led coalition has been stepping up its search in eastern Afghanistan for al-Qaeda and Taleban fighters.
The fighters are thought to have dispersed into small groups and blended in with local residents or fled across the border to neighbouring Pakistan.
'Enemy contact'
Speaking about Thursday's fighting, Brigadier Roger Lane, the top British commander in Afghanistan, said the Australian troops battled a "substantial force".
"I can confirm that the coalition has made contact with the enemy and that some have been killed," he said.
Brigadier Lane added that no coalition casualties had been reported.
Brigadier Lane said the fighting was taking place in a mountainous area at an altitude of 2,400 metres (8,000 feet), but declined to be more specific.
Coalition troops had reported no enemy contact for several weeks.
US-led forces launched an offensive in Afghanistan in October to root out al-Qaeda leader Osama bin Laden - the prime suspect in the 11 September attacks in the United States - and to punish his Taleban protectors.
Taleban and al-Qaeda fighters were expelled from the capital, Kabul, in November, and were pushed out of their main southern strongholds in December.
Arms dump
On Monday, British-led forces ended a two-week search operation in eastern Afghanistan saying they had dealt a "significant blow" to al-Qaeda's ability to mount future strikes by blowing up a big ammunition dump located in several caves in Paktia province.
That operation involved about 1,000 British and Afghan troops and was backed by American air support and special forces.
However, a local anti-Taleban commander later cast doubt on the British account of the operation, saying the arms no longer belonged to al-Qaeda and were being held in reserve for Afghanistan's future army.
Meanwhile, the mystery illness which has affected British army medical workers in Afghanistan has now spread to operational staff.
A further 22 cases have been reported, all from outside the original group of three-hundred still under quarantine.
The illness, which causes vomiting and stomach pain, broke out earlier this week at a field hospital at the Bagram air base.