The two sides have had troops and equipment in the area since July
Two Cambodian soldiers have been killed in an exchange of gunfire with Thai troops along a disputed section of their border.
The clash, near the ancient Preah Vihear temple, has prompted Thailand to urge its nationals to leave Cambodia.
Tension has been high since July, when hundreds of soldiers on both sides faced off metres apart.
Both sides have said they want to find a peaceful solution and will hold talks to discuss the conflict.
Thailand and Cambodia both claim they own the area around the temple, which recently became a Unesco World Heritage site.
Officials from both countries have claimed the other side fired first.
The exchanges of small arms fire and rocket-propelled grenades were reported to have continued for about an hour.
Cambodian Foreign Minister Hor Namhong said two soldiers had been killed and two wounded while Thai officials said five Thai troops had been wounded.
Mr Hor Namhong said another 10 Thai soldiers had been captured but would be well treated and returned home if Bangkok requested. However, Bangkok said that none of its soldiers were missing.
A Cambodian army official told the Associated Press news agency that commanders from both sides were trying to negotiate a ceasefire.
Thai Prime Minister Somchai Wongsawat said the fighting was "small scale" and that he was committed to reaching a settlement over the issue with Cambodia, which he described as "a good neighbour".
After fighting broke out, Thailand alerted air force jets and readied transport planes to evacuate Thai nationals from Cambodia.
The military stand-off began in July when Cambodian troops detained three Thai protesters who had entered the site illegally.
More than 1,000 soldiers from both countries moved into the area, digging trenches into the rough terrain around the temple.
TEMPLE DISPUTE TIMELINE
1962: International court awards temple to Cambodia, but surrounding land remains undesignated
1970s-1990s: Khmer Rouge guerrillas occupy site
2001-2002: Thai troops block access over water row
July 2008: Unesco lists temple as a World Heritage Site
July 2008: Thai FM quits after court rules he violated constitution for backing Cambodia's Unesco bid
July 2008: Both sides move troops to temple area
August 2008: Troops withdrawn after high-level talks
October 2008: Fighting erupts around temple area
In August, military personnel agreed to withdraw most of the troops from the area but in early October, Cambodia claimed that Thai troops had returned.
Thailand has denied that its troops entered Cambodian territory.
But on Tuesday, Cambodian Prime Minister Hun Sen threatened to turn the area into a "death zone" if the Thai troops did not withdraw.
The two countries have held several rounds of talks but have so far failed to reach a settlement.
Mr Hor Namhong said that talks to be held on Thursday were a good sign that the countries could begin to resolve this week's conflict, which was said was "an incident between soldiers and not an invasion by Thailand".
The stand-off between the two countries centres on 1.8 square miles (4.6 sq km) of scrub near the 900-year-old Preah Vihear temple, which sits on a jungle-clad escarpment dividing the countries.
The temple is only accessible from Thailand and the area around it is heavily mined - a legacy of Cambodia's long war against the Khmer Rouge guerrillas.
Preah Vihear is listed as a Unesco World Heritage Site
An international court awarded the temple to Cambodia in 1962, but land surrounding it remains the subject of rival territorial claims.
The decision by the UN in June to list Preah Vihear as a World Heritage Site reignited lingering nationalist tensions.
Disputes between the two countries date back centuries when the Thai and Khmer monarchs fought each other for territory and power.
In 2003, the Thai embassy in Phnom Penh was torched by rioters angry over the alleged comments of a Thai actress who said that the Angkor Wat temple complex should be returned to Thailand.