The project could affect 60,000 people, reports say
Balfour was the main contractor on the $1.5bn dam, which aimed to form a reservoir on the upper River Tigris in the largely Kurdish south-east of the country.
The departure of Balfour, together with Impregilo of Italy, its civil engineering partner, throws the Ilisu dam's future into doubt.
Balfour's announcement pre-empts the British government decision on whether to grant it export credit financing - a decision that might have proved tricky after the dam failed a key environmental assessment in July.
"The decision follows a thorough and extensive evaluation of the commercial, environmental and social issues inherent in the project," the firm said in a statement.
"With appropriate solutions to these issues still unsecured and no early resolution likely, Balfour Beatty believes that it is not in the best interests of its stakeholders to pursue the project further.
"Given the substantial difficulties which remain to be addressed... Balfour Beatty believes the project could only proceed with substantial extra work and expense and with considerable further delay."
Multiple defections
Balfour's defection is the latest in a stream of bad news for the dam project, which has struggled to secure backing since its final designs were approved in 1982.
The Swedish construction firm Skanska quit the Ilisu project a year ago, citing the complexities of negotiating with the various parties involved.
After the defection of Balfour, only Austria's VA Tech - which specialises in hydro-electric technology - is involved as a foreign partner in the consortium.
The British government Export Credits Guarantee Department, which had been due to help with financing, confirmed that Balfour's pull-out meant that their involvement in the project was over.
Balfour Beatty's contract was worth nearly £200m.
Contentious project
British involvement in the Ilisu dam has been highly contentious.
Environmentalists, who have campaigned fiercely against the project, say it would affect the lives of 60,000 people, who would be displaced from the area around the Tigris.
International development agencies also worried that it would disrupt supplies of water around the Middle East - something that has been a najor political flashpoint in recent years.
And there have also been concerns that the flooding could destroy some of Turkey's most ancient archaeological sites - notably the Mesopotamian city of Hasankeyf.
Political questions
Previously, the UK government had thrown its whole weight behind the project, arguing that it was good for relations with Turkey.
It is not yet clear whether the Balfour pull-out will cause the project to be postponed, or even cancelled, but experts say it seems unlikely that VA Tech will be able to push the project through alone.
"Now there have to be big doubts over the project," Kerim Yildiz, executive director of the Kurdish Human Rights Project, told BBC News Online.
"We think it is fantastic news."
But while environmentalists and other campaigners will be cheered, the move will be seen as a blow to Turkey, which has struggled to attract major foreign investment projects in recent years.
The Turkish embassy in London said it was awaiting a statement from the foreign ministry in Ankara on the likely future for the project.