The two airlines said they would not go ahead if the regulatory price was too high.
"Regrettably this has proved to be the case," British Airways' chief executive Rod Eddington and American Airlines chairman Don Carty said in a joint statement.
"The conditions laid down by the US government do not make sense for either company."
"We will not acquiesce to unrealistic, and in our view, unnecessary demands. For us the price is just not right."
Open skies talks off
The US Department of Transportation said the two airlines would be allowed to coordinate their ticket pricing and flight schedules, provided they surrendered 224 of their lucrative Heathrow landing slots to US carriers.
US officials added they would withhold final approval until the US and the UK governments sign an 'open skies' agreement, committing both sides to liberalising the market for air travel between the two countries.
Talks on a transatlantic open skies deal were due to resume between US and UK officials on Monday, but have now been postponed.
"We still want to work with the Americans towards a full and genuine liberalisation of the UK/US market," the UK Foreign Office said in a statement.
"Ministers have told their US counterparts that we hope to re-engage with them soon."
More talks?
Some analysts reckoned the airlines would try and persuade the US Transport Department to ease the conditions.
"This may just be posturing by the airlines as part of their effort to get an alliance on their terms," said Richard Gritta, an airline finance professor at the University of Portland.
"I think they will go back to the negotiating table and try to reduce the conditions and keep more slots," said ABN Amro airlines analyst Ray Neidl.
But he warned "it'll be a long drawn-out process. Both sides have started to dig in."
'Waste of money'
Sir Richard Branson, the boss of rival airline Virgin Atlantic, was sharply critical of both British Airways and the government.
"This deal was never going to be acceptable to the regulators," he said.
"BA has wasted five year's of everyone's time."
He added that questions should be asked as to why BA had been allowed to "waste such a colossal amount of taxpayers' money."
"Both Conservative and Labour administrations have failed to secure a UK/US open skies deal because all recent proposals were based on the narrow, anti-competitive interests of one British company."
He called for open skies talks to be handed over to the EU to enable a deal to be struck between Europe and the US.
11 September fallout
BA and AA have both been hit hard by a slump in passenger numbers since the 11 September attacks.
BA, which reported 98% drop in profits for the three months to September last year, is carrying out a number of cost-cutting measures in a bid to avert major losses.
City analysts say the airline could lose as much as £750m by the end of the financial year in April.
Two weeks ago, American Airlines' parent company AMR said it lost $800m during the final three months of 2001.
BA shares closed 10.25p lower at 217.25p on Friday, off earlier highs of 240p.