The Federation of Tour Operators (FTO) has launched a legal challenge to Air Passenger Duty (APD), which may mean it has to be withdrawn altogether.
Air Passenger Duty may have to be scrapped altogether
The tax, doubled at the beginning of February, ranges from £10 for economy class travel within the EU to £80 for a longer-distance first-class flight.
The FTO says that the way the tax was doubled breached the Human Rights Act.
It also says that the government is not allowed to charge passengers for the right to leave a UK airport.
A Treasury spokesperson said: "The government is confident that APD is entirely legal and will robustly defend any challenge in the courts."
The FTO says that the 1944 Chicago Convention on International Civil Aviation, which has been part of EU law since 2004, only allows the government to make charges for providing a service - and that has to be related to the costs incurred.
Economy class flights in Europe, including internal UK flights: £10
Business and first class flights in Europe: £20
Economy class long-haul flights: £40
Business and first class long-haul flights: £80
If the tour operators are successful, the government may have to withdraw the tax and could also have to pay back more than £2bn that has been collected since 2004.
FTO Director General Andrew Cooper says: "Tour operators absorbing £50m of retrospective taxation is simply not an option."
He says the doubling of APD has had a disproportionate effect on tour operators, which, "unlike airlines, are largely precluded by law from passing on surcharges to customers who have already booked".
The legal challenge will argue that depriving tour operators of income to which they are entitled breaches their human rights.
The Chicago Convention has been used to overturn an airport tax in the past.
In 2005 the Administrative Court in Belgium finally ruled against the municipality in which Brussels Airport is based, which had made a charge per passenger on airlines using the airport.
The Court ruled that dues charged to airlines should be related to the cost of the services that they receive.