Andy Webster has been ordered by the Court of Arbitration for Sport to pay Hearts £150,000 for breaking his contract to join Wigan in August 2006.
Injured Webster has played just once for Rangers this season
Fifa had initially ordered the Scotland international defender to pay £625,000, and Hearts had been seeking £4.6m.
Hearts said: "We are extremely disappointed. This is a highly damaging decision for world football."
But players' unions have welcomed the ruling for increasing the freedom of employment for footballers.
PFA Scotland chief executive Fraser Wishart said: "This will be a revolution in book-keeping for professional clubs all over the world.
"Twelve years after Bosman, this is a new groundbreaking decision."
The 25-year-old Webster will split payment of the smaller sum with Wigan.
This is a very dark day for football clubs
Wigan, Webster and Hearts each appealed against Fifa's judgement in the case and the player, who is on loan with Rangers, may now make a permanent move.
The Court of Arbitration for Sport (CAS) decided Webster should pay up the remainder of his contract with Hearts, which had less than a year to run, as compensation for his departure from Tynecastle.
A statement said: "The CAS has determined that an amount of £150,000 has to be paid by Webster to Hearts as compensation for unilateral breach of contract."
Hearts were adamant they should receive more compensation because of the cost of replacing the Scot.
The CAS hearing took place in October 2007, in Lausanne.
Webster became the first player to invoke article 17 of Fifa's regulations, which enable a player aged under 28 to terminate his contract after three years - and older players to leave after two years.
The rule was introduced in 2001 after a compromise deal with the European Commission, which had threatened to abolish the transfer system.
FIFPro, the international players' union, has supported Webster throughout.
FIFPro lawyer Wil van Megen said: "This decision is perfectly in line with Fifa regulations and the Fifa-EU agreement.
"It respects labour law as well as the specific nature of sport. It is a further normalisation in the relationship between a professional player and a club.
"From now on, the market is more transparent and all parties will know where they stand at the end of a protected period."
Hearts described it as "a very dark day" for football clubs and would be discussing their next course of action with their lawyers.
Stephen Sampson, who advised the club on the case, told the club website: "This decision means that contracts can be torn up by players after the protected period, undermining the stability of player contracts across the football world."