It was England's last chance to get the match changed. They are worried about safety and want to game moved from Zimbabwe to South Africa.
The England and Wales Cricket Board (ECB) had already been ordered by cricket's world council, the ICC, to play the match.
And now Judge Albie Sachs has agreed with the ICC.
The team now have to make up their minds on what to do. They have two options:
It could also threaten this summer's tour to England by the Zimbabweans.