He made the shock announcement at the annual congress of the ruling United Malays National Organisation (Umno).
His intentions are unclear - Deputy Prime Minister Abdullawh Badawi says he has now retracted.
"I wish to announce I am resigning from Umno and all positions in the National Front [coalition government]," Mr Mahathir had told stunned delegates.
He was immediately surrounded by supporters begging him to change his mind.
Mr Mahathir, who is 76, has been prime minister of Malaysia for almost 21 years.
He did not specifically say that he was stepping down as prime minister, but correspondents say his announcement implied that was the case.
Out of the blue
The veteran prime minister had waited until the end of the South Korea-Spain match at the World Cup to make his address, which was shown live on television.
There was little sign of anything out of the ordinary as he made a detailed presentation of his government's plans.
Then came his shock announcement, at which point his cool poise began to desert him.
Shocked party loyalists chanted "Withdraw, withdraw".
"Why?" shouted Trade Minister Rafidah Aziz, according to Associated Press news agency.
"I have decided," Mr Mahathir said. "It's been a long time."
The prime minister was eventually led away to a waiting room by Ahmad Abdullah Badawi, his deputy prime minister and deputy leader of Umno.
Mr Badawi then returned to the podium and announced Mr Mahathir had retracted.
"After much persuasion from Umno Supreme Council members and Umno members, Dr Mahathir has agreed to withdraw his resignation," he told delegates.