The Philippines is investigating several recent bombings
All three countries have made a series of arrests over the last six months as the spotlight has turned on Asian Muslim militants in the wake of 11 September.
Mr Mahathir goes to Washington next week to receive thanks from US President George W Bush for Malaysia's help on America's war on terror.
'Useful weapon'
Mr Mahathir and visiting Philippines' President Gloria Macapagal Arroyo witnessed the signing of the trilateral accord by foreign ministers from the three countries at Malaysia's new administrative capital, Putrajaya.
"It is an important mark in our regional co-operation to combat terrorism. It is an agreement signed by three countries... but it is also open for other Asean (Association of South-East Asian Nations) members to join in," said signatory Hassan Wirayuda, Indonesia's foreign minister.
Officials said that Thailand, and possibly Burma, may join the pact later.
Malaysian leader Mahathir, addressing a meeting in Kuala Lumpur of the Organisation of Islamic Conference (OIC) earlier, said the pact would be a useful weapon against regional terrorism groups.
"As you know, these people want to form a single Islamic state out of Malaysia, Indonesia and Philippines," Mr Mahathir said.
"What is their Islamic state I do not know. I have not seen them do anything except to do a lot of damage to people and carry out a lot of terrorist attacks," he said.
Terror crackdown
South-East Asia has detained several members of Jemaah Islamiah and Malaysian Mujahideen Group (KMM), both of which are believed to be fighting for a pan-Asian Islamic state.
But the US and local officials also allege that they have links with Osama Bin Laden's al-Qaeda.
US troops are also helping to train the Philippine armed forces to combat the rebel group Abu Sayyaf, which is currently holding an American couple hostage.
Abu Sayyaf say they are fighting for a Muslim state in the south of the country, although they have predominantly engaged in kidnap for ransom.
The Philippines has also made arrests in connection with a series of bombings in the southern city of General Santos two weeks ago.
The authorities say the are investigating possible links between the bombers and Bin Laden.