Sri Lanka's oil minister AHM Fowzie, who was at the site, told Reuters news agency the bomber appeared to have targeted six ministers as they walked in a procession toward the mosque.
"This is an area where nobody expected a thing like this to happen," he said.
The ministers were attending a ceremony to mark the birthday of the Prophet Muhammad when a bomber on a bicycle set off the explosion behind them, officials told the BBC.
Post and Telecommunications Minister Mahinda Wijesekara was injured in the attack and is said to be in intensive care.
A helicopter has airlifted the critically wounded back to Colombo.
One witness, Ahamed Nafri, told Associated Press he was walking toward the mosque at the time of the blast.
Footage showed the moment of the attack
"I heard a huge sound and then I saw people had fallen everywhere. They were covered with blood and flesh and the wounded people were screaming."
The blast comes as Sri Lanka's military continues its operation to defeat the Tamil Tigers in the north-east.
Fears have been rising over the safety of civilians in rebel-held areas in recent weeks, with reports hundreds of deaths.
A senior health official told the BBC Tamil Service that over 1,000 casualties were brought to a temporary government hospital functioning in the rebel-held village of Pudumatalan hospital over the last 10 days.
Dr T Vartharaja said that 218 bodies were brought to the hospital in addition to 1,205 injured people since 1 March.
Dr Vartharaja said that 277 injured civilians were brought to the hospital on Tuesday alone.
"There were 66 children among the injured - and 45 people from those brought to the hospital earlier died today as a result of lack of medicine," he said.
He said sicknesses and diseases has been worse because of heavy rainfall over the last few days.
On Tuesday the military said it had restricted the rebels to an area of less than 50 sq km (19 square miles) but that heavy fighting was continuing.
Also on Tuesday, the government announced it had appointed the former deputy leader of the Tamil Tigers - known by his nom de guerre, Col Karuna - as a minister.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.