The four known survivors, named as Jesus Gica, Oliver Amorin, Jessie Buot and Renato Lanorio, have been telling local media about their experiences.
Mr Lanoria said he had been on the top deck when a crew member ordered passengers to put on life jackets at about 1130 on Saturday.
He said that about 30 minutes later, the ship tilted and elderly people and children slipped on the rain-soaked deck.
Jesus Gica said he was worried that many people had been trapped below deck.
"There were many of us who jumped overboard, but we were separated because of the big waves," he said.
"The others were also able to board the life rafts, but it was useless because the strong winds flipped them over."
The four managed to swim to the shore of Sibuyan, in Romblon province, where they were found by villagers, according to government officials.
Why did you allow it to sail and why was there no ample warning? I want answers
Gloria Arroyo Philippine President,
Battling huge waves, a rescue ship only managed to reach the ferry more than 24 hours after it lost radio contact.
Four bodies - believed to be from the ferry - were washed on shore on Sunday in the town of San Fernando in Sibyan.
The town's mayor said the upturned ferry could be seen from shore with its bow jutting out of the water.
As rescue efforts are due to resume on Monday, officials are now checking reports that a number of survivors might have made it to a small island.
The 23,824-tonne ferry has a passenger capacity of 1,992.
There is some confusion over the number of people on board. The ship's list states it was carrying 626 passengers and 121 crew. Local reports put the figure as high as 820, including more than 50 children.
Anxious relatives have gathered at the ferry's office in the capital, Manila
The Philippine President, Gloria Arroyo, has demanded an explanation as to why the ferry was allowed to leave port on Friday despite warnings that a typhoon was about to hit.
"Why did you allow it to sail and why was there no ample warning? I want answers," she told civil defence and coast guard authorities during a briefing on the accident.
Worried relatives have been gathering at the offices of Sulpicio Lines.
One relative, Lina Salinas, said she had waved off her sister on the 22-hour voyage on Friday.
"We knew it was signal number 1 [the first stage of typhoon alerts] at the time, but we were not really worried because it was not raining here at all," Ms Salinas was quoted by AFP news agency as saying.
"But the ship should not have been allowed to leave," she said.
A spokesman for the company said the priority was the search and rescue operation and that there would be no talk of responsibility at present.
Typhoon Fengshen swept across the central Philippines on Saturday.
Emergency workers rescue those trapped by flood waters
In Iloilo, a central province where 101 people were reported dead, chief administrator Manuel Mejorada said most of the victims had been killed as a result of flooding caused by the storm.
"Yesterday there was hardly any villages and communities which were not under water, and where the people were literally stranded on rooftops with no food and no water, and chilling in the cold.
"Right now the floodwaters have receded, and we are shifting our efforts to bringing food, water, medicine and clothing," he told the BBC.
At its peak, the storm was packing gusts of up to 93mph (150 km/h). It changed course on Sunday, hitting the capital Manila with heavy rainfall at dawn.
Thousands of people across the country have been evacuated from their homes. Many roads are blocked and there have been widespread power cuts.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.