The striking asylum seekers claim they were deceived
The 32 men are angry at being detained in jail while their applications to remain in this country are being determined.
The men began the action on Wednesday, claiming they were not told they were bound for the jail before deportation later this year.
The Home Office said the policy of housing asylum seekers in prison is a temporary measure.
A spokesperson for the Prison Service said it was Cardiff Prison's policy that after 72 hours, those refusing food must undergo "physical and psychological testing" by health care staff.
The asylum seekers held in the prison have been declining to eat jail food since Wednesday morning.
However, the spokesperson denied the asylum seekers were on a hunger strike, saying they were on "food refusal".
She added that the detainees who originate from Kosovo, Sudan, Kashmir and Afghanistan are refusing food brought to them but have access to food from the visitors' canteen.
It is not known if they have eaten anything from there.
Oxfam has expressed sympathy for the immigrants while the Immigration Service has called for more detention places to be made available for asylum seekers.
Detention policy
The men have been kept at the prison for around four months under a Home Office policy announced earlier this year to hold them while specialist detention centres are built.
They claimed they were deceived by immigration officials and say they were told they would be sent to Cardiff, but not the city's jail.
A pair of asylum seekers in Liverpool took similar action last week, but have now stopped.
The controversy of holding would-be immigrants gathered pace in May it was revealed that some were taken to hospital for routine tests in handcuffs.
A demonstration outside the prison just a few weeks later drew a range of groups opposed to the policy.
Welsh Assembly Finance Minister Edwina Hart said it was not appropriate.
Cardiff North MP Julie Morgan visited the prison in July and on her return, she told the Commons the policy of pre-deportation penal housing was "deeply inhumane and inappropriate."
The government had pledged no immigrants would be in prisons after Christmas.