Sir John Krebs, chairman of the FSA, said the survey highlighted concerns and promised that the FSA's targets over the next five years would be protecting people against food poisoning and BSE.
The agency, which has carried out a UK-wide survey of consumer attitudes, says official statistics massively under-estimate the number of cases.
Its own survey found that many consumers did not trust health messages from the government.
A quarter thought Department of Health information was fairly or very unreliable. Two-thirds are still worried about the threat of BSE and 43% are concerned about GM foods.
Three quarters of people living in the UK are worried about food safety, the study also reveals, a statistic blamed on two decades of food scares.
But despite their worries, only a third of people interviewed understood the best before and after labels on food and just over a half bothered to check them.
The FSA blames two decade of food scares for the lack of confidence in what we eat.
Shoppers said food poisoning and BSE were the big worries, but despite this only one in 20 households in the UK includes a vegetarian.
Less than half the 3,100 people surveyed were confident about current food safety, but nearly half did think that standards had improved last year.
The survey, compiled just six months after the formation of the FSA last year, found food poisoning was the highest in England with 14% suffering from it.
This dropped to 13% per cent in Scotland, with Wales saying they had 11% rates and Northern Ireland 10%.
But only one in five sufferers reported cases of food poisoning.
The elderly were found to suffer less than younger age groups.
20 years of crises
Only four in 10 people had a fridge thermometer. Virtually all of them said they knew the correct temperature to store food, but only 61% were correct.
Sir John Krebs, chairman of the FSA, said the survey highlighted concerns and promised that the FSA's targets over the next five years would be protecting people against food poisoning and BSE.
He said that the FSA must work hard to correct the poor image which followed two decades of food crisis after food crisis.
"It highlights consumer concerns on food poisoning and BSE. Both are priorities for the FSA over the next five years.
"The survey highlights that less than half of consumers are confident about current food safety measures," he said.
And he called on the whole food industry to bring in changes.
"It may be a reflection of two decades of food crises but it is an unacceptably low level. The responsibility for change rests with everyone involved in the food business.
"Almost half of consumers think that food safety has improved over the last year, so there are real improvements taking place," he said