Mr Brown started his summer holidays with a visit to a country park
Local residents afraid that the government is neglecting the east coast have threatened to cloud Prime Minister Gordon Brown's summer holiday.
Mr Brown, his wife Sarah and sons John, four, and Fraser, two, are taking a break near Southwold in Suffolk.
But locals said the choice was ironic because Defra may allow parts of the nearby coast to flood as part of long-term sea defence plans.
Protesters will stage a demonstration in Southwold on Wednesday.
They said they believed the government should spend more cash on protecting the Suffolk coast.
'Fantastic, timeless place'
"Isn't it ironic that Gordon should choose to come to a place that his government is abandoning to the sea," said a spokesman for the Blyth Estuary Group.
"Southwold is a fantastic, timeless place that hundreds of thousands of people love to come to every year. We know Gordon and his family will have a fabulous time.
"But then we need him to go back to Westminster and tell Defra to reverse its policy of retreat to save wonderful parts of our coast like this before it's too late.
"If Gordon Brown has his way there will be nowhere for his two boys to get their buckets and spades from and nowhere to buy an ice cream because it will all be washed away."
Mr Brown has said nothing is finalised and he understands locals' concerns.
He said the government was committing hundreds of millions of pounds to combating coastal erosion as global warming causes sea levels to rise.
Playing on beach
"We have got to respond to people's concerns," he said, after arriving in East Anglia yesterday.
"All these decisions have got to be made on the basis of different areas' needs."
The Browns are staying in a country house which costs £4,500 a week to rent and is rated by travel agents as one of England's finest country retreats.
Mr Brown and his wife began their break by visiting a country park near Norwich on Saturday.
They are expected to spend about two weeks in the area. Mr Brown said he was looking forward to playing on the beach with his children and reading.