Eighty positions out of 480 will be lost at the Town Hall over three years
Great Yarmouth Borough Council is having to cut nearly one in five of its workforce during the next three years.
Senior positions will be targeted first and the council is hoping to save £500,000 from its management budget.
The council employs 480 people and 80 jobs will go. Forty vacancies have already been frozen so that positions can been filled internally.
Total savings of £2.6m have to be made this year by Great Yarmouth borough council.
It's being claimed that government grants have not kept up with inflation and the council has had to pay out more than expected on concessionary bus fares.
Chief executive Richard Packham said: "There is a hard message which I think our staff understand. We will be operating with a lot fewer people in the future."
"The key thing is to avoid compulsory redundancies and redeploy people so we make the very best use of the staff we have," he added.
Great Yarmouth Borough Council is also warning services such as economic development and cultural services that they could have smaller budgets in the future and Mr Packham thinks the council needs to assess everything it does to make sure it adds value.
"I see a smaller organisation delivering a smaller range of services or certainly tackling fewer major projects, particularly around regeneration.
"The key thing is for us to retain a focus on our customers so we can continue to serve the community well."
Council leader Barry Coleman says that could mean waiting longer to get a third river crossing.
"It's always been realised that it's a long-term project. The Breydon Bridge was a 10 to 15-year project - in fact, if you look back to the war records I think they were talking about it then, so I think we're talking five to eight years for the third river crossing, but it is vital," he said.
During his recent visit to Great Yarmouth, the Prime Minister denied the government was leaving local authorities with a lack of money.
"We have tried to do our best in funding them for what they have to do. There's been enormous advances made in schools," said Mr Brown, defending his record.
"We've introduced neighbourhood policing in every community of the country. Now we have provided the funds for that to be maintained and it's up to local authorities to make sure that happens."
Great Yarmouth Borough Council also wants to try and work more efficiently by sharing certain services with other local authorities.
Talks are taking place for Norfolk County Council to provide Great Yarmouth's IT services and that could save £1.1m over the next five years, which is 20% of Great Yarmouth Borough Council's current IT budget.