West Somerset Council has been given a zero rating for its housing benefits service, putting the authority into the bottom 25% for England.
The Audit Commission report covering 2007/8 says 3,500 residents have faced "unacceptable delays", with claims taking 65 days to process.
It also says the service cannot demonstrate it is value for money.
The council says it has addressed the criticism and has cut the application time to 11 days.
Leader of West Somerset Council Keith Ross said: "The figures relate to two years ago.
"There had been a lot of redundancies in the benefits department which slowed down the processing time, but this last year we've improved to processing claims within 11 days."
The Audit Commission inspection team accepted the council has made a number of improvements in the last year, such as investing in new information technology, better working practices and improvements in the continuity of service delivery.
Sophie Trim, Audit Commission senior manager said: "The benefits service at West Somerset Council is now showing signs of improvement following changes in both working practices and technology.
"There is a strong commitment from councillors and staff to deliver improvements and these improvements have had an impact on customers.
"People now receive their benefit quicker and have better access through telephone, website and frontline services."
However the report says further work is still needed to improve the accessibility to benefits for customers and to ensure service standards and performance targets are relevant to customers' needs.
Performance monitoring and management also need to be further improved, and the service needs to be more proactive in identifying and deterring levels of benefit fraud.