Workers have protested against the closure plans.
Plans by Cadbury to sponsor the 2012 Olympic Games have angered workers at one of its plants facing closure.
The confectionery company wants to shut its factory in Keynsham near Bristol by 2010 with the loss of about 500 jobs.
Campaigners trying to save the site said they were furious the firm was planning to spend more than £20m in sponsorship of the games in London.
Cadbury said the closure decision was tough, but sponsorship was part of its continuing marketing activities.
Amoree Radford, spokeswoman for Save Our Keynsham Cadbury's, said: "We're really, really cross.
"That [money] could well be spent on those wonderful people that have struggled so hard to keep that factory going.
"Cadbury will tell you the £20m to £40m, which will enable them to be the sole supplier of chocolate, is coming out of their little pot for sponsorship."
One worker, who did not want to be named, added: "The money Cadbury's are using to advertise for the Olympics could keep Somerdale [the Keynsham site] running for more than three years."
A statement from the company said: "We took the difficult decision to close Keynsham in order to secure the long term success of the company.
"We are keeping the factory open until 2010 and we have sought to support the affected staff in the way that would be expected from a company like Cadbury through advice, outplacement support and good redundancy packages.
"Clearly Cadbury will continue making and selling confectionery in the UK and our support for 2012 is be part of our continuing marketing and sponsorship activities."
The company is planning to move production to Poland.
Together with the closure of the Keynsham site, a further 200 jobs are due to go at Cadbury's Bournville plant in Birmingham, which is being restructured.