Cadbury's new owner, Kraft, says it plans to close the company's Somerdale factory in Keynsham, near Bristol.
Just last week it said it would keep it open. The shutdown would mean the loss of 400 jobs.
Cadbury had earmarked the plant for closure but Kraft's takeover had raised hopes of a reprieve.
The move comes just weeks after Cadbury's chairman told the BBC that job losses were an "inevitability" after the takeover by the US giant.
In a statement, Kraft said it would stick to the plans put in place by Cadbury in 2007 to close the plant and transfer the work to Poland.
Kraft Foods said that after what it called "extensive talks" with senior management at Cadbury, it found that Cadbury's plans to close Somerdale were so far advanced that it would be "unrealistic" to reverse them.
It added that it had become clear that the investment required to reverse the closure programme "would be so significant that alternative plans were not viable".
The company pointed out that Cadbury had already invested more than £100m in building new production facilities in Poland and the majority of the lines have, or are about to be, transferred by the middle of this year.
Irene Rosenfeld, chairman and chief executive of Kraft Foods, said: "It became clear that it is unrealistic to reverse the closure programme, despite our original intent to do so. While this is a difficult decision, we have moved quickly to end any further uncertainty."
She said the planned £30m investment plans for the Bournville site remained in place.
The company has said that it will honour Cadbury's previous undertakings to Somerdale employees concerning the terms and conditions of the closure and a commitment to rebuild the Fry Club on the Somerdale site.
Staff at the Somerdale factory in Keynsham were told the news at a meeting on Tuesday afternoon.
One worker told the BBC a statement by Kraft that the factory might stay open had been "a big fat lie".
He said: "Apparently the plans to move to Poland were too far gone to save the factory so unfortunately, we're still up for the chop."
Keynsham resident Amoree Radford, of the Save Cadbury's Campaign said the company [Kraft] was "absolutely despicable".
She said: "We were taken in by Kraft. I really thought they were sincere."
The union Unite reacted angrily to the decision and said it sent the "worst possible message" to 6,000 other Cadbury workers in the UK and Ireland.
Jennie Formby, Unite national officer for the food and drinks sector, said the US firm had "deliberately misled many hundreds of decent men and women".
Business Secretary Lord Mandelson, who met Ms Rosenfeld after the takeover was sealed, said: "This will confirm the fears of those who felt the takeover would result in job losses. Kraft gave me no indication of this announcement when we met last week."
He added: "It is for the company now to prove the worth of their other statements about investing in the UK."
The Somerdale plant is in Keynsham, near Bristol and Bath, and was originally built by the Fry family. It merged with Cadbury Brothers in 1919.
Products made at Somerdale include Fry's Chocolate Cream, the Double Decker, Dairy Milk, Chocolate Buttons, Creme Eggs and Mini Eggs, Cadbury's Fudge, Chomp and the Crunchie.