The Keynsham plant will close by 2011, Kraft said
Cadbury workers have been warned of up to 150 redundancies, following the takeover by US firm Kraft.
The job cuts will mainly affect the head office in Uxbridge, west London, although some jobs in Bournville, Birmingham, may go too.
A company spokesman said the cuts were "as a result of the Kraft acquisition" and Cadbury being delisted from the stock exchange next Monday.
Unite union said a 90-day redundancy consultation started on Monday.
A spokesman for Cadbury said: "It could be up to 150, it could be less as Kraft has said it wants to identify opportunities to retain skilled Cadbury people."
The confectioner's shareholders voted to accept the £11.5bn ($18.9bn) Kraft takeover bid on 2 February.
Cadbury employs 4,500 people in the UK, of whom 2,500 are based at the factory in Bournville.
Another 400 staff are based at the head office. Cadbury also has plants in Marlbrook, Herefordshire, and Keynsham, Somerset.
Kraft announced in February it would close the Keynsham factory by 2011 and transfer the work to Poland.