Members of the council's executive board approved on Tuesday a series of plans which include an extension with a new reception and shop.
A dedicated space for the collections of the National Museum and Library of Wales would also be included.
The council now plans to apply to the Heritage Lottery Fund for a £1m grant towards the total costs.
Most Popular Now | 69,743 people are reading stories on the site right now.