The Welsh Assembly Government has been warned it faces opposition if it tries to absorb Wales' major arts bodies.
The National Library was set up by Royal Charter in 1907
The National Library of Wales, National Museums and Galleries of Wales and Welsh Arts Council could become part of the assembly government as it shakes up the quangos.
But Peter Finch of arts group Welsh Academy said change would be damaging.
The Aberystwyth-based National Library has also said it will oppose change to its status.
The museums and the library believe there are potential legal obstacles, with concerns over their access to lottery funding as they could lose their charitable status.
The Welsh Academy, which promotes writing in Wales, has also written to First Minister Rhodri Morgan, asking him to reconsider on the arts council.
Library officials have warned Culture Minister Alun Pugh any attempts to bring it under governmental control would threaten its effectiveness.
They also said the library's charitable status could be eroded or destroyed by any change.
The concerns were made in a submission sent to Mr Pugh and seen by BBC Wales.
According to the library, change would be costly, time-consuming, and could affect the professionalism of its staff. It also warned that any move would probably need parliamentary legislation.
The library, set up by Royal Charter in 1907, has the right to receive a free copy of every book published in the United Kingdom.
It has thousands of manuscripts and archives, pictures and photographs, maps, sound recordings and moving images, available for consultation.
A number of cultural bodies are worried about the situation.
"The library is a statutory body, an organisation with a good record of looking after our literary and archival treasures," said the chair of the Welsh Academy, author Harri Pritchard Jones.
"The assembly does not have a track record.
"But more importantly, I am against totalitarianism where the government takes care of things they know nothing about.