The UK wanted China to know the nuclear strength of the US could be unleashed if the Chinese attacked Hong Kong, previously secret papers show.
Hong Kong was considered vulnerable to a Chinese attack
In 1961 the UK felt nuclear retaliation was the only alternative to abandoning its colony if China attacked.
Officials wanted "to encourage" China to believe nuclear action against it would follow any hostile action, papers released by the National Archives show.
Britain handed Hong Kong back to China in 1997 having held it since 1842.
The letters, circulated to prime minister Harold Macmillan, were written between 1957 and 1961 when concern was growing about China's intentions.
Hong Kong was thought to be vulnerable, particularly as water and food supplies, from the mainland, could be cut off at any time.
The letters do not say who initially suggested the nuclear option, but they do show that British officials were keen for the Chinese to be aware of the threat while not giving the impression that Hong Kong was an American military outpost.
Defence minister Harold Watkinson wrote to the foreign secretary and prime minister, saying: "Our object is to encourage the Chinese to believe than an attack on Hong Kong would involve US nuclear retaliation."
Foreign Secretary Sir Alec Douglas-Home wrote a "top secret" letter to Mr Watkinson and Mr Macmillan in February 1961.
He wrote: "It must be fully obvious to the Americans that Hong Kong is indefensible by conventional means and that in the event of a Chinese attack, nuclear strikes against China would be the only alternative to complete abandonment of the colony.
"In these circumstances it is perhaps not so much formal staff talks with the Americans that we need so much as an informal exchange of views involving a discussion of the use of nuclear strikes.
"I need hardly say, however, that I agree entirely with your view that while we should encourage the Chinese to believe that an attack on Hong Kong would involve nuclear retaliation, we must avoid anything that would allow the Chinese to claim that the Colony is a military outpost of the United States."
Secret meetings between British and American officials were held in Hawaii with the possibility of further meetings on board a US naval carrier during its frequent visits to Hong Kong.
However, the idea stalled after Admiral Harry Felt, the commander of the US Pacific Fleet, responded cautiously when it was put to him.
Any further response from the US at the time was not included in the National Archives file.