The shooting dead of three unarmed IRA members was a hugely controversial event.
The shooting dead of an IRA unit in Gibraltar in 1988 was a mistake, according to the official history of MI5.
Writer Professor Christopher Andrew was given access to the security service's records to prepare the centenary history of the service.
The shooting dead of three IRA members by an SAS team was a hugely controversial event.
But Prof Andrew said the files showed no evidence of a shoot-to-kill policy.
He said MI5 discovered the IRA plot to attack the garrison in Gibraltar at a "pretty early stage".
The book includes surveillance pictures of Siobhan O'Hanlon reconnoitring in Gibraltar and a map of her movements.
However, she spotted Spanish surveillance and returned to Ireland, Mr Andrew said.
Three other IRA members, Daniel McCann, 30, Sean Savage, 24, and Mairead Farrell, 31, were being watched by intelligence officers and were shot dead.
"I am entirely satisfied that even though they were not carrying any means of causing an explosion, that at the time they were shot British surveillance concluded they were - so it was a mistake," he said.
"They were in a position in which it was legitimate to think they were about to cause an explosion, but there was no shoot-to-kill policy in Gibraltar, of that I feel confident."
Professor Christopher Andrew insisted he was given complete access to MI5's files for the book - Defence Of The Realm.
He alleges British intelligence could barely cope with the number of Soviet spies in the UK during the Cold War, and that MI5 did not get to grips with Soviet espionage in Britain until the early 1970s.
He had access to all 400,000 files created by MI5 since it was founded in 1909, but the agency did limit what he could publish.