Lawyers, security services and police were "deeply reluctant" to act against radical cleric Abu Hamza al-Masri before 2004, David Blunkett has said.
Abu Hamza was an imam at Finsbury Park mosque in London
Mr Blunkett, home secretary from 2001 to 2004, wrote in his Sun newspaper column: "Some people did not want to believe how serious it all was."
Abu Hamza, 47, from London, was jailed on Tuesday for seven years for inciting murder and racial hatred.
Prosecutors said there was insufficient evidence to charge him before 2004.
The preacher was well known to the authorities for years before his arrest.
Mr Blunkett wrote in the Sun: "There was a deep reluctance to act on the information coming out of Abu Hamza's own mouth."
He added: "So much for those in the police, security services and Crown Prosecution Service who told me when I was home secretary that I was exaggerating the threat."
The CPS confirmed that it twice received files on Abu Hamza before 2004.
Officials concluded on both occasions "there was clearly insufficient evidence for a prosecution".
The cleric, who was imam at Finsbury Park mosque, was found guilty of 11 out of the 15 charges he faced at London's Old Bailey.