A spokesman for the Taleban, Zabiullah Mujahid, told agencies that three men had carried out the attack.
He said two fighters had shot at the guards and then fled as the third ran inside and detonated the bomb.
The Voice of Jihad website, quoting the Afghan Taleban, named the bomber as Naqibollah, from the province of Khost.
Many of the six-storey building's windows were blown out by the force of the blast and some windows in neighbouring buildings were also shattered.
The minister was not in the building at the time of the attack.
Afghan President Hamid Karzai, who is visiting Turkey, condemned the attack, saying it was "another attempt by the enemies of Afghanistan to undermine peace and security".
Health ministry officials said that of the 23 wounded, eight had already left hospital with minor injuries.
One witness said that the wounded included some children from a kindergarten on the compound.
The BBC's Milton Nkosi, in Kabul, described scenes of panic outside the ministry building, with emergency services rushing into action amid the wailing of sirens.
"Kabul is still a tense place, but people are saying that the security situation is still much better than it was a year ago," he said.
"It may seem grim at the moment, but things this time last year were a lot worse."
Bookmark with:
What are these?