Burundi's last remaining rebel group, the National Liberation Forces (FNL), has attacked the north of the capital, killing one civilian, the army said.
President Nkurunziza hopes to bring his rivals into the peace process
This follows their rejection of an offer of peace talks with the new government.
UN leaders have called on the FNL to join the peace process that has ended the war between the main Hutu rebels and an army led by the Tutsi minority.
A new constitution guarantees a balance of power between the ethnic groups.
On the eve of his inauguration last month, President Pierre Nkurunziza, who led the larger FDD Hutu rebel movement in the war, vowed to engage the FNL in peace talks.
Some 300,000 people were killed in the civil war, which was sparked in 1993 by the assassination of Burundi's first Hutu head of state and democratically elected president, Melchior Ndadaye.
Late on Tuesday night explosions in northern Bujumbura could be heard throughout Burundi's capital.
"They launched some rockets and hand grenades. One civilian was killed by a rebel shell fired near some homes in the area," army spokesman Adolphe Manirakiza told reporters.
He said soldiers were pursuing the rebels in the hills surrounding Bujumbura.
Correspondents say the attack reiterates the rebel group's determination not to talk peace with Mr Nkurunziza.
Peace in Burundi is seen as essential to a wider settlement in the troubled Great Lakes region, but analysts say that the country's peace prospects depend on Mr Nkurunziza's success in bringing the rival FNL group into government.
In New York, nine African leaders and UN head Kofi Annan have donors to make good their support for the new administration in Burundi.
Donors have pledged $1bn to rebuild the war-torn country, but so far have provided only 20%.