Mr Rumsfeld - the most senior US official to visit since the Taleban were defeated
On a surprise visit to the Bagram airbase near the capital, Kabul, Mr Rumsfeld met US troops and held talks with Hamid Karzai, Afghanistan's new interim leader.
He told Mr Karzai the United States "coveted no territory ... We were here for the sole purpose of expelling terrorists from the country and establishing a government that would not harbour terrorism".
Mr Rumsfeld, who has been the driving force in the war to hunt down al-Qaeda leader Osama Bin Laden, said it was important to sit down face to face and talk with Afghan leaders.
"There's a good deal left to be done and I want to make
sure that we are all on the same wavelength," he said.
Mr Karzai said the Afghan people were thankful for America's help in battling terrorism and the Taleban.
"You came on board and provided help for us, provided the opportunity that we wanted," Reuters news agency quoted him as saying.
Commenting on proposals for an international peacekeeping force, Mr Rumsfeld said he expected it to number 5,000 at most.
A British-led military team is currently in Kabul to urge Afghanistan's new administration to accept a substantial international peacekeeping presence.
Afghanistan's interim Defence Minister Mohammad Fahim has insisted it should not exceed 1,000.
Danger area
Security surrounding Mr Rumsfeld's visit was intense.
He was warned before climbing down from a US military C-17 cargo jet not to step off concrete surfaces into surrounding fields because the area was peppered with landmines.
He did not leave the former Soviet airbase, which is an hour and a half's drive north of the capital.
Just before he landed, American fighter aircraft were seen flying low over the runway.
The American special envoy to Afghanistan James Dobbins is also in the country.
On Monday, he will reopen the US embassy in Kabul as a liaison office.
The embassy was closed in 1989 amid the chaos following the withdrawal of Soviet troops.