"We know that they are there. We know that they are in Lebanon, working closely with Hezbollah. We know that they are in the region," he said.
The Palestinians' chief negotiator, Saeb Erakat, dismissed Mr Sharon's announcement as an attempt to discredit them.
The issue of al-Qaeda has been the subject of intense speculation in Israel since the attacks on Israeli citizens in Kenya one week ago, which were followed by a claim purporting to be from the organisation.
Palestinian anger
"For some time we have received reports of al-Qaeda members entering [Palestinian areas inside] Israel," the Israeli premier told journalists in Tel Aviv.
"The reports at the moment are of the presence of a few in the Gaza Strip."
He added that Israel was taking "all steps" to protect itself from an attack.
Speaking for the Palestinian Authority, Mr Erakat said that Mr Sharon's allegation that al-Qaeda was operating in Gaza was "part and parcel of Sharon's attempts to use the 11 September terrorist events for his own benefit".
He was referring to the airliner attacks on America last year which al-Qaeda is widely believed to have organised.
"The world should not pay attention to his words and accusations," said the Palestinian negotiator.
Lebanon has not responded officially to Mr Sharon's remark that al-Qaeda is also operating there.
Only on Wednesday, Israel announced that it had questioned a suspected member of al-Qaeda believed to have been distributing funds to Palestinian militants on the organisation's behalf.
The BBC's Jeremy Cooke says that the Israeli Government is keen to demonstrate to the Americans that they are facing a common enemy.
Football row
Meanwhile, the Israeli Football Association (IFA) has angrily rejected media reports that al-Qaeda planned to attack the national team in October at a match it played in Malta.
The IFA said the report was an attempt to damage Israel's ability to play football abroad.
"We were surprised to discover the publication of the apparent plot to attack the national team... There has not been any confirmation form any security source," it said in a statement.
"The publication of unconfirmed reports could cause great harm to Israeli soccer at the international level, damage which will be very difficult to put right in the future," the football association said.