Pakistani cricket was already suffering from serious security concerns.
Pakistan invited Sri Lanka to tour only after India's cricket team pulled out of a scheduled cricket tour on security grounds, following the Mumbai attacks.
And International Cricket Council president David Morgan told the BBC it would be "very difficult for international cricket to be hosted in Pakistan for quite some time to come".
Reports suggest 10 to 12 gunmen ambushed the team coach and its accompanying police detail on the Liberty Square roundabout in the heart of Lahore, as the convoy was on the way to the Gaddafi stadium for a Test match.
The BBC's Barbara Plett, in Islamabad, says accounts suggest the attack was sophisticated in nature, with one group of gunmen firing a rocket-propelled grenade in order to create a diversion, while others then approached, firing guns on the convoy.
She says the gunmen - two of whom were shown in TV pictures carrying backpacks - seemed to be well-trained.
Sept 08: 54 die in an attack on the Marriott hotel in Islamabad
June 08: Six killed in car bomb attack near Danish embassy in Islamabad
Dec 07: Former PM Benazir Bhutto assassinated along with 20 others at a Rawalpindi rally
March 06: Suicide car bombing kills US diplomat in Karachi
June 02: 12 killed in car bomb attack outside US consulate in Karachi
May 02: 11 French engineers and three Pakistanis killed in an attack on Karachi Sheraton hotel
Sri Lanka's assistant coach Paul Farbrace, who is British, told the BBC: "There was a lot of shouting and people hitting the floor and when I got to the floor I realised that the blood that I could see was coming from me - luckily superficial wounds."
The Sri Lankan wicketkeeper Kumar Sangakkara told Australia's ABC radio that he and his team mates probably owed their lives to their bus driver.
"We had an amazing driver who just kept driving the bus straight through all of that to the ground and that's probably what saved us."
Officials in Lahore said two members of the Sri Lankan team, Thilan Samaraweera and Tharanga Paranavitana, were sent to hospital.
The Pakistani politician Imran Khan, a former captain of the country's cricket team, said the Sri Lankan visitors had been given inadequate protection.
"This was one of the worst security failures in Pakistan," he said. "The Pakistan government guaranteed the Sri Lankan cricket team that they would provide them security."
None of the attackers was killed or apprehended at the scene. Grenades and rocket launchers were found there, police said, and a manhunt is under way.
Our correspondent says security forces will be investigating any connections to al-Qaeda and Taleban militants as well as Kashmiri jihadi groups.
A Pakistani air force helicopter has now airlifted uninjured members of the Sri Lankan team out of the Gaddafi Stadium.
Sri Lankan President Mahinda Rajapakse condemned the "cowardly terrorist attack" and ordered the players' evacuation back to Sri Lanka. A plane is being chartered for this purpose. Mr Rajapakse is cutting short a visit to Nepal.
Thilan Samaraweera (L) and Tharanga Paranavitana went to hospital
Pakistani President Asif Ali Zardari also strongly condemned the attack, and ordered an immediate investigation "so that the perpetrators are identified and their motives exposed", said a statement from his office.
Meanwhile, a spokesperson for India's foreign ministry offered sympathy and urged Pakistan to take "prompt, meaningful and decisive steps to dismantle the terrorist infrastructure once and for all".
But a Pakistani minister, Sardar Nabil Ahmed Gabol, reportedly told private Geo TV that evidence suggested the attackers came across the border from India.
He said the assault came in reaction to the Mumbai attacks, and was a "declaration of open war on Pakistan by India".
The third day of play in the Second Test was scheduled to begin, but officials said the match has now been cancelled.
Pakistan is engaged in a bloody struggle against Islamist insurgents who have staged high-profile attacks on civilian targets before.
Meanwhile, Sri Lankan authorities are waging their own domestic military campaign against Tamil Tiger separatist rebels.
India and Australia have pulled out of cricket tours in Pakistan in the recent past citing security concerns.
The sport's world governing body, the International Cricket Council, last month decided not to hold the 2009 Champions Trophy in Pakistan due to safety worries.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.