The army claims rebels fired on the civilians as they crossed the lagoon
The Red Cross says its staff in Sri Lanka are witnessing an "unimaginable humanitarian catastrophe" in the area where troops have trapped Tamil Tigers.
The agency says a ferry loaded with aid has been unable to reach the battered north-eastern coastal strip for three days because of fighting.
There are also reports that staff have quit the last hospital in the war zone.
A senior UN envoy is on his way to Sri Lanka to try "to help resolve the humanitarian situation", the UN says.
Vijay Nambiar, Secretary General Ban Ki-moon's chief of staff, is expected to arrive on Friday.
Last month Mr Nambiar met Sri Lankan President Mahinda Rajapaksa but failed to secure access to the war zone for humanitarian teams.
The UN says about 50,000 civilians are trapped in the war zone, although Colombo disputes this figure.
The government has rejected international calls to stop its offensive against the Tamil Tiger rebels, saying it would give them time to recover. Now that it has trapped the Tamil Tigers, it hopes to soon end the 25-year-old civil war.
The International Committee of the Red Cross (ICRC) said its chartered ferry, the Green Ocean, had been unable to deliver aid or evacuate the wounded for three days.
"Our staff are witnessing an unimaginable humanitarian catastrophe," said ICRC director of operations Pierre Krahenbuhl, based in Geneva.
"Despite high-level assurances, the lack of security on the ground means that our sea operations continue to be stalled, and this is unacceptable," he said.
"People are left to their own devices."
The ICRC says it requires security and unimpeded access to the area immediately.
In a statement, it said another aid ship, from the World Food Programme, was also waiting to deliver supplies to the war zone.
The BBC's Charles Haviland in Sri Lanka says there are also unverified reports that medical staff have abandoned the main hospital in the rebel-held area because of persistent shelling.
One report said that about 400 badly wounded patients had been left behind, along with more than 100 bodies awaiting burial.
The government and the rebels blame each other for civilian casualties
Dozens of civilians have been reported killed in artillery attacks on the facility in recent days.
Earlier on Thursday, a military spokesman told the BBC that unmanned aircraft had filmed more than 2,000 people wading across the lagoon which borders the fighting zone on the non-seaward side.
Brig Udaya Nanayakkara said the civilians had braved rebel fire to reach government-held areas.
"There is a large number of people crossing, and the (rebels) fired at them. Four people were killed, 14 were wounded," he said.
The Sri Lankan army's version of events cannot be independently verified and there has been no comment from the rebels.
The authorities and the rebels blame each other for civilian deaths.
As the fighting continued, Britain said on Thursday that it supported an early inquiry into whether war crimes have been committed in Sri Lanka.
"We would support an early investigation into all incidents that may have resulted in civilian casualties," said junior foreign minister Bill Rammell.
He said the UN's estimate of more than 6,500 civilian deaths since January was - if accurate - "truly shocking and appalling".
The UK-based charity Save the Children said on Thursday that a growing number of children were becoming separated from their families as they fled the war zone and entered government-controlled camps.
"The camps are chaotic," said spokesman Branko Golubovic.
"These children are coming out of combat areas where they have been severely traumatised only to find themselves in yet another harsh environment in the camps."
Nearly 200,000 civilians are believed to be living in the government's overcrowded displacement camps.
Have you been affected by the fighting? Are you in a displacement camp? Have you seen people trying to flee the war zone? Send us your stories using the form below.
The BBC may edit your comments and not all emails will be published. Your comments may be published on any BBC media worldwide.