One of the aid workers has told the BBC by satellite phone that the group has been threatened but that so far they had been treated relatively well by their captors.
The LRA has been fighting the government for 16 years and says it is fighting to rule Uganda according to the Biblical Ten Commandments.
The LRA say they carried out the attack on a refugee camp in Northern Uganda to avenge the killings of LRA supporters in Sudan.
Sat phone
In a statement issued on Thursday, LRA claimed the Ugandan army carried out the attack in collaboration with the Sudanese government and rebels of the Sudan People's Liberation Army (SPLA).
It said International Rescue Committee (IRC) workers at the Achol-pii refugee camp had been taken away because the United Nations had not protested at the death of civilian refugees killed by the Ugandan army in Sudan.
Speaking to BBC's Focus On Africa programme, one of the hostages said they were exhausted and feared for their lives.
"We've been walking since Monday and some of us have fallen ill," Paul Kalama said on a satellite phone provided by the LRA.
"It's been a long distance and I don't know where we are. But we are still within Uganda."
'Operation Iron Fist'
The attack on the refugee camp left more than 50 people dead.
It came only three days after the Ugandan President, Yoweri Museveni, moved to the north of the country to take charge of operations against the rebels.
It was later followed by another rebel attack on Tuesday - in which at least 15 rebels were killed when they took on two army units in northern Uganda.
The camp at Achol-Pii is now safe, according to the army, but the United Nations refugee agency has been moving the refugees to the central area of Masindi, which it believes to be more secure.
Earlier this year, the Ugandan army was given permission to enter Sudanese territory to wipe out the rebels.
However "Operation Iron Fist" has driven them further into Uganda, where they have intensified their operations.
President Museveni has given permission for religious leaders to try and negotiate with the rebel group.