Mr Brown and Muammar Gaddafi met at the G8 talks in July
Downing Street has denied any U-turn in its support for IRA victims' families seeking compensation from Libya.
Libya, which supplied the IRA with explosives, has paid compensation to US victims of Libya-backed terrorism.
But last year Gordon Brown wrote to the UK victims' lawyer saying it would not be "appropriate" for the government to push Libya to compensate them.
On Sunday he said a government unit was being set up to help them. The Tories called that a "partial U-turn".
In the latest row to follow the release of the Lockerbie bomber, Abdelbaset Ali al-Megrahi, by the Scottish government, the Sunday Times published ministers' letters to campaigners for IRA victims, which were sent last November.
In them Mr Brown wrote that the government did not "consider it appropriate to enter into a bilateral discussion with Libya on this matter".
But on Sunday he said he was setting up a dedicated Foreign Office team to assist the IRA families' victims.
Gordon Brown's decision to leave this to the courts and not to make this an issue between governments looks hard to justify
The prime minister's spokesman said that was "entirely consistent with the approach taken so far" and that Foreign Office officials in Libya would "facilitate not negotiate".
Speaking on BBC Radio 4's Today programme, Schools Secretary Ed Balls said: "It's not a U-turn because, as I understand it, what Gordon Brown said last night is the same as he said to families [in the past]."
He added that it had not been possible to pursue a "government-led case for compensation" as Britain had already agreed with Libya to "recognise what had happened in the past, put it behind us and move on".
There had been an attempt to get a government agreement in 2004 for compensation for both the IRA and Lockerbie victims, he said.
"That couldn't be negotiated and the judgement the government made and still makes, is that to attempt to do so now, would both not succeed and sour and damage a relationship which is not about economics or oil but is actually about working together to make sure British people are safe from international terrorism."
But shadow foreign secretary William Hague said: "It is a very partial U-turn. It is not exactly clear what is going to happen, what support is going to be provided for the families looking for compensation of terrorism in Northern Ireland.
Alan McBride's wife and father-in-law were killed in the Shankill bombing in 1993
"I think it is time he [Gordon Brown] changed his position that it's not appropriate for the British government to raise this directly with the Libyan government."
He added it would be "very strange" for the British government to help the families involved but for ministers, in discussions with their Libyan counterparts, not to mention it.
Edward Davey, the Liberal Democrats' foreign affairs spokesman, added: "The prime minister is still failing to explain why he rejected the idea of the British government negotiating directly for compensation for the IRA victims' families.
"Everyone will wish the families' legal teams good luck, but Gordon Brown's decision to leave this to the courts and not to make this an issue between governments looks hard to justify."
Meanwhile Colonel Gaddafi's son Saif al-Islam Gaddafi said any claims for compensation based on Libya's supply of explosives to the IRA would be a matter for the courts.
He told Sky News: "They have their lawyers. We have our lawyers."
And when asked if his answer to the compensation demand would be "no" in the first instance, he replied: "Of course."
The entire fabric of the extraordinarily difficult relationship between Libya and the UK is becoming unravelled
But lawyer Jason McCue, who represents some of the victims, welcomed the response as a sign of "engagement" adding: "We always expected this to go to the courts and now it means there will be a process to getting compensation."
Democratic Unionist MP Nigel Dodds, who has campaigned for compensation for UK victims, said the Foreign Office unit was a "belated but welcome" move.
"What is polluting the atmosphere politically at the moment is the fact that there isn't closure for the victims, the UK victims," he said.
"This isn't just about Northern Ireland victims but victims of IRA terrorism right across the United Kingdom who see no compensation or justice to them as far as Libya is concerned but who look at their US [counterparts] who were victims who have been compensated."
Sinn Fein leader Gerry Adams said that while he supported compensation for all victims there should be no "hierarchy".
"Some victims were killed directly by British agents, others by their surrogates. British state forces have also killed or injured many citizens," he added.
"Republicans are not surprised by the hypocritical stance of successive British governments on this issue."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.