Labour MP Ann Clwyd has attacked fellow party members who "continually snipe" at its leadership.
Ms Clwyd is due to meet members of the new Iraqi government
Speaking from Baghdad, Tony Blair's special envoy to Iraq added speculation about his departure from Downing Street had given people abroad the wrong idea.
"A small number of individuals are giving the idea there is a big rift in the Labour party," she told BBC News.
If members wanted Labour's "very strong programme" to succeed they should unite to give it every support, she added.
"I talk to more people than most in my own party - and they are very angry.
"Very loyal supporters of the Labour Party say, 'What on earth is happening?
"'We elected this prime minister. We elected this government. Why don't they get on with it. And why don't these other people just go away.'"
On Wednesday an ICM poll for the Guardian newspaper indicated the Tories had pulled ahead on key issues like education, health and law and order, where they trailed Labour at the last election.
The poll put the party on 38%, its highest rating for 13 years, with Labour on 34% and the Lib Dems on 20%.
But it also suggested Labour's support would fall if Gordon Brown was leader.
Asked how they might vote in an election with Mr Brown, the man widely expected to be the next Labour leader, pitted against Tory leader David Cameron and Lib Dem leader Sir Menzies Campbell, 40% backed the Conservatives, 31% Labour and 19% the Lib Dems.
But 46% of those surveyed thought Mr Brown was more trustworthy than Mr Blair, whom 26% said they trusted more.
Mr Blair had a narrow lead over the chancellor in who was seen as having the wider appeal - and that lead was extended when only Labour voters were polled.
ICM interviewed a random sample of 1,001 adults by telephone between 19 May and 21 May. The results were weighted to reflect the population.
During her four-day visit to Iraq, Ms Clwyd is due to meet members of the new Iraqi government and US and UK military chiefs, as well as attending Saddam Hussein's trial.