Sir Alan Sugar has advised Gordon Brown to "kick out" members of the government failing to support his leadership.
The businessman said it was impossible to run a company or government "unless everybody is on side".
He told the BBC News Channel that it was easy to "blame the top man", but current tough economic conditions were not the fault of the prime minister.
His comments come as Skills Secretary John Denham has called for an end to speculation over the Labour leadership.
Mr Denham said such calls get in the way of the party putting its message across.
He has joined the party's deputy leader, Harriet Harman, and Chancellor Alistair Darling in making a new declaration of support for Gordon Brown in an article for Sunday's News of the World.
The chancellor says Mr Brown will lead Labour into the next election - and win the confidence of the country.
The past week has seen speculation about possible challenges to Mr Brown's leadership of the party.
Sir Alan Sugar on why he is backing Gordon Brown
Labour's shock defeat at the Glasgow East by-election on 24 July raised questions over Mr Brown's leadership.
Days later, a newspaper article about Labour's future by David Miliband that made no mention of the leader prompted speculation about the foreign secretary's ambitions.
Sir Alan, star of TV's The Apprentice, said Mr Brown should "sort his own people out" and then be allowed to get on with the job.
"You can't run a government and you can't run a company or anything like that unless everybody is on side," he said.
"If they are not on side he should kick them out and then what he should do is tell the rest of the world that he has been appointed to do a job for two years and let me get on with it and then at the end of that period of time - judge me then."
He is a very, very clever man and a man who is over everything and knows what is going on
Sir Alan Sugar
Sir Alan, who sits on the government's Business Council, added: "It's very easy for people to blame the top man when things are no good but what you have to look deeper at is what these problems are."
He said Tory leader David Cameron seemed "quite a nice fellow" and thought even he would concede the current economic difficulties were caused by outside influences.
"It's not just recently that I have backed Gordon Brown," he said.
"I've known him for a long time as chancellor, and I have got to know him quite well and out of the last four prime ministers that I have had the pleasure to have met I think he is a very, very clever man and a man who is over everything and knows what is going on.
"He may not come across as some kind of actor of some kind but he has got his hand on the pulse."
He added his comments on Mr Brown should be seen as a backing for a "good man" who was an "excellent" chancellor in the past rather than support for the Labour party, as the time to give such a declaration would be before a general election.
Sir Alan said he had donated money to the Conservatives at the time of Margaret Thatcher and to Labour before Tony Blair was elected in 1997.
He said there appeared to be little difference between Labour and Conservatives these days.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.