Lord Jones made his thoughts known at a private meeting
Trade Minister Lord Jones has given his "100% support" to Gordon Brown and said there was "nothing new" in reports he would quit before the next election.
The ex-CBI chief, who is not a Labour member, was one of the "government of all the talents" appointed by Mr Brown.
He "never claimed to be a political animal", adding: "I am a supporter of... and believe in what he is doing. My plan now is to get on with the job."
But the Tories say the claims shows the peer has "no confidence" in Mr Brown.
According to The Times, Lord (Digby) Jones told a private meeting with businessmen and lobbyists in January that he wanted to stand down in order to avoid questions on whether he supported the prime minister.
The report comes as Mr Brown held talks with leading City figures to discuss the global credit crunch crisis, and critics speculate over the state of his leadership.
Lord Jones, speaking at a trade conference in London, said there was nothing new in the Times newspaper report.
"I've never claimed to be a political animal and as you know I believe trade and investment should transcend the factionalism of party politics," he said.
"But one thing I'm absolutely clear on - I am 100% committed to this job and to the prime minister.
"I am a supporter of Gordon Brown and I believe in what he is doing.
"I will continue to give him personally my full support and I am delighted to be doing this important job for my country.
"My plan now, as always, is to get on with the job."
Mr Brown said Lord Jones was "doing a good job" and was "staying on as a trade minister".
"He came in because he supported me and will support me," he said.
A spokeswoman for UK Trade and Investment (UKTI) said Lord Jones' intentions should not come as a surprise.
"It is no secret that he does not wish to pursue a political career in the longer term and he has always said that he will continue his ministerial role for a finite period," she said.
"With a successful 30-year business career behind him and the benefit of a single focus on trade and investment, he will continue to use his ministerial position to deliver more intensive support for UK business internationally and bang the drum for brand Britain around the world."
Treasury Minister Angela Eagle told BBC2's Newsnight: "He is an unusual man. He is not in the Labour Party. I think that he is saying that as a non-affiliated party person, he doesn't want to be around the government during a general election.
"That could be two years away. I suggest we talk about the economy and stop worrying about this tittle-tattle around the edge."
HAVE YOUR SAY
Gordon Brown does not have the vision, understanding or leadership qualities to get us out of this situation
Keith Brown, St Albans
But Conservative leader David Cameron said Mr Brown's "government of all the talents" policy "hasn't really worked out when you've got your own trade minister saying... he's going to pack up and leave the government because he can't campaign for you".
"I think Digby Jones is a very effective trade minister - I'm glad he's there because we need someone flying the flag for British business," he said.
"But I don't think it says much for the government when he's not prepared to speak up for them."
Shadow business and enterprise secretary Alan Duncan added: "It's unprecedented for a minister to say that they have no confidence in their own prime minister.
"If Digby Jones does not want to campaign for Gordon Brown at the next election, I suspect he is at the head of a long queue."
Lord Jones was one of several appointments Mr Brown made from outside the political arena.
But his so-called "government of all the talents" has caused some controversy.
Despite being a minister, Lord Jones refused to take the Labour whip in the House of Lords and caused embarrassment when he expressed concerns about Chancellor Alistair Darling's plans to reform capital gains tax and tax wealthy "non-domiciled" foreigners living in Britain.
Security minister Admiral Lord West had to withdraw comments that appeared to conflict with the government view of whether the police need to hold terrorist suspects for more than 28 days.
And development minister Lord Malloch-Brown got into trouble when he revealed that British officials had been in contact with Hamas.