The defence secretary has rejected a suggestion that the armed forces are being treated with contempt because he also acts as Scottish secretary.
Mr Browne said he was able to cope with both portfolios
Former military chief Admiral Lord Boyce said Des Browne should not be allowed to continue in both jobs.
However, speaking to BBC Radio Scotland, Mr Browne insisted he could continue with both portfolios.
He hit back at the admiral's claims and said he would put his record in the defence role "against anyone's".
Admiral Lord Boyce had earlier criticised the dual role.
He said: "When you have got people who have been killed and maimed in the service of their government, and you put at the head of the shop someone who is part-time, that sends a very bad message."
Following his comments, it emerged that Conservative leader David Cameron had written to Prime Gordon Brown Minister asking him either to remove the role of Scottish secretary from Des Browne, to allow him to work full time as defence secretary, or to appoint someone else into the job.
His letter reads: "At a time when our forces are engaged in two highly dangerous missions in Iraq and Afghanistan, the need for the defence secretary to be able to concentrate full-time on his role is surely a matter of plain common sense.
"People do not understand why you have chosen to combine the job of defence secretary with Scottish secretary at such a crucial time.
"Now Admiral Boyce has made clear this concern extends to those on the front line, I hope you will take this opportunity to reverse your decision and appoint a defence secretary who is able to devote his total attention to this vital job, unencumbered by other responsibilities within government."
But Mr Browne said that being Scottish secretary did not detract from his duty as defence secretary, as most powers were now devolved to Holyrood.
He said: "Tell me what it is that a secretary of state for defence should be doing, that I am not doing... This is not an issue that has ever been raised with me by any serving soldier when I have been in an operational theatre."
Mr Browne added: "Every single member of parliament who is a minister starts off with two jobs in the first place.
"If anything, the additional responsibilities that I have taken on - which I share with a minister of state, David Cairns, who is a very hard working person - has bitten into the time that my family had and my constituency had."