First Minister Alex Salmond has called for the Scotland Office to be scrapped in the wake of the criticism over Des Browne's dual role.
Alex Salmond said the current situation was unsatisfactory
Mr Browne has insisted he can continue as both Scottish and defence secretary.
But SNP leader Mr Salmond said the current situation was "unsatisfactory" and called for wide-ranging reforms.
He said it would be possible for the devolved administration in Edinburgh to deal directly with the prime minister, bypassing the Scottish secretary.
On Thursday, former military chief Admiral Lord Boyce said Mr Browne should not be allowed to continue as both Scottish and defence secretary.
Tory leader David Cameron then wrote to Prime Minister Gordon Brown asking him to either remove Mr Browne from the role of Scottish secretary or allow someone else to be a full-time defence secretary.
Mr Salmond said: "The current dual role is clearly unsatisfactory from a defence point of view, as well as from Scotland's point of view.
"I would encourage Gordon Brown to take the opportunity of the criticism of the current structure to bring forward a package of reform measures to strengthen direct relations between the devolved administrations and the Westminster government.
"Having a part-time secretary of state combined with a UK department is clearly anomalous and now is a good time to consider and bring forward improvements."
He said it would be "perfectly possible" for the Holyrood administration to deal directly with the prime minister.
"That would be far better all round, and should be part of a wider package which would see the revival of the joint ministerial committees to provide focus and structure to the formal links between the governments in London, Edinburgh, Cardiff and Belfast," he said.
On Friday, Mr Browne rejected suggestions that the armed forces were being treated with contempt as a result of his dual role.
He said: "Tell me what it is that a secretary of state for defence should be doing, that I am not doing.
"This is not an issue that has ever been raised with me by any serving soldier when I have been in an operational theatre."