The House of Lords constitutional committee also said it did not know why Helen Liddell's Scotland Office had so many staff.
Ms Liddell defended her post saying her job was vital to Scotland's interest.
The peers said they doubted whether a minister of cabinet rank was needed to argue Scotland's case in Whitehall now that the country has a Scottish Parliament and Scottish Executive.
"
Those who argue against having a Scottish voice in the cabinet are arguing to reduce Scotland's influence
"
Helen Liddell
Instead, the committee said it might make better sense to have a single minister in the UK cabinet responsible for intergovernmental relations.
The committee also questioned the number of people employed by the Scotland Office, which is more than twice that of the Welsh Office.
Peers said they have been offered no ''cogent explanation for this discrepancy'' and they did not understand how it had arisen.
The Scottish National Party's Westminster Chief Whip, Pete Wishart MP, said: "Although Helen Liddell is the most under worked minister in the Cabinet, the Scotland Office still costs more to run now that it did at the start of devolution.
"During Scottish Questions last week, the Secretary of State boasted that she would be spending her time from now until May campaigning for Jack McConnell.
"Abolishing the Scotland Office now would allow millions of pounds of public money to be spent on hospitals and schools, rather than on campaigning for New Labour on the public purse, which is totally unacceptable."
When asked by BBC Radio Scotland about the possible abolition of her job, Ms Liddell said: "My job is a matter for the prime minister.
"But there is a very important job to be done and I think those who argue against having a Scottish voice in the cabinet are arguing to reduce Scotland's influence.
"There is hardly a day goes by when I don't raise an issue that is distinctively Scottish.
"It is a job that is vital to Scotland's interest and I think it has contributed greatly to the stability of the devolution settlement."
Many powers were devolved to the Scottish Parliament under the constuitutional shake-up which became reality in 1999.
However, the post of Scottish Secretary was retained and Mrs Liddell's Scotland Office was given the responsibility of "ensuring that Scottish interests are represented" at Westminster.