The government has been criticised for asking licence fee payers to help fund the switch from analogue to digital TV.
The licence fee could rise to £180 by 2014, the committee predicted
The BBC has been told to use licence fee money to help elderly and disabled people pay for digital TV equipment.
But a Lords committee examining BBC charter renewal said the switchover could push the licence's cost too high and the government itself should pay.
The Department for Culture, Media and Sport said it was "far too early" to speculate on new licence fee payments.
Every household which wants digital TV must have it by 2012, when the analogue signal will be switched off.
The House of Lords select committee's report said ministers were "loading costs onto the BBC which will result in a higher than necessary licence fee".
The fee will rise to £180 per year in the next eight years, it predicted - up from the current £126.50 price.
If the fee continues to go up steeply, "it will damage the BBC by undermining support for the corporation", the committee said.
"The substantial cost of helping the elderly and disabled with digital switchover should not be a cost picked up by the licence fee payer," committee chairman Lord Fowler said.
"The government rightly takes on the responsibility for providing free licences for the over-75s and funds it from general taxation. It should do the same with help for switchover."
The BBC should also work harder to reduce other parts of its budget, the committee said.
"The licence fee should only rise significantly if there are exceptional and well substantiated reasons for it to do so," the committee said.
The BBC and government are negotiating how much the licence fee will cost between 2007-2014 - a process that the public sees as "secretive and opaque", according to the committee.
The National Audit Office and Parliament should help set the cost, it said.
"The licence fee is rising at an unprecedented rate and it is time that it was open to proper scrutiny."
A spokesman for the Department for Culture, Media and Sport said it was "far too early" to speculate on how much viewers, listeners and readers will pay under the new licence fee settlement.
He also said the BBC "should play a leading role in digital switchover, not least because it will ensure that all licence fee payers can receive the BBC's digital channels".
He said: "It's therefore right that the licence fee should play a role in making this happen."
The BBC has agreed to help "the most needy" afford digital TV - but on two conditions, a statement from the corporation said.
"This cost should not be at the expense of existing services, and it should not undermine long-term support for the licence fee," it said.
The BBC's bid for a future licence fee increase "represents good value for money and meets the objectives set by the government for building a digital Britain", it added.