Fears have been raised about police powers
Pressure is also coming from the Conservatives, who claim they have wide support for a wide range of changes to the proposals.
The Anti-Terrorism Crime and Security Bill received an unopposed second reading in the Lords on Tuesday but now it faces a tougher test as peers continue line-by-line scrutiny in its committee stage.
It remains to be seen whether ministers offer piecemeal concessions to stave off the risk of peers making major changes to the legislation.
The government wants the bill on the statute books before the end of the year and argues the plans are a measured response to the terror atrocities of 11 September.
Plans such as the creation of a new offence of incitement of religious hatred and powers to allow detention of foreign terrorist suspects without trial, have sparked particular controversy.
Privacy fears
Ahead of the start of the committee stage of the bill on Wednesday, there was criticism from Lib Dem legal affairs spokesman Lord Phillips of Sudbury.
The bill as it stood was a "snoopers' charter", argued Lord Phillips, highlighting new powers for prosecutors to gain access to information held by public authorities.
The plans would mean a "revolution in our traditional rights to privacy...
"It would enable literally thousands of government and non-government organisations to hand over intimate information about any of us to anyone involved in bringing criminal proceedings, however minor."
Rough ride ahead
There was a warning shot fired too by Lord Strathclyde, who said his party had a constitutional duty to urge the government to rethink its proposals.
"Peers from all parties and the crossbenches are voicing important concerns - this is not a party issue," he told reporters.
The Conservatives wanted tough and effective laws against terror and their Lords leader said there was broad backing for a bill which focused on fighting terrorism.
But Lord Strathclyde argued the bill's aims were confused because the measure ranged too wide.
Some of the raft of new powers could not be described as "emergency legislation", he said.
The controversial new offence of religious hatred could actually prove counter-productive and inflame tensions, added the Tory peer.
During the second reading debate on Tuesday, Conservative and Lib Dem peers warned ministers they would not give a "blank cheque" for the anti-terror plans.
Tory home affairs spokesman Lord Dixon-Smith likened the bill to a "premature baby dragged kicking and screaming into the world far too soon".
And Lib Dem spokesman Lord McNally urged peers to use their "collective wisdom and experience" to improve the plans.
Away from the political parties, the Rt Rev Christopher Mayfield, Bishop of Manchester, gave a stark warning about the dangers the measures held.
Race relations worries
Particularly criticising extension of police powers, he said: "Much of the good work that has been done in improving community relations in the past few years could be undone by heavy handed introduction of the bill."
Home Office Minister Lord Rooker defended the anti-terror plans as a "moderate and precautionary response", which generally commanded whole-hearted public support.
He emphasised how the events of 11 September showed how the nature of security threats had changed.