By Iain Watson
BBC News political correspondent
The Liberal Democrat home affairs spokesman Chris Huhne called it a crushing defeat. The government would probably prefer to see it as a tactical retreat.
MPs voted in favour of the plan, by just nine votes
The Counter-Terrorism Bill will continue its passage through the House of Commons, but minus its most controversial provision - the detention of terrorist suspects for up to 42 days.
Home Secretary Jacqui Smith's previously unscheduled statement to the Commons followed a much bigger defeat on 42-day detention in the Lords than had been anticipated.
She confirmed that the government would not attempt to override the will of the Lords by using the Parliament Act. Instead a new bill to allow 42-day detention would be introduced if an emergency warranted it.
The government has decided on this approach for two reasons.
To overturn the Lords vote, it would have to wait until next spring.
That is because Parliamentary convention dictates that they have to wait a year from the date the matter was first discussed in the Commons.
Secondly the government would have to seek the support of the House of Commons all over again.
And that could prove tricky.
When the Commons last voted on 42 days -in June -the government scraped home with a majority of just nine - and only with the help of Ian Paisley's nine Democratic Unionist MPs.
Thirty six Labour MPs rebelled but government whips had identified around 50 potential dissidents.
So the fear is that if the whole issue is revisited after a resounding defeat in the Lords, some Labour MPs who had previously abstained or voted with the government only because they believed the prime minister's own position was vulnerable might change their minds.
To ensure that the government's slender Commons majority is maintained then a massive programme of persuasion -friendly or otherwise - would have to be undertaken by the whips amidst a global financial crisis which, the government says, has its undivided attention.
So the government decided not to try to cash-in possibly diminishing political capital on this issue.
But by drafting a separate bill on 42 day detention, which could be introduced in an emergency, the government are in effect taking out a political insurance policy against a future terrorist attack.
They would then be able to say that they have the measures ready to detain suspects and that - had the unelected Lords and the opposition not stopped them - they would have had the measures in place much sooner.
The opposition calls this "playing politics" with a serious issue - the government would say it very clearly wants to demonstrate that they are on "the right side of the argument".
The new bill will not include some of the compromises which were incorporated into the Counter Terrorism Bill in order to win over wavering rebels in the first place .
But behind the rhetoric, what has happened tonight is significant.
Unless a serious terrorist incident occurs, the government's plans to extend from 28 to 42 days the period of pre-charge detention - albeit only in 'grave exceptional circumstances' - has been defeated, and will not become law this side of the election.
But it is virtually guaranteed to become a dividing line between the government and opposition when voters go to the polls.
There is certainly no consensus on keeping politics out of national security.