Plans for a super-casino in Manchester, and 16 smaller venues across Britain could be thrown out by MPs.
The panel recommended a super-casino be built in east Manchester
The Tories have said they will vote with Labour rebels against the Gambling Order 2007 in the Commons on Wednesday, threatening defeat for the government.
They back the 16 small casinos, but say a Lords report into the super-casino raised questions about its suitability.
The decision to award Manchester the casino over Blackpool has sparked a revolt by some Labour MPs.
More than 100 MPs - including 83 Labour - have signed a Parliamentary motion expressing "surprise and regret" at the recommendation.
There was criticism that the city's relatively high unemployment and income support might make residents more vulnerable to problem gambling.
And supporters of Blackpool's bid - which was tipped as the favourite - say the super-casino is needed there to turn around the seaside resort's fortunes.
The order, which grants the powers to issue licences for new casinos, also faces opposition from those who oppose the government's attempts to liberalise gambling laws.
The Tories had said they would back the recommendation, but a party spokesman said they had been persuaded by a critical Lords committee report to vote against the package.
Both MPs and peers will vote on the Gambling (Geographical Distribution of Casino Premises Licences) Order 2007 on Wednesday.
But they cannot vote to change the location, so critics will have to vote against the entire Order.
The Conservatives have called on Culture Secretary Tessa Jowell to offer two separate votes - one on the super-casino, the other on the 16 smaller casinos - on Wednesday.
In the Lords, Labour peer Baroness Golding and the Lib Dems have tabled amendments calling for a joint committee of MPs and peers to be set up to consider the process by which Manchester was picked.
That decision was made by an independent Casino Advisory Panel - but has been backed by the government.
The panel also recommended that large casinos should be licensed at Great Yarmouth, Kingston-upon-Hull, Leeds, Middlesbrough, Milton Keynes, Newham, Solihull and Southampton.
And it said that small casinos should be sited at Bath and North East Somerset, Dumfries and Galloway, East Lindsey, Luton, Scarborough, Swansea, Torbay and Wolverhampton.