The Speaker has said he has 'regrets' about the raid
Thirty-two MPs questioned as part of BBC research say they no longer have confidence in Commons Speaker Michael Martin after the Damian Green affair.
Fifty MPs said he was at fault over the arrest of the Tory MP and the search of his office at the Commons as part of a probe into alleged Home Office leaks.
Tory home affairs spokesman Dominic Grieve later said many MPs did not wish to threaten Mr Martin's position.
"But they want to see action taken to ensure this episode is never repeated."
The Speaker remains under intense pressure ahead of Monday's debate in Parliament about the Damian Green affair and calls from three MPs, including one Labour backbencher, for him to stand down.
Shadow home secretary Dominic Grieve says there were 'serious failures by House authorities'
Mr Grieve said the House of Commons authorities had "seriously failed" to protect Mr Green in his pursuit of his legitimate parliamentary duties, describing their handling of the issue as a "complete disaster".
But The Sunday Times reported that Mr Martin, who became Speaker in 2000, intended to put himself up for re-election after the next general election.
According to the paper, a spokesman for the Speaker's office said his position had not "changed" since his declaration in 2007 that he planned to stand again.
Mr Grieve said the issue of Mr Martin's re-election was a "matter for MPs to decide, not for the Speaker to decide".
"He has been Speaker for quite a long period of time," he added.
The row erupted after Mr Green, the Tory immigration spokesman, was arrested 10 days ago and held for nine hours by the Metropolitan Police, while his homes and parliamentary office were searched, as part of an inquiry into allegations of leaks from the Home Office.
The move has provoked outcry among MPs from all parties, who say the police action represented a fundamental breach of their right to hold the government to account.
Two Conservative MPs have called for the Speaker to stand down.
However, in a questionnaire of 130 MPs for the BBC Radio 4's World This Weekend programme, 36 disagreed with this and 56 said they still had confidence in him.
The World This Weekend attempted to contact all backbench MPs to ask them how they felt about Mr Martin in light of recent events.
The programme put two questions to them: Do you believe the Speaker was culpable in the arrest of Mr Green and the search of his Parliamentary office? And, do you still have confidence in him as Speaker?
The programme said the answers showed many MPs were more unhappy as a result of a Commons statement Mr Martin made on Wednesday, than they were before.
In the statement, he said one of his officials had allowed the raid by signing a consent form without consulting him, and that he had known in advance about the search, but had not been told that the police did not have a warrant.
A number of MPs told the World This Weekend they had been "shocked" and "disturbed" by the Speaker's statement.
But Work and Pensions Secretary James Purnell told the Andrew Marr Show that "of course I have confidence in the Speaker".
But he added the issue was a matter for Parliament and not the government.
"If you start getting in a situation where ministers are the people who are responsible for the Speaker's position that would be completely against every tradition we have," he said.
On Friday, backbencher Bob Marshall-Andrews became the first Labour MP to publicly call for Mr Martin to resign.
He said the Speaker had been guilty of "a deplorable breach of his duties to the house" and the office of an MP should be sacrosanct.
On Monday MPs will debate Parliament's response to the episode.
But opposition parties are unhappy about the motion put forward for debate and the fact that the parliamentary inquiry into the affair will not begin until after the police investigation has finished.
Mr Grieve said the inquiry initiated by the Speaker had effectively been "pushed into the long grass" and said the police should consider winding up their investigation as soon as possible as "it is going nowhere".
But housing minister Margaret Beckett said it was "alarming" that any politician should seek to judge the outcome of a police investigation before it had been completed.
"I genuinely think this is getting close to intimidation of the police," she told Sky News.
"I don't think anyone should be commenting on an inquiry in course."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.