"Speaker Martin is not the right man to have at the helm," he told the BBC. "He is too compromised."
"It is a question of competence and acting fairly and in a non-partisan manner," he said.
Mr Carswell, a long-time critic of Mr Martin, says he has already picked up more than half a dozen sponsors for his motion and that it has cross-party support.
But he told the BBC he wanted to give MPs the opportunity to go back to their constituencies at the weekend, to understand the public anger about the expenses furore and realise change is needed.
"We need a new speaker with a mandate for radical change to make politicians work for the people," he said.
"I'm not having a dig at Michael Martin because he is anything other than a decent, honest, honourable man.
"I just happen to think he is bad at doing the job of Speaker."
'Pearls of wisdom'
Newport West MP Paul Flynn told BBC Wales: "He hasn't led the House of Commons - he's been too defensive and has actually attacked the whistleblowers. We need someone who is interested in exposing the errors of the past and is willing to introduce new rules."
And Liberal Democrat frontbencher Norman Lamb told the BBC he would sign the motion.
"The final straw came on Monday when he attacked backbenchers in Parliament over this whole expenses debacle. And I felt that at that point he lost his impartiality," Mr Lamb said.
Michael Martin reacts angrily to MP Kate Hoey's questioning in the Commons
The BBC's political editor Nick Robinson said the number of public calls for Mr Martin to go was unprecedented.
On Monday, in a statement to the Commons, Mr Martin angrily defended the decision to ask police to investigate where the expenses leak came from.
He rebuked Labour MP Kate Hoey, who said it was a waste of money when police had a "huge" job to do in London, telling her: "I hear your public utterances and your pearls of wisdom on Sky News. It's easy to talk then."
He also rebuked Lib Dem MP Norman Baker, who has long campaigned for greater transparency on MPs' expenses, as "another member who is keen to say to the press what the press wants to hear".
Mr Baker said later he "appeared to be defending vested interests rather than leading us out of this mess".
In further bad-tempered exchanges on Tuesday Labour's David Winnick challenged Mr Martin to apologise for his comments to Ms Hoey.
Employment Minister Tony McNulty has criticised Mr Martin over his dealings with Ms Hoey and Mr Baker.
You have got to understand that a lot of things happen in the heat of the moment
He said he did not believe the speaker was doing a bad job, but he "thought he was a bit heavy-handed with Kate Hoey and Norman Baker".
Mr McNulty added: "I think that does detract from the overall reasonable job that he's doing."
Conservative leader David Cameron has not criticised Mr Martin, saying it was "a very important constitutional principle that the Opposition supports the Speaker's office and the role of the Speaker".
Prime Minister Gordon Brown has also said Mr Martin "does a good job" adding: "You have got to understand that a lot of things happen in the heat of the moment."
Mr Martin has been Speaker, the presiding officer of the House of Commons, since 2000.
He has found himself embroiled in some controversy - most recently over the handling of the arrest of the Conservative frontbencher Damian Green and search of his Commons office.
His critics have also questioned his own use of expenses and accused him of not being impartial.
But his supporters say the Glasgow North East MP - a former sheet metal worker - has faced unprecedented snobbery and slurs since becoming Speaker in 2000.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.