And doubt was cast on Mr Brown's plan to reassert his authority with a post-poll cabinet reshuffle by the early announcement that Ms Smith was stepping down.
Opposition parties have stepped up their calls for an immediate general election, with Lib Dem leader Nick Clegg saying the government was "in its death throes" and there were now questions about "whether Britain is being governed at all".
And the normally-Labour supporting Guardian newspaper has published a lengthy editorial calling for Mr Brown to quit.
But on BBC Radio 4's Today, Ms Harman denied the government was in a "mess", insisting Tuesday's cabinet meeting had been "a purposeful day of doing the business of the government".
Ex-cabinet minister Patricia Hewitt said she would not seek re-election so she could concentrate on her charity work in India.
But Universities Secretary John Denham denied the departures meant Gordon Brown was facing a crisis.
He said: "I've seen a number of reshuffles, many over the years I've been in Parliament. I think in almost every one there has been one or more ministers have indicated in advance that they didn't want to be considered."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.