Various estimates put the crowd at between 10,000 and 15,000, making it one of the largest in the series of demonstrations that has hit the country in recent months.
Mr Yushchenko, a former central banker appointed in December 1999, had come under fire for austerity measures and attempts to reform Ukraine's economy.
The move is the latest jolt for the former Soviet state which has been in political turmoil since President Leonid Kuchma became embroiled in a scandal over a murdered journalist last year, sparking mass street protests in the capital Kiev.
Protest
As the vote was taken, protesters chanted "Shame, shame - Kuchma out, Kuchma out," and carried placards reading "Hands off Yushchenko" and "Yushchenko for Ukraine".
Various estimates put the crowd at between 10,000 and 15,000, making it one of the largest in the series of demonstrations that has hit the country in recent months.
"As a citizen, I am convinced that democracy in Ukraine has suffered a serious loss," Mr Yushchenko said after the vote.
"I am not going to leave politics. I am going now in order to come back!"
President Kuchma said he was not pleased with the outcome.
The BBC's Jackie Rowland in Kiev says Mr Yushchenko is now likely to become a focus of opposition to Mr Kuchma.
Speaking in Brussels, the European Union's foreign policy chief Javier Solana warned Ukraine not to abandon political and economic reforms.
"We would like very much that the process of reform - regardless of the result of the vote - will continue," he said.
A coalition government is widely expected to replace the outgoing cabinet, ahead of new elections in 2002.
Trusted politician
In the hours before the vote President Kuchma and Mr Yushchenko had been holding urgent consultations with parliamentary faction leaders, trying to defuse the crisis.
Mr Kuchma met with faction heads late on Wednesday, calling for political stability to be preserved and Mr Yushchenko continued talks with them on Thursday morning.
Mr Kuchma has yet to sign the decree formally dismissing the prime minister.
In spite of his struggles with the legislature, polls indicate that Mr Yushchenko remains Ukraine's most trusted politician.
In a country notorious for corruption, he has a reputation for honesty.
Russian orbit
He is credited with reviving the country's chronically sluggish economic reforms, paying a significant portion of overdue wages and pensions, and achieving the first signs of economic growth since Ukraine gained independence in 1991.
In his parting speech he took a swipe at the business "oligarchs" who wield considerable influence in Ukrainian politics behind the scenes.
"Those who voted against the Ukrainian Government today showed that they are not ready to recognise the legal economy and public politics as the only possible means for the public development," he said.
Western diplomats have said that the vote against Mr Yushchenko may be a sign that Ukraine is drifting back into the orbit of its big neighbour, Russia.