But she said she would not call people on to the streets to protest, as she had done after the 2004 presidential election.
"I will not call another Maidan [Independence Square demonstration] and will not allow public protests," she said.
Mr Yanukovych was pronounced the victor six years ago, only for the result to be overruled because of vote-rigging. The Orange Revolution eventually brought Ms Tymoshenko's erstwhile ally, President Viktor Yushchenko, to power.
The prime minister said more than a million votes, which she said were decisive to the outcome, were invalid.
"With all this proof, I have taken the only possible decision: to challenge the results of the election in court. I will defend our state and the choice we made on the basis of legal documents," she said.
"Not going to court would mean giving up Ukraine to criminals without a fight."
On Wednesday, Mr Yanukovych demanded his rival resign as prime minister and go into opposition.
"I want to remind Ms Tymoshenko that the basis of democracy is the will of the people. Democratic leaders always accept the results of the elections. The country does not need a new crisis," he said.
The BBC's Daria Merkusheva in Kiev says Ukraine's Central Election Commission is expected to confirm Mr Yanukovych as president-elect early next week, after which the legal challenges are likely to start.
Ukraine has been in political deadlock for several years, undermining its ability to deal with a severe economic crisis. Analysts say a challenge by Mrs Tymoshenko would prolong the instability.
With Mrs Tymoshenko still heading the government, stalemate is likely to continue. Mr Yanukovych has said he wants to form a new coalition, and may try to call snap parliamentary elections.
BBC News website readers in Ukraine have been sending us their comments on this story. Here is a selection of their views.
Since the Orange revolution, corruption is higher and the poor are poorer. Yanukovych has a lot of work to do. Ukraine will probably move nearer to Russia, which is in my opinion, the right thing to do. Tymoshenko is stopping progress, she knows the election was fair. I am looking forward to some stability - the bread and water I buy has gone up in price over 50% in the last year. Bill, Kiev
I was a legal observer of part of the election process and I should say nothing was wrong at my voting point, neither the poll process, nor the counting. Ruslan, Kiev
I agree with Ms Tymoshenko. There should be a recount of ballots, especially in Donetsk, Luhansk and Crimea regions where Yanukovych's Party of Regions has absolute political dominance. Moreover, if Yanukovych is confident of his victory, he should go for a ballot recount. There should be a recount to end this story! Oleksiy, Kiev
I remember the same situation just four years ago during the Orange revolution. Now we can see the incompetence of Tymoshenko's state. Our politics and politicians now are the main source of fun for people and comedy shows. Alexsander, Donetsk
Yanukovych is the elected president, whether you like him or not. This is called democracy. Tymoshenko wants power at any cost. She doesn't care about small things like the law or democracy. Alex, Kharkiv
People of Ukraine are tired of being in limbo. Ukraine needs a strong ruler who can tell people what to do, you might call it a dictator. Favours and backhanders are the way Ukraine works. I would like like to see the Mayor of Kiev start to clean up this city instead of pocketing tax payer's money. A EU citizen, Kiev
Despite the fact the election met legal and democratic requirements, it's not enough. Unfortunately it was just a choice between two evils. Either of them is a way to nowhere! Artem, Donetsk
I believe there were elements of "unfair play" but I also believe they were on both sides, so the cumulative effect should be neutral. Ms Tymoshenko's decision to challenge election results in court will prolong the agony of Ukrainian governance in the time of economic crisis. Dima, Nikolaev and Kiev
Both Mr Yanukovych and Ms Tymoshenko represent business tycoons and oligarchs. The problem with Ukraine is its elite is divided, thus preventing a clear winner to emerge. The winner may be undecided, but there is a clear loser - the Ukrainian people. Masroor, Donetsk
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.