Ivanovic was playing in her first event since Wimbledon
Ana Ivanovic opened the door for Jelena Jankovic to take her world number one spot after suffering a shock third-round loss at the Rogers Cup.
Ivanovic displayed little of the form that carried her to the French Open title as she lost 6-2 1-6 6-2 to 17-year-old Tamira Paszek in Montreal.
Ivanovic said after she was bothered by an inflamed thumb and hoped it would not affect her Olympic preparations.
Jankovic can now take the number one spot if she makes Sunday's final.
Jankovic, who sits just 88 points behind Ivanovic in the world standings, reached the last eight with a 76-minute 6-3 6-2 demolition of Canadian Stephanie Dubois.
Meanwhile Ivanovic, playing in her first event since her surprise third-round exit at Wimbledon, lost her first two service games to go 4-0 down in the first set.
She recovered to win the second set but Paszek, ranked 94th in the world and who has won only three matches since the Australian Open at the start of the year, hit back to record a famous victory.
Ivanovic said afterwards that she now plans to travel to Europe for treatment on her thumb instead of flying straight to the Olympics.
"I was in pain through whole match," she said. "I spoke to many doctors, and they all said it just takes time for inflammation to get out.
"Now obviously I'm a little bit scared because MRI shows nothing, but I still have pain. They say it's going go away, but it's there for already 10 days so I'm going to try and see the doctor now and try to get it out.
"I was planning to go straight to Beijing. But now I might go back to Europe to see the doctor."
World number four Svetlana Kuznetsova also advanced to the quarter-finals of the Canadian event with a 7-5 2-6 6-4 win over 15-year-old Portuguese qualifier Michelle Larcher de Brito.
Kuznetsova will play seventh seed Dinara Safina, who won in Los Angeles last weekend, following her 6-2 6-2 thrashing of Swiss ninth seed Patty Schnyder.
Jankovic will next face unseeded Slovakian Dominika Cibulkova, who beat 12th-seeded Russian Nadia Petrova 7-6 6-2.
Frenchwoman Marion Bartoli will take on Japan's Ai Sugiyama after coming from a set down to beat sixth seed Anna Chakvetadze 4-6 7-5 7-6.