IOC director general Francois Carrard accused the International Bobsleigh Federation of fixing the decision to allow Prusis to compete after he tested positive for nandrolone last November.
The IBF originally banned the Latvian for two years, but then reduced the ban to two months on appeal and so allowed the 36-year-old athlete to compete in Salt Lake City.
But IBF president Bob Storey defended the decision and hit back at Carrard's comments.
"I take exception to that remark," said Storey.
"Our executive committee went to great pains to make sure the appeal was not linked to the Olympics.
"The ban was reduced to three months because there were exceptional circumstances."
Storey admitted that his federation had failed to find any common ground with the IOC over the issue, saying "President Jacques Rogge and I agreed to disagree."
The IOC executive board tried to have Prusis banned from the Games but the Court for Arbitration for Sport (CAS) rejected their case.
Storey added that Prusis' involvement would not unfairly affect the Olympic competition.
"Anyone who knows our sport knows that Prusis has no chance to win a medal.
"He has not been seeded because he did not win any points during his suspension and he will start so far down it is impossible to get a good result."