The Scottish skier had finished third in the slalom at Salt Lake City - the first ever podium finish for a British competitor in an alpine event.
But two days after the event, as he was celebrating his achievement in his home village of Aviemore, Baxter took a call to say he had tested positive for drugs.
On Thursday, breaking his silence for the first time since he found out about his positive test, the skier outlined the sequence of events which led to his fall from grace.
"I didn't think I was doing anything wrong"
"We got to Park City (in Utah) where we were staying and the next afternoon we had meetings with BOA doctors to clear any medication we were taking, if any.
"I had some sleeping tablets, that was the only thing and they were cleared a couple of days later," he said.
Baxter said that was when his coach Christian Schwaiger decided to go out and buy a humidifier to enable the skier and younger brother Noel to sleep easier in the hotel room they were sharing.
"While he was getting this humidifier I asked if he would pick me up a Vicks inhaler because my nose was blocked and I wasn't sleeping so well," Baxter revealed.
"He actually came back with a sinex, which is a liquid rather than a stick which you just inhale.
"I didn't like that so much - it's not as comfortable to use - but that was also cleared by the doctors."
Baxter then crashed during training and took the following day off - spending it shopping in Park City which is around 30 minutes away from Olympic venue Salt Lake City.
"I got food and some videos and then went to the chemist part of the supermarket.
"I saw the inhaler that I wanted Christian to buy in the first place because I had been using it since I was a kid.
"All that time it never crossed my mind that it would be different from the British one. So in my mind I had no reason to get it checked, I didn't think I was doing anything wrong."
Baxter produced the inhaler he bought in America and the British version at his press conference - at Shoeless Joe's sports bar in London's Embankment.
He then challenged journalists to tell them apart.
"I don't know if you would be able to tell the difference, I've got the British one in my right hand, the American one in my left," said the skier, brandishing the sticks.
He went on: "After the race was finished we were all supposed to go and do a urine sample for the doping control.
"That was no problem, I did my sample. You have to put down on a list what you've been taking and I put down the sleeping tablets that were cleared but once again I didn't even think to put the Vicks inhaler down."
He continued after producing his sample to a celebratory party - but two days later took the fateful call which would change everything.
"I had no idea what this positive sample could be," Baxter said.
Unfortunately, he knows all too well now.