Sixsmith says smears have come from near Number 10
The statement comes after the civil servant at the heart of the transport department spin row - Martin Sixsmith - insisted Mr Byers was behind his controversial departure.
Mr Sixsmith says he was forced out of his transport department job over his dealings with the spin doctor Jo Moore.
Mr Byers is due to make a Commons statement at 1530.
"I don't feel the strong sense of support for Stephen Byers that was around even until yesterday."
He added: "I sort of sense - and no-one has said directly this - I sense a withdrawal of love, if I can put it that way from Downing Street for this minister and I think it is going to be a very interesting afternoon."
Blunkett defends Byers
Earlier, Home Secretary David Blunkett defended Mr Byers, describing him as "an excellent minister".
"If he is allowed to get on with his job, he will deliver in those areas that matter a lot more to the public than whether Martin Sixsmith has a set of notes to use against those who employed him," said Mr Blunkett.
Mr Sixsmith insists he did not agree to go in the first place and was promised no announcement would be made before he was consulted further.
Downing Street continues to say Mr Byers, a close ally of Tony Blair, still has the prime minister's full confidence.
Parallel departures
But speaking on BBC Radio 4's Today programme, in his first broadcast interview about the row, Mr Sixsmith said Mr Byers had only agreed to accept Ms Moore's resignation on condition he went too.
"Richard told me he had spoken to Stephen Byers about this and that the decision was it had to be a symmetrical resignation.
"I had to go at the same time as Jo."
That decision came despite Sir Richard Mottram telling the press chief he believed Mr Sixsmith had done nothing wrong, he continued.
The transport secretary has said the affair was a "personnel" issue handled by Sir Richard.
Mr Sixsmith said he had only gone public with his story after talks to find a "face-saving compromise" for all involved, including Mr Byers.
Friends had warned him he would face a smear campaign but he had not believed them - now he was not too sure.
Nothing murky
"I've had well-respected journalists from well-respected newspapers ringing up and saying sources close to Number 10 are saying: 'Ring Martin and try to find out something about his property portfolio'."
Mr Blair's official spokesman said there had been no smear campaign against Mr Sixsmith from Downing Street and described the affair as a "soap opera" and a "farrago".
Earlier, Conservative leader Iain Duncan Smith said it was "bizarre" government resources were being exhausted on defending a transport secretary who could not run his department and whose word no one believed.
Liberal Democrat transport spokesman Don Foster said it was time Mr Byers took responsibility and resigned "for the sake of public confidence".