When a complaint was made against foster parents Ian Wathey and Craig Faunch by the mother of a child in their care, no further action was taken.
Children's charity Kidscape said the couple's sentences were too short
Wakefield Council decided not to tell police after it emerged the couple had taken a photograph of a boy urinating.
The decision was criticised by a judge as she jailed the pair on Friday.
And her comments have been backed by a children's charity, who said the council now had to take part of the blame for the abuse which followed.
Michele Elliott, director of Kidscape, said: "The council was told there was a possible problem and they labelled the perpetrators as 'silly and naive'.
"This was silly and naive of the council and they are, in a way, subsequently responsibly for the abuse that followed.
"The reality is these men were abusing children. This is shoddy. It's really incredible the council would do this, really appalling."
Mrs Elliott also condemned the six and five year sentences received by Wathey and Faunch.
She said: "They have left four children traumatised and abused the trust that was placed in them. I think the fact they used foster caring to get to and abuse children is such a breach of trust they should have got a minimum sentence of 10 years."
David Holmes, chief executive of the British Association for Adoption and Fostering, said Wathey and Faunch's crimes were a "shocking betrayal" of the children placed in their care.
He said: "They are also a shocking betrayal of the amazing work done every day up and down the country by thousands of dedicated, hard working foster carers in providing safe, loving and stable homes for vulnerable children."
Mr Holmes said there were many examples of highly successful adoption and fostering by gay and lesbian couples and he hoped the "awful" case of Wathey and Faunch would not deter other couples from coming forward.
Judge Sally Cahill QC demanded an explanation of the council's actions after sentencing Wathey and Faunch on Friday.
A council spokeswoman said its corporate director of family services, Elaine McHale, would co-operate fully with the judge's request.
A team led by the former of director of social services at Surrey County Council, Brian Parrott, will start gathering information at the start of July, before carrying out an inquiry into the council's actions later in the year.