Members of Pendle Borough Council in Lancashire will decide on Tuesday night if the wendy house has to go.
They say the house needs planning permission because it's less than 2m high and 1.2m wide, and has been built outside.
The children's grandmother, Audrey Goodwill, said the council's behaviour was "absolutely stupid".
"It's not in the way, it's in the garden," she said. "But it seems the council has decided there is a problem. The children are very upset."