That's what happened to kids in part of Arkansas in the United States, until a judge said it just wasn't fair.
A school board had banned kids from taking any of the books out from libraries without their parents' permission.
It was because some parents in Cedarville school district had complained about them.
'Encourage witchcraft'
They said the books encourage witchcraft, are evil and tell kids to disobey orders.
But there were also many people really angry about the decision, because they think it harms kids' freedom to choose what they read.
The court agreed with them, and told the school board to make sure kids can now borrow all four books and read as much Potter as they like!
Hear story!
Harry Potter Message Boards
Try our Potter book quiz
