Scolari's team thrashed Costa Rica 5-2 in their final Group C game but the scoreline belied their defensive frailities.
Costa Rica - needing just a draw to progress - were all too often free to walk through Brazil's defence and squandered plenty of chances as they crashed out of the competition.
And Scolari - commonly known as Big Phil - said he will go straight to work to shore up Brazil's defence ahead of the knockout stages.
We suffered a lot back on defence
"We must carry on improving. We understand people see us as the favourites - but we must continue to work," said Scolari.
"We made a few mistakes. When Brazil get ahead we stop doing the things we work on in training. We have a few points to iron out.
"We gave chances to our opponents but we will work on this.
"Over the next few days, I will show my team a video of the first 15 minutes of the Costa Rica match and the last 15 minutes because that's when we made mistakes.
"We will concentrate specifically on those problems."
Brazilian playmaker Junior was the catalyst for many of his side's attacking thrusts and scored the fifth himself.
But he also conceded that the leaky defence was putting more pressure on the forwards to score.
"We suffered a lot back on defence because Costa Rica was very dangerous," Junior said.
"We will have to talk during the week on what happened and then will have to correct because those things should not happen."