A Portuguese family was attacked minutes after England were knocked out of the World Cup on Saturday.
Their car was vandalised and a brick thrown through their house window when a crowd gathered on an estate where Portuguese families have settled.
Several of the families had earlier put up flags in support of their team, but these were taken down before the match.
The crowd scattered when police arrived at the estate in Cumbria, but officers say they will be tracked down.
About six families from Portugal have settled on the estate after coming to the UK as migrant workers.
Police said they had patrols in the area in case of trouble and arrived within four minutes of getting the 999 call.
Insp Adrian Sowerby, of Cumbria Police, said: "We are all disappointed when your national team gets knocked out but it can in no way justify what has happened.
"It is being treated as a hate crime and a racist incident."
The family were badly shaken but otherwise unharmed in the attack, which caused about £400 of damage.