South Africa will reconsider future tours of Australia if racial abuse of players continues.
Spectators were thrown out of three grounds following racist taunts as the tourists lost the Test series 2-0.
"If it continues, yes, we would look very seriously about whether we return for another series," cricket board head Gerald Majola told several newspapers.
"It is not right when a country has a history like ours. It is something that we never thought we would hear about."
South Africa are currently involved in the one-day VB Series, with their next game against Sri Lanka in Perth on Tuesday.
Sri Lankan players were also reportedly subject to racist shouts during Thursday's Australia Day one-day match in Adelaide.
If caught during the game, the perpetrators face instant fines and bans from attending future fixtures.
"The sad thing is, it has continued around Australia. It hasn't just been limited to one state or one city," Majola added.
Tours between Test-playing nations are governed by International Cricket Council rules, which threaten fines if they are not fulfilled.
Cricket Australia boss James Sutherland said a methods of cracking down on racist fans were being investigated
"Look at the way they address the issue of hooliganism in England," he said.
"They make attempts to identify serial offenders and keep them out.
"That certainly is an option and something we want to understand better before we go forward. These types of people are not wanted at cricket matches.
"If there is a cost-effective method of keeping them out, we will certainly look into it."