On Tuesday, Tepco said 100 barrels containing radioactive waste had been knocked over and burst open and that major exhaust pipes had been knocked out of place.
Small amounts of gas containing radioactive cobalt-60 and chromium-51 were emitted into the atmosphere.
"They raised the alert too late," said Mr Abe. "I have sent stern instructions that such alerts must be raised seriously and swiftly."
Officials at the plant are keeping all seven of its reactors closed while further inspections are carried out.
There have long been concerns about the safety of Japan's nuclear power plants, which many fear are vulnerable in earthquakes.
The country is heavily reliant on the industry - it provides about a third of Japan's electricity needs.
The government requires nuclear reactors to be able to withstand earthquakes of up to magnitude 6.5 - weaker than Monday's quake.
Other businesses are also beginning to count the cost of the earthquake.
Riken Corp, which makes car parts for companies such as Honda and Toyota, says it is unsure when it will be able to resume production at its factory in Kashiwazaki after the quake injured some of its employees and damaged equipment.
Fuji Xerox has also had to halt production at its Kashiwazaki plant, which mainly assembles printers, because it is without power and there has been some damage to the building.
Do you live in the area affected by the earthquake? Have you suffered any damages? Send us your experiences using the form below:
You can send us pictures and video to: email@example.com or to send via MMS please dial +44 (0)7725 100 100
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.