The newly-detected fault lies off the south-eastern coast of Japan and may have been responsible for the magnitude 8.1 earthquake which struck the country in 1944, they say.
Jin-Oh Park and his colleagues at Jamstec, the Japan Marine Science and Technology Center, say that an earthquake along the fault would threaten cities along the Japanese coast.
The fault is close enough to the Japanese coast for there to be only minutes between a substantial earthquake along it and the tsunami reaching land.
Uncharted sea bed
The fault, which lies close to where the Philippine Sea plate is sinking beneath the Eurasian plate, is only dozens of kilometres away from land.
"Any tsunami would hit the mainland with only a few minutes' warning," explained Bill McGuire, director of the UK's Benfield Greig Hazard Research Centre.
"Most people in Japan live along the coast and evacuating them in only a few minutes would be impossible," he said.
"It's an additional tsunami risk in a country that has many of them," he added.
The newly-discovered fault was uncovered by seismic reflection imaging.
Submarine faults are difficult to find and much of the ocean floor is still poorly understood.
Faults may never protrude above the ocean floor or may be covered in sediment.
History of disaster
Jin-Oh Park and his colleagues believe that the fault they have found may have been responsible not just for a magnitude 8.1 quake in 1944, but a nearby magnitude 8.3 quake two years later.
Very large earthquakes tend to occur every 100 to 200 years along faults of the kind now discovered, they explain.
Details of the fault's discovery appear in the journal Science.