Mr Nakayama had made a series of controversial remarks
Japanese Transport Minister Nariaki Nakayama has resigned, just four days after taking the job.
The resignation will be seen as a setback for new Prime Minister Taro Aso, who took office on Wednesday.
Mr Nakayama was criticised over a series of controversial remarks. He called Japan's largest teachers' union a "cancer" in the education system.
He also angered Japan's indigenous Ainu people last week, when he described the country as ethnically homogeneous.
The remark was seen as particularly insensitive because Japanese parliament passed a landmark resolution in June recognising the Ainu as "an indigenous people with a distinct language, religion and culture".
Chief Cabinet Secretary Takeo Kawamura said the controversy of Mr Nakayama had been "damaging".
"We must show the people how hard the Aso government is working, and try to win back the public's confidence. That is all that we can do," he told a news conference.
Mr Nakayama is no stranger to controversy, having previously angered China by saying that reports of Japanese wartime atrocities, including the 1937 Nanjing Massacre, were exaggerated.
He joins a growing line of Japanese ministers who have risked their jobs by sharing unguarded opinions.
Mr Aso is under pressure to call a general election
Earlier this month, farm minister Seiichi Ota resigned after admitting that his ministry had known about a rice contamination scandal but that he had seen no need to make "too much of a fuss over it".
Fumio Kyuma resigned as defence minister in July 2007 after implying that the US atomic bombing of Nagasaki and Hiroshima in 1945 was inevitable.
And in January 2007, former health minister Hakuo Yanagisawa was sharply criticised for referring to women as "birth-giving machines" during discussions about Japan's low birth rate.
Mr Nakayama, a former minister for education, had said he would "stand at the forefront to destroy the Japan Teachers' Union, which is a cancer for Japanese education".
Defending his comments, he said he had "meant to stir the interest of the Japanese people that distorted education is now conducted in schools".
"If my remarks have made any impact on parliamentary proceedings, it would not be what I had intended," he said.
The union's secretary general said he was "flabbergasted" by the comments" and questioned Mr Nakayama's judgement.
Pressure is growing on Mr Aso to call a snap election in a effort to shore up his authority.
His Liberal Democratic Party (LDP) has dominated Japanese politics for more than 50 years, but is now facing a resurgent opposition.
The latest newspaper opinion polls show public support for Mr Aso at lower than 50% and the country is facing stormy economic conditions.
Last week, Japan announced its sharpest fall in economic output in almost seven years.
The last prime minister, Yasuo Fukuda, resigned earlier this month after less than a year in office, frustrated by the ability of the opposition-controlled upper house of parliament to stymie his legislative plans.