The Tanzania Teachers' Union is taking legal action after 19 primary school teachers were given the cane.
The teachers were caned by a police officer in front of their pupils after an investigation into poor exam results at three schools.
The report blamed teachers for being late or not showing up for work and not teaching the official syllabus.
One of the caned teachers, Ativus Leonard, 33, told the BBC he was now too ashamed to meet his pupils.
The deputy education minister has said those responsible for the caning "should have their heads examined".
The union is planning a large demonstration next week in protest at the incident.
The case comes at a time when parents and human rights groups in Tanzania have been calling for a ban on flogging of schoolchildren throughout the country.
The union says it will sue District Commissioner (DC) Albert Mnali, who ordered the caning in the northern region of Kagera.
"The caning of our teachers is shameful. It's intolerable and it's time the teachers take action against Mr Mnali through the judiciary," Kagera teachers' union chairman Dauda Bilikesi told the BBC.
"We have informed the police in Bukoba [capital of Kagera region] that we will be marching through the town to demand the government takes disciplinary measures against Mr Mnali. We want him to leave, he is not fit for his job", he said.
Mr Leonard, said he had been kicked by a police officer to make him lie before being beaten.
"He hit me everywhere; my legs, my chest, my arms, my hands. When it was over, I went to the hospital for treatment. I was given medicine but I still have a lot of pain in my chest," he said.
"I have not been able to teach since this happened. I feel inferior to the children now."
Union officials have told the BBC that the caning happened in the wake of an ongoing row between the union and the government.
The teachers had complained that their salaries are often delayed and that they are a denied transfer allowance when they move to a new school.
The BBC's John Ngahyoma says the Kagera teachers were told by the DC that they were disturbing the government with their demands and that they were not performing well.
Deputy Education Minister Mwantumu Bakari Mahiza told the Tanzanian Broadcasting Corporation that the DC's action was "abnormal" and called for an investigation.