It became famous as a favourite London hang-out of Bob Marley's in the 1970s and survived for 30 years despite its status as a squat.
Seymour Mclean said the temple was a valuable community asset
But the Rastafarian temple in St Agnes Place, south London, now looks set for closure after a police raid which saw 23 people arrested on suspicion of drug dealing.
Police said the temple's managers had told them the building was no longer used as a temple and had been taken over by drug suppliers.
But the temple's chaplain Seymour Mclean told the BBC News website that people had been worshipping there right up until the night before the raid, which happened just before 0300 GMT on Thursday.
He, and other temple users, reacted with anger to news that Lambeth Council had said the building was to be demolished and the temple relocated.
"I have heard nothing about this," Mr Mclean said.
"What the police and council say is not always correct. The temple was in use seven days a week. I lead prayers there myself.
"Usually we would get about 25 people. On the night before the raid we had a meeting.
Bob Marley visited the temple several times in the 1970s
"The police threw their device through the window of my office at the front of the building. Luckily I was not there at the time."
The Rastafarian temple was established in the early 1970s after squatters moved into a row of Victorian town houses at one end of Kennington Park.
It has been called the centre of the UK's Rastafarian community and was Rastafarian reggae star Bob Marley's favourite place to stay when he was in London.
He was said to have enjoyed several games of football in front of the temple in Kennington Park.
Grassroots projects in the properties included housing for the homeless, a social centre, music events and a community radio station.
However, unpaid rent over 30 years was estimated to amount to £4m, plus a further £400,000 in council tax.
In 2005 a number of homes in the street were cleared of 150 squatters by riot police and knocked down to make room for new homes and a sports centre, but the temple was left standing.
An Ethiopian flag flies above the building and the tops of the railings outside are neatly painted green, gold and red.
Situated on the bottom floor across four red-brick terraced homes, the temple features photographs of Bob Marley and former Ethiopian Emperor Haile Selassie, who Rastafarians believe was the incarnation of God.
It also has Ethiopian flags draped across its walls and contains the world's biggest bass drum, according to members.
Many Rastafarians see smoking cannabis as a spiritual act and Mr Mclean said it was smoked in the temple while prayer meetings were held.
He said the community would lose a vital centre if the temple was shut down.
"There are lots of boys who are rebels without a cause here," he said.
"They might steal cars and get into trouble. But we offer them another way here. We can channel the boys' anger and we preach peace to them."
Another member of the temple, who did not want to be named, said: "This was a positive thing in the community.
"I don't know what was happening but it will be a huge loss if it goes. People will be very, very angry. It won't just be Rastafari anger, it will be black anger."
One Rastafarian, who used to worship at the temple, supported the raid.
"I'm shocked that what was once a friendly, happy place to be seems to have been taken over by criminals," he said.
"Everyone locally supports the police when they take action against drug dealing.
"Hopefully the council will act quickly to find somewhere else for Rastafarians to worship, but there is a feeling that some people at the council are not working as quickly as they could.
"Two years ago, when many of the squats on this street were demolished, the heart was ripped out of the community."
Lambeth Council leader Steve Reed said the council was not targeting Rastafarians.
"I would like to make it clear that this is not an attack on our Rastafarian community - this is an attack on drug dealers and criminals," he said.
He said the police operation marked the launch of the borough's "get tough" policy on drug dealing.