Muslims in Cornwall are looking for a new place to worship after an enforcement notice against the use of their current building was upheld.
The centre in Truro caters for up to 40 Muslim worshippers
Government planning inspector Roy Gardner said Carrick District Council was right to ask for the centre in a house in Truro to close.
The semi-detached property in Tresawls Road is also used as a residence.
The council found the centre, which is used by up to 40 worshippers, was leading to an increase in traffic and noise, though that is disputed by worshippers.
Secretary of the Cornwall Islamic Centre, Tipo Choudhury, said he was very disappointed by the appeal decision as the centre was also used for educational and social purposes.
"When we bought the premises we knew we would have to go through the formal planning procedures.
"But to be turned down is very disappointing."
He said he felt people in general remained suspicious of the Muslim faith.
"It is not a religious or racial thing, but it is more a fear of the unknown," he said.
Mr Gardner said the house was in the wrong place for the use it was being put to and therefore the planning objections were valid.
Doctor Duriad Rifai, a surgeon at the Royal Cornwall Hospital who worships at the centre, agreed the centre was not in an ideal place.
"We would like to be somewhere else, but the council has been very obstructive when we have come up with a solution."
But Councillor Maggie Vale, chairwoman of Carrick's planning committee, said they were sympathetic to the Islamic community and would work with them to find a solution.
The inspector has given the worshippers more time to find a new home.
Mr Choudhury said: "We are looking for maybe even something temporary."