More than half (56%) of recent mental health inpatients rated their ward as unpleasant or very unpleasant, a survey has found.
Many patients were unhappy with their wards
The charity Mind also found 29% were dissatisfied with the state of repair of their ward, and 28% with standards of cleanliness.
Mind is calling all patients to be treated in a therapeutic environment.
It warned that lack of activities leads to boredom, which causes frustration and slows recovery.
The average mental health inpatient stay is 58 days - nearly 12 times longer than the average for patients with physical problems.
A report by the Healthcare Commission, published last year, found that standards of cleanliness were poorer in mental health hospitals than in acute units.
Mind said good ward design could play a crucial role in providing a calm and therapeutic environment , as could access to gardens and other green spaces.
But its research had found access to gardens was being used in some wards as a reward for good behaviour - and being denied as a punishment.
Mind is calling for service users to be involved in the design of new hospitals, and for all wards to be maintained to the highest levels of cleanliness.
The charity also said that all patients should have access to green spaces and single sex sleeping accommodation.
Chief executive Paul Farmer said: "Going into hospital with mental health problems can be very traumatic.
"We rely on hospitals to help us get well, but at the moment many are not providing a therapeutic environment.
"We would like to see all mental healthcare environments come up to the standards of the best.
"On many wards there's simply no alternative to, at best, boredom, at worst, fear.
"This can and must be addressed as a matter of urgency."
A Department of Health spokesperson said: "We agree that the hospital environment is a significant factor in aiding the recovery of people with mental health problems.
"That is why we have significantly increased capital investment in mental health services.
"On top of the £1.6bn capital spent by mental health trusts over the past four years, we have made available another £190m to improve patient accommodation."
In a separate survey Mind quizzed office workers in England and Wales to find out how their office environment impacted on their mental wellbeing.
The survey found one in four thought conditions in their office had negatively affected their mental health.
One in five (22%) of respondents had formally complained about their office environment, 42% were dissatisfied with the temperature in their office, 27% were unhappy with the amount of natural light, and 27% were unhappy with the amount of working space they have.