The Royal College of Physicians has called for a big increase in the number of doctors in England who specialise in hearing and balance problems.
Hearing and balance disorders are very widespread
The RCP says half of people suffer the disorders at some time, but can wait years to see a specialist and often remain undiagnosed and poorly managed.
Specialist services are only available in a few centres, with no provision in most of the country, the body says.
The Department of Health for England says waiting times are falling.
Balance disorders alone, which are often associated with major illness such as cardiovascular disease, are estimated to affect up to 40% of the population by the age of 60.
The RCP report said patients who should be referred to an audiovestibular physician supported by a specialist multidisciplinary team often have to make do with a specialist from another field due to lack of available specialist services locally.
It said the cost to the health service of these failings is enormous, and for patients it may mean lost education or jobs, and huge psychological damage
The RCP is calling for a big increase in audiovestibular specialists, a greater emphasis on the specialty in medical training, and the creation of more multidisciplinary specialist support teams.
It also wants a network of services in primary care designed to offer patient's high quality care close to their home, and where necessary refer on rapidly to specialist units.
Professor Linda Luxon, chair of the RCP working party and an expert in audiological medicine at the University of London, said hearing and balance disorders often had a serious impact on quality of life.
However, because they were not life threatening, they had not been deemed to be a priority.
"Adequate medical and scientific resources for accurate diagnosis and optimal management of patients with complex hearing conditions and, particularly, balance disorders remain woefully inadequate throughout most of the UK."
Professor Ian Gilmore, RCP president, said: "Hearing loss and balance disorders are hidden handicaps with profound personal, occupational and economic consequences.
"Despite at least half the population suffering such symptoms during their lifetime, people with these debilitating conditions do not get the priority service they deserve."
Jackie Ballard, chief executive of the charity RNID, said: "Patients with hearing loss can suffer from depression, isolation and can find it hard to get by in work and social situations.
"RNID knows of areas where the waiting time for hearing aids is up to two-and-a-half years - so it comes as no surprise that patients are also waiting years to see hearing and balance disorder specialists."
Health minister Ivan Lewis said: "The Department of Health recognises the importance of early access to both diagnostic assessment and treatment for hearing difficulties.
"Waiting times for audiology assessment and treatment are falling.
"And we have made it clear to the NHS that by the end of 2008 we expect the vast majority of patients to be waiting no more than 18 weeks from referral to the fitting of the hearing aid."