Difficulty swallowing was one of the symptoms looked at
GPs should be quicker at sending some patients with worrying symptoms for tests, say researchers.
A study of patients with "red flag" symptoms such as coughing up blood showed that half did not have a proper diagnosis three years later.
The researchers said such symptoms indicated serious disease and investigations should be done quickly, especially in older patients.
The study appears in the British Medical Journal.
Researchers used GP records to look at 760,000 adults who had gone to the doctor with blood in the urine, difficulty swallowing, coughing up blood and rectal bleeding.
These "alarm" symptoms are known to be serious and should prompt speedy investigation.
Alongside other signs, they can indicate cancer, but until now it has not been clear how predictive they were of other serious conditions.
The researchers found that 90 days after they had first gone to the doctor around one in five patients with these symptoms was found to have a potentially serious non-cancerous condition such as kidney stones.
However, after three years, over three-quarters of patients presenting with rectal bleeding did not have a definite diagnosis, with comparable figures of 67% for trouble swallowing, 64% for blood in the urine, and 46% for coughing up blood.
They worked out that if tests such as X-rays were done quickly in all patients with these symptoms then for every four to seven patients looked at, one would get a diagnosis of a serious condition within three months.
Instead of a wait-and-see approach, GPs should send such patients for tests quickly, the researchers said.
"GPs in the NHS are seen as the gatekeepers and there is a bit of a feeling that they should restrict access," said study leader Professor Roger Jones.
"This is about investigating the right patients sooner - perhaps have a slightly lower threshold for it than before."
He added that most GPs had direct access to tests such as blood tests and endoscopy without too much of a wait.
If patients are suspected to have cancer, they are supposed to be fast-tracked anyway and seen within two weeks, he added.