Women with denser breasts had a 21% risk of cancer recurrence.
Women treated for breast cancer are at a higher risk of a relapse if they have "dense" breasts, say researchers.
Those with denser breasts are four times more likely to see their cancer return, compared to women with less dense breasts, says the journal Cancer
The Canadian authors advise that women with the densest breasts should have radiotherapy while the women with less dense breasts could be spared.
Experts say further studies are necessary before any treatment change.
The researchers from the Women's College Hospital in Toronto, Canada, looked at the medical records of 335 women who underwent surgery for invasive breast cancer and for whom a pre-treatment mammogram was available.
Patients' breasts were categorised as low density, intermediate or high.
The researchers used the Wolfe scale which classifies breast tissue as seen on a mammogram.
Low density means less than 25% dense tissue, intermediate density means 25-50% dense tissue and high density means more than 50% dense tissue.
Breast density was higher in the younger women in the study, which surveyed women with an average age of 63.5 years.
The women in the high density group experienced a much greater risk of breast cancer returning.
Over a 10 year period, they had a 21% risk of cancer recurrence, compared to a 5% risk for women with the least dense breasts.
After 10 years, the difference in the rates of disease recurrence for women who did not receive radiotherapy after surgery was more pronounced, with 40% of women with high density breasts experiencing cancer recurrence compared with none of the women with the lowest density of breasts.
The authors said their findings indicated that women with low breast density, may not need radiotherapy, but that women with high breast density could significantly benefit from the therapy.
'Breast density modifiable'
The researchers believe the hormonal profile of denser breast tissue makes it more susceptible to cancer.
Dr Steven Narod, of the Women's College Hospital in Toronto, Canada, said: "The composition of the breast tissue surrounding the breast cancer is important in predicting whether or not a breast cancer will return after surgery.
"Breast density has been found to be modifiable to some extent by physical activity and hormone therapy."
Stephen Duffy, Cancer Research UK's professor of screening, said: "This is intriguing.
"One reason for this could be that we know breast density increases the risk of a breast cancer in the first place, so it may simply increase the risk of a second one.
"Another possibility is that the dense tissue makes it more likely that other areas of cancer in the breast are not visible at initial diagnosis and so are not removed when surgery takes place.
"The potential to spare women who do not have dense tissue from radiotherapy is a very attractive prospect. But further studies will be necessary to be sure that this is safe practice."
Maria Leadbetter, clinical nurse specialist at Breast Cancer Care, said: "By being able to determine which women may or may not benefit from radiotherapy it may be possible to offer more tailored treatment.
"It could also reduce the total number of women having radiotherapy, which can have significant long-term side effects."