The study reveals that the fewer orgasms a man has the greater his risk of being diagnosed with the disease.
Although breast cancer is rare in men, the mortality rate is high: around 250 cases are reported in the UK each year, with around 100 deaths.
Researchers in Greece compared the lifestyles of 23 men with breast cancer with 76 healthy men.
They found that men who had fewer than six orgasms per month were significantly more likely to develop breast cancer.
Writing in the British Journal of Cancer, they also concluded that first born sons are at greater risk of breast cancer than younger brothers.
Like breast cancer in women, the assumption is there is a strong link with the female hormone oestrogen.
Women produce higher levels of oestrogen during a first pregnancy making it more likely the baby will have greater exposure to the hormone.
Orgasm link
And researchers who carried out the study in Greece have also confirmed for the first time the link with male hormones and male orgasms.
Professor Dimitrios Trichopoulos, of the Harvard University of Massachusetts, said: "The frequency of orgasms is thought to be related to testosterone, which drives men's sexual behaviour."
Dr Ian Banks, chairman of the Men's Health Forum in the UK, said that although male breast cancer is rare there is a problem with awareness.
"Men don't even think of themselves as having breast tissue so they don't believe they can get breast cancer.
"And the tragedy is there is a real problem with stigma. I had a patient with breast cancer who was a lorry driver and he committed suicide - he felt how could he tell his mates something like that?"
Dr Banks points to other recent research linking male fertility with certain forms of cancer.
"All the research about hormone imbalances and cancer risk is starting to come together so these findings are not as surprising as you might think," he added.
Professor Gordon McVie, director general of the Cancer Research Campaign said: "This research will hopefully raise awareness of a disease which, although rare in men, often proves difficult to treat."
His views were echoed by chief executive of Breakthrough Breast Cancer, Delyth Morgan.
"This issue should not be trivialised. The disease can have a devastating impact on all those affected and those close to them," she said.