More than 2.5 million men in Britain could have low fertility, a report says.
Sperm number and quality are important for male fertility
The research by Norwich Union Healthcare estimates that 9% of the UK's 28.5 million men might have difficulty conceiving.
Of GPs questioned, a third were worried declining male fertility would add to ageing population issues. Many blamed smoking and alcohol for the trend.
Sperm quality and number have declined in the last 30 years, work suggests.
Male infertility accounts for about a third of couples' problems conceiving.
More than a quarter (29%) of the 202 GPs surveyed said that low male fertility would have a detrimental impact on the nation's population in the future unless men changed their bad lifestyles now.
Nearly half (44%) said smoking was the overwhelming main contributing factor to low male fertility. One in 10 (11%) blamed alcohol and 7% blamed stress.
However, other factors could also be involved. For example, many couples are postponing having children to their thirties when their fertility has naturally decreased.
GP Ann Robinson said: "The results of this survey are shocking and should be a wake-up call to men and women that drinking and smoking too much not only gives you a bad headache in the morning but can affect your ability to start a family."
Over half of the 797 men surveyed by Norwich Union Healthcare believed women were most likely to suffer from fertility difficulties and only 12% viewed their own fertility as a concern when thinking about starting a family.
Cigarettes and Alcohol
Half of men were more concerned with the financial implications of raising a child than their ability to father one.
Dr Doug Wright, clinical spokesman for Norwich Union Healthcare, said: "The report clearly disproves the common misconception that women are the only sufferers of fertility issues.
"With the next generation facing increasing pressure as a result of declining fertility and mortality rates, it's only fair that men accept their responsibility in the equation and change their lifestyles now before it's too late."
Clare Brown, chief executive of Infertility Network UK, said: "It is commonly thought that women are the only sufferers of fertility issues but in around one third of couples who suffer from infertility, it is a male factor problem which is the issue.
"We would encourage men as well as women to consider their fertility and adopt a healthy lifestyle, even before they consider trying to conceive.
"Discovering you have a fertility problem has a devastating effect on people's lives and any steps which couples can take to minimise the chances of suffering infertility such as giving up smoking and cutting down on alcohol should be encouraged."