Many new fathers are refused their right to two weeks paid paternity leave because they have not given their employers enough notice, a report says.
New fathers are already entitled to two weeks off work
According to the Citizens Advice Bureau (CAB), many men do not realise they have to give 15 weeks notice - the same as women planning maternity leave.
The charity urged longer, better paid paternity leave for fathers, with more choice over when it could be taken.
The government says firms have a right to notice so they can cover leave.
The CAB gave the example of a man from Staffordshire who was denied paternity leave after giving 13 weeks notice.
And a father from Sussex, who had worked for his firm for 12 years, was denied time off after giving a month's notice, it said.
The charity urged the government to give fathers better-paid, longer periods of paternity leave and more choice over when they can take time off.
It described plans contained in the Work and Families Bill to enable fathers to use some of their partners' maternity leave as well-intentioned but fundamentally flawed.
CAB chief executive David Harker said the plans would help many families but 40% of working fathers would not benefit and it was a problem that both parents could not choose to take leave at the same time.
He also said the planned change would make the law too complicated for small employers to cope with.
He added: "The government should abandon its well-intentioned but flawed plans on paternity leave and consider instead how it might best enhance the individual rights of working men to take time off work to be with and care for their children at a time of their choosing."
A spokeswoman for the Department of Trade and Industry said Equal Opportunities Commission research had found awareness of their rights among fathers was "relatively high".
And she said the Work and Families Bill would give fathers the right to more time off and more pay when children were born.
She added: "Along with the father's right to leave, employers also have the right to know when their staff will be away from work so that they can plan for that period.
"This is why fathers, like mothers, need to give their employer 15 weeks notice."
But Duncan Fisher, chief executive of fatherhood information centre Fathers Direct, said: "It is a terrible shame that mothers and babies are missing out on having Dad around at such a crucial time after the birth because the regulations require such an unnecessarily long period of notice for paternity leave.
"Families need Dad at this time."