The Royal Mail claims it is losing five pence for every First Class stamped letter, leading to an overall loss of £247m in the last financial year.
These pillar boxes are a drag on Royal Mail's profit margins
The figure is even higher for second class mail, at nine pence a letter.
The losses were offset only by the profits generated by bulk business mail, said the Royal Mail's chief executive, Adam Crozier.
It added that it was in talks with regulator Postcomm about linking stamp prices more closely with costs.
Stamps went up by 1p in May, but prices are capped under a formula agreed with
Postcomm, with the next adjustment due next April.
The Royal Mail told News Online that the findings did not mean a rise in stamp prices was imminent.
But Mr Crozier said it was clear prices would have to be "rebalanced" as competition intensified, which could mean prices actually rising.
The Royal Mail recently announced that it was in the black for the first time in four years, posting a £220m profit from its day-to-day business.
First class stamps (28p) and second class stamps (21p) prices in the UK are among the lowest in Europe, the company said.
"But this is only achieved by cross-subsidy by the customers of bulk and franked mail," said Mr Crozier.
Losing money on stamps means the return on its £8.6bn turnover was just 2.5% compared with margins of 20% generated by other European postal operators.
"Royal Mail can't keep sustaining heavy losses on stamped mail as competition intensifies," said Mr Crozier.
"We are determined to charge a fair price, and that means having prices much more related to the actual cost to Royal Mail of providing the service."
The Royal Mail handles 82 million items of post every day from 120,000 post boxes, delivering to 27 million homes.