Credit and debit card fees are pushing up retail prices by as much as 2.5% in some EU markets, according to the European Commission.
Card fees are adding to inflation
Depending on where customers live in the EU, they could pay up to double in annual fees for a Visa or Mastercard.
Likewise, the fee banks levy on business for processing a credit card transaction - also varies widely across the EU.
The EU Competition Commission said the card market lacked competition.
"The payment cards industry in Europe remains national and some local players are preventing competition from developing," said Neelie Kroes, the EU's competition commissioner.
Ms Kroes said she was "fed-up" with some banks and card providers taking advantage of consumers.
The findings were part of an ongoing EU investigation into competition in the financial services sector.
Ultimately, if debit and credit card providers are found to have breached EU law, they can be fined.
"Let me be clear, European competition law provides a series of powerful tools to bring about more competition," Ms Kroes warned, adding that the "paradise days" were over for some lenders.
In a statement, Visa Europe said that it had to examine the contents of the EU's interim report before commenting further.