Ministers are being urged to rethink the way they are trying to persuade pensioners to collect their cash from banks rather than post offices.
Royal Mail say post offices are not generating enough money
Martin O'Neill, chairman of the Trade and Industry Select Committee, says the changes are not always appropriate for people on the "economic margins".
The MPs are also concerned about plans to close down a number of post offices.
The Royal Mail has defended its plans saying that many post offices are under used and not generating enough money.
Since 1 April this year, pensioners and those on benefits have been faced with a change in the way they receive money from the government.
They can either have the money paid into their bank accounts or apply for a Post Office card account where they are given a card and pin number and claim the cash from the post office.
The National Federation of Sub postmasters, which represents the majority of sub post office owners in the UK, claims there is a campaign to force customers into banks and building societies.
However, Consumer Minister Gerry Sutcliffe said people were simply opting not to use post offices - but he agreed action must be taken.
"People have simply decided not to use the Post Office and the business must face up to this and make itself more relevant to modern needs," he said.
"People can still get their pensions from the Post Office across the counter," he said, before accusing opposition MPs of "scaremongering".
Mr O'Neill, whose committee has been looking into the changes, argued that people living on the "economic margins" have no savings, bank accounts or credit cards.
"For them the attractions of a hole in the wall is merely a hole in the wall
and they don't have access to ATMs," he said.
He rounded on the Department for Work and Pensions, which introduced the changes, accusing it of having "malign motives".
He said the tone of internal memos, uncovered by the National Federation of Sub postmasters, were "not the tones of people who are being instructed to help the disadvantaged".
The objective was not to give people an "infinite choice", he said.
Statistics showed that there were a lot of people applying for the Post Office card account, but some were dropping off at the first stage of the application process.
The watchdog PostWatch suggested that there "will be a sizeable several million people ... who will be in considerable difficulty ... they will not have a card, they will not have a bank account, they will be left in some kind of limbo", Mr O'Neill said.
He pointed to the human factors that took people to the post office, adding that for some "it is an outing".
"The pensioner meets their friends, their contemporaries and they can buy their specific post office goods, like stamps for their television licences," he said.
The poor and the disadvantaged were "entitled to have their views heard on how their benefits should be paid", Mr O'Neill insisted.
"On the issue of direct payment, the government is paying attention to neither the many nor the few."
Tory MP Patrick McLoughlin said he was "shocked and horrified" to find out that four out of five post offices in one area in his constituency were set to close.
He said the government was forcing people not to go to the Post Office.
Labour's Rachel Squire received cross party cheers when she stressed: "I do worry about just what pressure has been put on these sub postmasters and mistresses to agree that the closure of their sub post office should go ahead and the at least implied threat they may lose the compensation offered."
Colin Baker, general secretary of the National Federation of Sub postmasters, said his members had "no problem" with the concept of direct payment.
"But our investigations have confirmed what we already knew, namely that there is an active campaign to force our customers out of our post offices and into banks and building societies," he said.
"We believe that our customers should be able to choose where they can receive their direct payments. It is now clear that the Department for Work and Pensions does not share this view."