Older people may find it harder in future to get face-to-face tax help, says the Shadow Paymaster General and the union representing Revenue and Customs staff.
HMRC is cutting thousands of jobs and closing 29 offices
More than two million people over 60 - including many on relatively low incomes - have to fill in a self-assessment tax return every year.
In the past many have been able to get advice in person at their tax office or even in their own home.
But as part of a major restructuring, HM Revenue and Customs is cutting 12,500 jobs and closing 29 tax offices.
Critics fear this could result in older people having to rely more heavily on telephone advice, which is not always appropriate for their circumstances.
Mark Francois MP speaks on Revenue and Customs matters for the Conservative Party.
He told BBC Radio 4's Money Box programme: "Quite a lot of local tax offices are closing.
"What that means is senior citizens who maybe would like to sit down and go through their tax return face-to-face with someone are having to do it over the phone."
Mark Francois MP is worried some senior citizens will suffer
He added: "What we don't want to develop is a system that is so remote that people who really do need to sit down and go through their tax affairs would no longer be able to do that".
His concerns are shared by Ian Lawrence from the Public and Commercial Services Union, which has 83,000 Revenue and Customs employees amongst its members.
"Before, if someone was clearly unable to reach a local enquiry centre, a member of staff would go and visit them and try to assist them. Now what we've seen is a complete diminution of that service."
Revenue and Customs could not tell the programme whether the number of home visits made by staff has declined, but insisted the option remains available to a range of people, including older taxpayers.
It said face-to face advice will continue to be offered at enquiry centres close to the offices which have been earmarked for closure.
But there is an alternative source of help for pensioners who are struggling with their tax return.
Tax Help for Older People, or TOP, is a charity which provides free advice to anyone over 60 with a household income of less than £15,000.
After a series of successful pilots, the service has now been expanded across the UK.
Top offers a free tax advice service for older people on low incomes
The charity's call centre in Dorset offers initial advice over the telephone but can refer taxpayers to more than 300 tax advisers in England, Wales, Scotland and Northern Ireland.
Face-to-face advice can be given at one of 223 local surgeries or in the taxpayers' own home. More than 6,000 people have been helped in the last 12 months.
The service is supported by the tax profession, the Big Lottery Fund and HM Revenue and Customs itself.
TOP director Paddy Millard says his service is invaluable to older people, many of whom are on a low income but who still have to navigate their way through the complex self-assessment maze.
"Once you've retired you lose the support of your personnel department and colleagues at work, you're on your own," he told the programme.
"You've never had contact with the tax office before, you didn't need to. Now you have lots of little sources of income, you don't know how the system works and you need professional help.
"TOP provides free professional advice to those who can't afford to pay for it."
BBC Radio 4's Money Box was broadcast on Saturday, 21 October 2006, at 1204 BST and was repeated on Sunday, 22 October at 2102 BST.