More than one million people from the EU's eight eastern European states have visited the UK since their countries joined last May, figures show.
Many eastern Europeans work in the UK's agricultural sector
Around 90,000 said they intended to stay longer than three months.
The Conservatives have criticised the disparity with a previous government estimate of up to 13,000 net migration.
But the Home Office said Wednesday's figures from the Office for National Statistics focused mainly on arrivals and not migration.
"The ONS' International Passenger Survey is not measuring migration - only arrivals.
"The 5,000-13,000 estimate related to net migration - inflows minus outflows - and to longer-term migrants, those coming for one year or more," a Home Office spokesman said.
Visitors from the eight former Soviet bloc countries represented 9% of the total number of visitors from the EU and 91% of them came for short-term visits, he added.
Immigration and asylum-monitoring group Migrationwatch UK welcomed the influx of eastern Europeans but said their arrival must be counter-balanced by restricting other immigration.
"The eastern Europeans now have a right to come but we cannot have immigration from all over the world," Migrationwatch's chair Sir Andrew Green said.
The ONS figures showed the number of visitors from the so-called "A8 states" - Poland, the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Slovakia and Slovenia - decreased as the year progressed, falling to 315,000 during the last quarter of 2004 compared to 502,000 for the previous three months.
Nearly half of A8 visitors said their stay was for non-leisure purposes, including work and study, compared to 37% for other EU visitors.
The UK, like the other countries in the EU prior to last May, placed restrictions on east European workers when their countries joined.
Migrant workers from those countries are required to register with the Home Office and are not allowed to claim benefits until they have worked continuously for one year.
A TUC study last November found eastern European workers had settled mainly in rural areas of the UK in response to labour shortages in the agricultural sector.
Only a quarter had migrated to London and other urban areas, reversing the trend of earlier waves of migration.
Sixty percent of workers from the new Europe were Polish, the TUC study found.