The Transport and General Workers' Union (T&G) says it has seen a rise in the number of Eastern Europeans joining in Herefordshire and Worcestershire.
It claims many are seeking protection from bosses and colleagues.
Some say they have been the victims of racism and in some cases have been violently attacked, the union said.
Roger Shutt, from the T&G, says the workers are often working in the lower-paid industries but some of their roles are very important.
"They're doing low-paid jobs, jobs very often other people don't want to do and being treated quite badly by a lot of employers," he said.
"They're very intelligent people. They're young, well-educated...so they're no mugs, they're no pushover."