The head of the UK's race equality body says low numbers of black and Asian people in the countryside is a form of "passive apartheid".
Mr Phillips said problems stemmed from misunderstanding
Commission for Racial Equality chairman Trevor Phillips said work to stop the countryside being seen as a no-go area for ethnic minorities was crucial.
In many country areas, fewer than 1% of people are from ethnic minorities, compared with an 8% national average.
Mr Phillips was alarmed many feel they do not belong outside towns and cities.
He told the BBC's Today programme: "This is not [something that has happened] by anybody's will, there is no law and I doubt anybody in the countryside wants to keep people out.
"But I think what we are seeing is a gradual drift towards a difficult situation in which people from ethnic minorities feel uncomfortable."
Research by the UK's national tourism boards consistently shows only a tiny fraction of visitors to the countryside are of ethnic minority origin - just 1% in 2002/03.
The Country Land and Business Association say they would like to see more diversity.
Spokesman David Fursdon told the BBC migrants often bring dynamism and originality to can be a staid society and a stagnant economy.
Mr Phillips believes problems stem from mutual misunderstandings between country and city dwellers but acknowledged some people might have exaggerated fear of rural hostility.
Figures and map explaining the ethnic make-up of rural England.
Wilfred Emmanuel-Jones, from west Devon, one of the only black farmers in Britain, said he felt accepted by the community.
But he said black people were often seen as a "novelty" in rural areas and should be prepared for the fact that things were different.
He added: "What I always say to black people who are considering going to the countryside is there are certain things you have to come to terms with.
"For example, down in the countryside they would call you 'coloured' rather than black. If you are going to be offended by that, don't even consider putting on your green wellies."