By Dominic Casciani
BBC News Online community affairs reporter
Civil servants concluded reforms to help Gypsies were doomed because councillors had no electoral gain in taking on 'popular prejudice'.
Eviction: More than 30 years of stand-offs
Official documents reveal a major 1970s' attempt to provide caravan sites unravelled amid major resistance.
Within a decade Whitehall conceded defeat as battles between travellers and councils continued.
A recent government report says Whitehall has still not given a much-needed lead.
In 1968, MPs won government backing to compel local authorities to provide caravan sites, a huge victory for Gypsy campaigners.
Experts had said land once used by Gypsies was disappearing, increasing the likelihood of conflict with residents and local councils.
The eventual 1968 Caravan Sites Act compelled local councils to provide residential "pitches" - but only if there was a proven need in their area.
According to the documents, placed in the National Archives, Gypsy campaign groups urged officials to force councils to provide sufficient "pitches" or to support willing private landlords as soon as the legislation became law in 1970.
Quick action meant hugely expensive disputes "could swiftly end", said the Gypsy Council.
But unbeknown to the campaigners, some local councils were also pressing Whitehall to do the opposite.
Authorities demanded exemptions, saying they had no Gypsies in their area - although campaigners claimed some councils were unlawfully evicting families as the deadline to provide sites came closer.
At the same time, authorities willing to provide sites said they would resist unless Whitehall guaranteed they would not be left "shouldering the problem" alone.
Right to stay: Gypsies want reforms for their own sites
By 1971 officials diplomatically conceded to campaigners they were "under some pressure" from local councils.
The campaigners were furious. Donald Kenrick of the Gypsy Council warned the government would create "Indian reservations" if it agreed with councils that travelling communities should permanently settle.
"The time surely has come to end this unnatural harassment - this insidious yet pervasive action by the authorities which is undermining a whole way of life," he told officials.
But relations soon completely broke down. The act had still not been implemented and evictions were continuing nationwide.
More and more councils were asking for exemptions while civil servants complained Gypsies themselves had not provided a national headcount of families needing sites.
GYPSY SITES: REFORMS
1960: Caravan Sites Act (restricts land use)
1965: First official count
1968: Caravan Sites Act (compelled councils to provide sites)
1994: Criminal Justice Act (repeals '68 act, introduces new eviction powers)
1994: New guidance to encourage Gypsies to buy own land
2003: Anti-Social Behaviour Act (new eviction powers)
Top officials privately concluded the act had become unworkable as Gypsy traditions were seemingly "incompatible" with modern life.
John Downie, a senior official at the Department for the Environment, told ministers to invest in education for Gypsy children in the hope they would be more likely to give up a nomadic lifestyle.
"Unauthorised encampments continue to grow and continue to cause worry and distress on as wide a scale as ever," said Mr Downie.
"The gypsies, the great majority of whom must perforce camp illegally without sanitary, water or refuse disposal facilities, endure this situation on the whole with rather more fortitude than the public whose sensibilities they offend."
But he added: "A study of the objections [from the public] suggests that they stem largely from a strong and ingrained popular prejudice against Gypsies.
"There is precious little electoral advantage [for councils] in providing a site and there are strong incentives to procrastinate and to hope that the problem will simply go away."
Policy torn up
In 1980 the new government introduced more funding which led to a growth in sites.
Then, in 1994, Parliament repealed the duty to provide pitches, preferring to encourage Gypsies to find, buy and develop their own sites.
But a report last year for Deputy Prime Minister John Prescott estimated the UK still needs up to 2,000 more pitches - and a further network of 2,000 places to facilitate nomadic movement.
It said ministers needed to provide "a strong lead" because there was "no clear, widely understood national policy".
The Office for the Deputy Prime Minister says it has an ongoing review and campaigners hope to get a clause into the current Housing Bill. Ministers have also recently announced £16m to improve existing sites in England.
"We are still pushing for a new duty on local authorities," said Andrew Ryder of the Traveller Law Reform Coalition, the body lobbying MPs.
"The problems are getting worse. There's a shortage of some 3,500 pitches which roughly equates to the number of families living in unauthorised encampments.
"That's not just having a negative impact on their health and life chances.
"It just keeps bringing them into conflict with the settled community."