The cartel practice involved the use of false invoices.
Construction giants Balfour Beatty and Carillion are among those the OFT accuses of taking part.
'Cancer'
"Cartel activity of the type alleged today harms the economy by distorting competition and keeping prices artificially high," said OFT chief executive John Fingleton.
Office of Fair Trading explains how the 'rigging' worked
"This investigation, together with the OFT's previous decisions in the roofing sector, will hopefully send out a strong message to the construction industry about the seriousness with which we view suspected anti-competitive behaviour.
"Businesses have no excuse for not knowing and abiding by the law."
The OFT said its investigations first started in 2004 following an initial complaint regarding building contracts at Nottingham's Queens Medical Centre.
And on public projects, it was council tax-payers who were among the victims of the collusion, said Edward Walsh of the Local Government Association.
"We'll be seeking to find out whether or not councils will get any of this money back in the shape of fines," Mr Walsh said.
"Ultimately, council tax payers have had to pick up the bill if collusion has taken place."
Former quantity surveyor Bryan Rylands told the BBC that he left the industry because he was so disillusioned by the scale of the problem of price-fixing.
"It continues from the self-employed builder right through to your major construction companies that are doing work here in the UK and overseas," he said.
"It's extensive, I mean it is a cancer, it's not benign, it is rife."
Time to respond
Balfour Beatty confirmed that it had applied for leniency.
It said in a statement that it had now reviewed its practices and was "confident that all of its subsidiaries are now fully compliant".
"Balfour Beatty neither promotes nor condones anti-competitive behaviour," it said.
Carillion said in a statement that the OFT's investigations only applied to its Carillion JM subsidiary, formerly separate company Mowlem.
This was bought by Carillion in February 2006.
Carillion added that the OFT had granted Carillion JM leniency.
An OFT spokesman said its accusations, issued in a Statement of Objections, will now go to the companies concerned.
They will then have a number of months in which to respond.
Any firm found to be a member of a cartel can be fined up to 10% of its annual turnover, but penalties are reduced for those who co-operate with an investigation.
Bookmark with:
What are these?