Supermarket firms Sainsbury's and Asda have admitted that they were part of a dairy price-fixing group that earned about £270m extra from shoppers.
Sainsbury's said it was pleased to reach a settlement.
The supermarkets, along with a number of dairy firms, have agreed to pay fines totalling some £116m after an Office of Fair Trading (OFT) probe.
Cases against Tesco and Morrisons will continue after no deal was struck.
The OFT said that in-store prices went up after the collusion, but the amount received by farmers did not increase.
However, the firms insist that the farm gate price paid for milk did rise and that they were not ripping off customers.
The OFT said the collusion saw customers being charged 3 pence extra for a pint of milk, 15p extra per quarter-pound of butter and 15p per half-pound of cheese, the watchdog said.
In September, the OFT provisionally found evidence of collusion by 10 firms to set prices during 2002 and 2003.
Now Sainsbury's and Asda have admitted price-fixing of milk and cheese, as has Safeway - before it was bought by Morrisons.
Safeway has also admitted colluding on the price of butter.
Dairy processors Dairy Crest, the Cheese Company and Wiseman also reached agreement with the watchdog.
Having admitted liability in principle, they now face smaller fines as a result, the OFT has said.
Another dairy firm, Arla, has been given immunity from fines if it continues to co-operate with the investigation.
But Tesco, Morrisons and the dairy firm Lactalis McLelland are challenging the OFT findings.
Sainsbury's, which has agreed to pay a £26m fine, said its price initiatives in 2002 and 2003 "were designed to help British dairy farmers at a time of considerable economic pressure and public debate about whether farmers were getting a fair price for their products".
"We shared what we intended to do in a very public way as far as retail prices were concerned with our suppliers. They shared that with some of our competitors, and it's that that broke the rules," chief executive Justin King told the BBC.
"There is no suggestion that what took place was an attempt to make more profit. In fact what actually happened was that higher prices were paid to farmers."
An Asda spokesman said there were "regrets" at the firm, but that its intention "was to provide more money for dairy farmers, who were under severe financial pressure at the time".
Meanwhile, Tesco said it was "vigourously" defending its position, which it said was different from its rivals.
"As we have always said, we acted independently and we did not collude with anyone," said executive director Lucy Neville-Rolfe.
Dairy Crest, which will pay £9.4m, said that the pricing policy occurred in the aftermath of the foot-and-mouth outbreak.
"The implementation of these initiatives was very well publicised at the time and received widespread support including strong political backing," it said.
Wiseman Dairies, which faces a £6.1m bill, said that "every penny of additional revenues paid to Wiseman was passed directly to our suppliers".
"Whilst we are disappointed with the outcome of the investigation, this agreement removes a long shadow hanging over the company," chairman Alan Wiseman said.
The National Farmers' Union (NFU) did not want to comment on the OFT announcement.
"This was a period of time that was extremely difficult for dairy farmers when prices were very low for a sustained period and there was great pressure on dairy companies and retailers for a farm gate price increase," an NFU spokesman said.
"We are pleased to say that the price for milk is now significantly higher, largely due to changes in the global market."