Insurer Guardian Assurance and its associated company Guardian Linked Life have been fined £750,000 for mishandling endowment complaints.
The fine is the fourth imposed by the FSA for mishandling complaints.
It is the fourth time that the Financial Services Authority (FSA) has fined an insurance company for not dealing with complaints properly.
The FSA said Guardian's complaints procedure had "serious systemic flaws".
As a result, 5,600 customers had their complaints wrongly rejected, and thus could have lost out on compensation.
Margaret Cole, the FSA's Director of Enforcement, said: "Guardian failed to treat its customers fairly by exposing those with a valid complaint to the risk that their complaint could be rejected inappropriately.
"Consequently, they may not have received the compensation to which they were entitled.
"These failings exposed a high number of consumers to potential financial loss," she added.
Pressure on insurers
Since 2000 the FSA has been putting pressure on insurance companies to sort out how many customers may have been mis-sold mortgage endowment policies, mainly in the 1980s and the 1990s, and to pay compensation where this has happened.
PREVIOUS FSA FINES FOR MISHANDLING ENDOWMENT COMPLAINTS
Friends Provident, December 2003, £675,000.
Dunbar Assurance, March 2004, £725,000
Abbey National, May 2005, £800,0000
Between 1988 and 1995 Guardian sold 233,000 such policies before it stopped marketing them.
It received nearly 20,000 complaints from April 2000 to the end of 2004.
As they built up, the company introduced a streamlined system for dealing with them, in order to cut costs.
This meant that between January 2003 and the end of 2004, 5,600 complaints were improperly rejected out of a total of nearly 12,000 received during that time.
At one stage it was rejecting more than three quarters of all its complaints.
The FSA said that the procedure was neither appropriate nor effective.
"The procedure led to complaints being dismissed if the customer understood and accepted the risk that the policy might not repay the sum assured - but there could have been other reasons for accepting the complaint," an FSA spokesman said.
Guardian was told by management consultants that it was not looking at all aspects of the complaints, so it changed its method again in 2004, though not sufficiently to comply with the FSA's guidelines.
The company has now agreed to a thorough review of the rejected complaints and to make good any losses suffered by its customers.