Chris Iwelumo's penalty was too little too late for Colchester as Ipswich came from behind to win thanks to substitute Danny Haynes' goal.
Karl Duguid converted Iwelumo's cross to open the scoring, before Alan Lee equalised from the penalty spot after he was fouled by Garry Richards.
Sylvain Legwinski headed his fifth goal of the season and when Haynes shot home from 12 yards, the win looked safe.
But David Wright saw red for handball and Iwelumo scored from the spot.
606 DEBATE: Give your thoughts on the match
Ipswich manager Jim Magilton:
"We switched off and conceded a poor goal defensively that jolted us.
"We showed character to come back and on the day just edged it. We are delighted to get the victory."
Colchester manager Geraint Williams:
"On balance of play Ipswich deserved to edge it. We have no excuses.
"To be fair I think Ipswich were more of a threat in the first half and we
were fortunate to be level at the interval.
"It was disappointing to concede a goal from a corner. In fact we could have
done better with all three goals we conceded."
Ipswich: Price, Wright, Bruce, Nalor, Harding, Peters, Legwinski, Williams, Roberts (Richards 90), Lee (O'Callaghan 86), Clarke (Haynes 67).
Subs Not Used: Supple, Wilnis.
Sent Off: Wright (90).
Booked: Clarke, Williams.
Goals: Lee 30 pen, Legwinski 56, Haynes 82.
Colchester: Gerken, Halford, Brown, Richards (White 86), Barker, Garcia, Watson, Izzet (Jackson 77), Duguid, Iwelumo, Ephraim (Guy 63).
Subs Not Used: Davison, Elokobi.
Booked: Halford, Brown.
Goals: Duguid 15, Iwelumo 90 pen.
Ref: G Laws (Tyne & Wear).