Summerfield powered in a 25-yard shot after a poor clearance, although the goal came against the run of play.
The home side had seen Richard Garcia thunder a 25-yard half-volley against the crossbar before Karl Duguid's volley was cleared by Barry Hayles.
Johnnie Jackson went close to an equaliser with a 20-yard shot but Plymouth held on for a hard-fought win.
Plymouth boss Ian Holloway:
"When you think how far the fans have come to watch a Tuesday night game it doesn't really make sense and I'd like to thank them all.
"I was delighted with the win. Colchester have a right to be in this division and they have done well.
"The boys defended very well and it was a fantastic strike to win it for us."
Colchester: Davison, Halford, Baldwin (Richards 55), Brown, Elokobi, Izzet (Iwelumo 54), Watson, Jackson, Duguid, Garcia, Cureton.
Subs Not Used: White, Gerken, Guy.
Booked: Elokobi, Garcia.
Plymouth: McCormick, Connolly, Aljofree, Doumbe, Capaldi (Djordjic 77), Norris, Summerfield (Sawyer 62), Wotton, Hodges, Hayles, Chadwick (Ebanks-Blake 59).
Subs Not Used: Barness, Reid.
Booked: Connolly, Hayles, Capaldi, McCormick.
Goals: Summerfield 30.
Ref: R Booth (Nottinghamshire).