Mr Rammell says the result is not a ringing endoresement of the Conservatives
Labour ministers Angela Smith and Bill Rammell have lost their Essex seats.
Cabinet Office minister Ms Smith lost Basildon South and Thurrock East to the Conservatives, while armed forces minister Mr Rammell lost Harlow.
Stephen Metcalfe won the Basildon South and East Thurrock seat with 19,624 votes, a majority of 5,772 over Ms Smith.
Conservative Robert Halfon took Harlow with 44.9% of the vote, against Mr Rammell's 33.7%.
The Liberal Democrats held Colchester, with Bob Russell receiving 22,151 votes, increasing his majority from 594 to 6,982.
Conservative Will Quince came second with 15,169 votes and Labour's Jordan Newell was third, getting 5,680 votes.
Shadow Europe Minister Mark Francois held Rayleigh & Wickford for the Conservatives, taking 30,257 (57.81%) of the vote.
Lib Dem Susan Gaszczak received 7,919 votes, while Labour's Michael Le-Surf polled 7,577 votes.
The Tories also held Basildon & Billericay, where there was a 9.3% swing from Labour to the Conservatives.
John Baron took the seat with 21,922 votes, compared to Labour candidate Allan Davies' 9,584 votes.
At Witham, Conservative candidate Priti Patel took the seat with 24,448 votes.
Ms Patel is the first Asian Conservative woman elected to parliament.
The Conservatives also took Castle Point and held Clacton and Braintree.
Ms Patel says the theme of change went down well on the doorsteps
Rebecca Harris got 19,806 votes, taking the Castle Point seat from independent Bob Spink, who left the Conservative Party in 2008.
Mr Spink received 12,174 votes, with Labour candidate Julian Ware-Lane receiving 6,609 votes.
In Clacton Conservative Douglas Carswell took 53% of the vote.
Labour's Ivan Henderson, who was MP for Harwich from 1997 to 2005, received 10,799 votes with Liberal Democrat Michael Green in third place.
Conservatives also held Chelmsford, Harwich and North Essex, Maldon, Braintree, Epping Forest, Rochford and Southend East, Southend West.
Conservative Party chairman Eric Pickles was re-elected to Brentwood and Ongar with 56% of the vote.
Conservative Jackie Doyle-Price won Thurrock from Labour's Carl Morris.
In Saffron Walden Conservatives maintained their hold with 72-year-old Sir Alan Haselhurst getting 30,155 votes, more than 50% of the vote - a 4.6% increase on the last election.
Liberal Democrat Peter Wilcock came second with 14,913 votes, and Labour candidate Barbara Light came third with 5,288 votes.