The rest of the panel is made up of writers, including poet Jo Shapcott and novelist Minette Walters, and is chaired by lyricist Sir Tim Rice.
The triple gold winner will join a panel for the Whitbread Book Awards to determine the top titles which will be announced on 23 January.
Oxford graduate Pinsent was in the triumphant coxless fours team in Sydney this summer and joins other well-known figures on the panel, including GMTV newsreader Penny Smith, comedian and Jonathan Creek star Alan Davies and TV and radio presenter Clare Balding.
Last year the Whitbread judging panel was criticised for including model and actress Jerry Hall.
The rest of the panel is made up of writers, including poet Jo Shapcott and novelist Minette Walters, and is chaired by lyricist Sir Tim Rice.
They will judge the Whitbread Book Of The Year and the award for children's book.
The shortlist for the main prize is made up of the winners of the Whitbread children's prize, novel, biography, poetry and first novel awards.
David Reed of Whitbread said: "Each of our panel this year combines excellence in their chosen field with a passion for books."
Heaney's prize
Previous judges have included Jonathan Ross and Jerry Hall.
This year's award was won by Nobel Prize winner and poet Seamus Heaney.
Heaney, from Bellaghy, County Londonderry, pipped children's favourite JK Rowling, creator of Harry Potter, to win the £21,000 prize for his translation of epic Anglo-Saxon poem Beowulf.
The work had already been named the winner of the Whitbread poetry award, beating the late poet laureate Ted Hughes.