A BBC series is asking some of the world's most influential people about the defining moments in their life.
Patten is also part of Romano Prodi's EU Commission
Chris Patten, now the chancellor of Oxford University, is a former member of the UK Government and served as a minister under both Margaret Thatcher and John Major.
After losing his seat in the 1992 general election, he was made governor of Hong Kong until the province was formally handed over to China in 1997.
Arriving in Hong Kong, more than anything else in my life, affected my views - both my views on political life and my views on leadership, not to sound too grand about it.
But I don't think I was noticed arriving so much as I was noticed leaving.
For me I guess I'll always live with the departure and the fact that, as Andy Warhol would have put it, it made me famous for 15 minutes.
I was famous in a way which I think people were able to identify with.
I remember shortly after I had come back from Hong Kong, walking near my house in France, and meeting an old gentleman who asked me where I came from.
I told him the village and he said: 'Ah, have you met the great man from Asia who has just moved in there?'
So I said no, I hadn't. And he said: 'Ah, he was a great man - the governor of Saigon!'
Defining Moments will run until 23 July on BBC World Service's World Today programme. You can also read people's recollections on BBC News Online.