She was in labour for nearly 12 hours at the hospital, where she herself was born, before giving birth to their 8lb 13ozs son at 1658 GMT on Friday.
Former model Miss Noble, 29, said she was looking forward to taking on motherhood and changing nappies as the couple left Queen Mary's Hospital, in Sidcup, Kent, on Saturday.
She was in labour for nearly 12 hours at the hospital, where she herself was born, before giving birth to their 8lb 13ozs son at 1658 GMT on Friday.
"He looks gorgeous and looks like his dad," she said.
But asked whether Harrison was the first of many, she replied:
"Let's wait and see, we just want to get this one home."
John Major, who spent an hour with his first grandchild on Friday, said: "It feels absolutely wonderful.
Big baby
"He is just absolutely perfect, he is a lovely looking baby and a big baby.
"James and Emma are absolutely beside themselves with pleasure.
"I am just so thrilled that he is here and he is safe and absolutely lovely."
The former premier's wife, Dame Norma Major visited on Saturday.
Miss Noble's mother Joan, from Sidcup, said she was "overjoyed" as she left hospital.
"It is a beautiful baby and just absolutely wonderful."
Conservative Party leader William Hague said: "Ffion and I send our warmest congratulations to John and Norma on becoming grandparents for the first time."
Miss Noble, who co-hosted Bruce Forsyth's Price is Right and starred in the Ben Elton play Popcorn, married James Major in May last year in the crypt chapel of the Houses of Parliament.
The 25-year-old former prime minister's son and night-club promoter was himself taken to hospital in July last year and had a heart pacemaker fitted.