Mr Hollingworth has repeatedly refused to resign
Mr Hollingworth is accused of ignoring sexual abuse in his church while he was Anglican archbishop of Brisbane more than a decade ago.
The timing of the controversy is embarrassing for the government as Queen Elizabeth is due to arrive in Australia next week.
Mr Hollingworth acts as the Queen's official representative and Mr Howard could ask her to remove him from office.
'Errors of judgement'
But the prime minister has refused to bow to calls from the opposition to dismiss the governor-general.
"We are all fallible... there is no man or woman alive who is not guilty of errors of judgment," Mr Howard told reporters.
When he was archbishop Mr Hollingworth was responsible for investigating a number of sex abuse claims against members of the Church.
It transpired that in one case a teacher was sexually abusing students at a church-run boarding school in a city covered by his diocese. The accused teacher, Kevin Guy, committed suicide in prison in 1990 before he could face trial. In a suicide note he admitted abusing 20 girls.
When the allegations about him first came out in December Mr Hollingworth apologised for not taking a more active role - but he has repeatedly said he would not resign.
The row deepened this week following a television programme alleging that he discouraged a victim from pursuing an abuse claim against a priest.
Mr Hollingworth apologised on Thursday for suggesting on that programme that a 14-year-old girl had consented to sex.
He said he had misunderstood and believed the sexual relationship occurred when the woman was an adult.
"I want to make an unreserved apology to the woman concerned and to the whole of the Australian public," Mr Hollingworth said as he left to meet Queen Elizabeth who is arriving in New Zealand on Friday.
Mr Howard said sacking the governor-general would create a constitutional earthquake but experts said that not removing Mr Hollingworth could be equally damaging.
"The entire justification for the constitutional monarchy rests on the idea that it is a symbolic position above politics and won't be subjected to this sort of partisanship," said Professor Elaine Thompson.