Mr Hollingworth is due to welcome the Queen next week
Peter Hollingworth has been accused of neglecting or covering up a sex abuse scandal at a girls' school more than a decade ago.
However, with Queen Elizabeth II visiting Australia next week, Mr Hollingworth - as the Queen's representative - is facing a growing chorus of criticism.
A number of government ministers reportedly now believe Mr Hollingworth must resign or at least stand down pending the outcome of the inquiry.
Child protection groups and religious leaders say Mr Hollingworth has lost the public's confidence.
"However justified he might feel... he cannot simply tough it out," said a Sydney Morning Herald editorial. "A governor-general mired in controversy is a dysfunctional governor-general."
Abuse scandal
The allegations relate to when Mr Hollingworth - who only became governor-general last June - was Anglican archbishop of Brisbane.
A teacher was sexually abusing students at a church-run boarding school in a city covered by his diocese. The accused teacher, Kevin Guy, committed suicide in prison in 1990 before he could face trial. In a suicide note he admitted abusing 20 girls.
In December, a court ordered the diocese to pay one of the abused students, now aged 24, $430,000 (Aus $834,000) in damages for failing to care for her.
"That some children in the care of the church and its schools have been abused causes me great pain and shame," said the current archbishop, Phillip Aspinall, announcing the independent inquiry on Tuesday.
In a television interview on Monday he said the scandal would not force him out of office, but said the allegations were putting a strain on his family.
The governor-general acts as head of state on behalf of the British monarch, who is also constitutionally Australia's head of state.
The Queen has distanced herself from the controversy.
A statement issued by Buckingham Palace said she was "obviously well aware of the current situation," but it was a matter for the governor-general to decide, "not us."