The society has ordained priests in defiance of the Vatican
Vatican officials are meeting leaders of a breakaway group of traditionalist Catholics in an attempt to heal a deep theological divide.
The Society of Saint Pius X, comprising some 500 clergy, opposes Catholic reforms introduced in the early 1960s.
In January this year, Pope Benedict XVI revoked the 21-year excommunication of four bishops in the Swiss-based group.
But one of them, Bishop Richard Williamson, is a Holocaust denier - and Jews condemned the Pope's move.
The talks at the Vatican are expected to last months, possibly years.
The Society of St Pius X (SSPX) split from the Vatican in 1988. It was founded by a French archbishop, Marcel Lefebvre, in 1970 as a protest against the liberal reforms of the Second Vatican Council.
The group wants to revert to the Latin mass and the practices of Catholicism before the Council, which was held between 1960 and 1964.
Some liberal Catholics fear that Pope Benedict is being unduly sympathetic to the traditionalists who have disrupted the Vatican's continuing dialogue with Jews, the BBC's David Willey reports from Rome.
The Pope says he wants to restore unity in the Church.
But our correspondent says it is difficult to see how the two sides can reach any agreement when they disagree so fundamentally about how best to uphold true Catholic tradition.