Archbishop Carey told the BBC that many Christians in the state had left in despair because they were unable to build churches or teach religion in schools.
He said he had expressed his concerns over Christians' rights to the governor of Zamfara state, Ahmed Sani.
Zamfara was the first Nigerian state to impose the Islamic legal code, or Sharia, which includes punishments such as stoning to death, amputation and flogging.
"I pressed the point quite strongly, and said I wished to be a voice for all Christians who feel excluded and marginalised," the archbishop said about his meeting with Mr Sani.
'Lively' discussion
"His response was to say that he felt as governor of the state he was very concerned," Dr Carey said.
The archbishop added that Mr Sani had courteously listened to his complaints and described the meeting as "lively". The archbishop is due to visit Abuja on Monday.
According to the Muslims in Zamfara, the fear of Sharia punishments has already made for a better society.
Men and women - of all faiths - are now prohibited from sharing public transport in Zamfara. Boys and girls are taught in separate schools and the sale and consumption of alcohol has been banned.
Bloodshed
Attempts to introduce Sharia in the neighbouring state of Kaduna - with its much larger Christian population - led to terrible bloodshed last year.
At least 2,000 people died in fighting between Christians and Moslems and Nigeria's Christian minority says the fear of Sharia is driving their numbers even lower.
However, Dr Carey told the BBC: "The Christians say they have been living with Muslims for many years and believe its not religion which is the issue."
He added that he had spoken to people during his visit who feel that the religious division of Nigerian society is being driven by politicians seeking to capitalise on the situation.