Our correspondent says the rare public appearance by Mr Mandela is the biggest possible endorsement of Mr Zuma, and conjures up memories of Nelson Mandela's last active campaign in 1994, when South Africa held its first democratic elections.
South Africa's first black president urged the ANC to remember its main task was to eradicate poverty and build a united, non-racial society.
"As we strive to secure a decisive victory for our organisation in the upcoming elections we must remember our primary task. It is to eradicate poverty and ensure a better life for all," he said in a brief pre-recorded message played as he sat on the stage.
"The ANC has the historical responsibility to lead our nation and help build a united non-racial society," he added.
Mr Zuma stressed the message of racial unity in his opening remarks to the rally.
Cope supporters rally in Limpopo
"We reaffirm that South Africa belongs to all of us, black and white," he said.
"Working together we will ensure that no South African ever feels they are less valued than others because of their race, culture or religion."
He dismissed the fears of those who had suggested the emergence of Cope would break the ANC's dominance of South African electoral politics, and pledged the party would remain a force for good if re-elected.
"In 15 years that it has been in power, the ANC has never used its electoral mandate to change the constitution, and it has no intentions to do so.
"We'll always uphold, defend, promote and protect the constitution of our country and all our democratic institutions," Mr Zuma said.
Fears over influence
The ANC is hoping to repeat the success of five years ago, when it secured a two-thirds majority in parliament, but this time, it is going to be a lot tougher, our correspondent says.
On Saturday, thousands attended a Cape Town meeting of the official opposition party, the Democratic Alliance.
Mr Zuma is expected to become South Africa's president
Leader Helen Zille told supporters: "Every vote counts in this election, every single vote, because your vote may be the one that keeps the ANC under that two-thirds majority, and your vote - especially here in this stadium today - your vote may be the very one that wins the Western Cape for the DA."
The DA fears that if the ANC achieves the two-thirds majority, it might use it to change the constitution to influence the independence of the judicial system.
Ms Zille also made frequent reference to criminal charges of corruption and racketeering brought against Mr Zuma which he denies and which were dismissed earlier this month.
The BBC's Andrew Walker, at the ANC rally in Johannesburg, said Mr Zuma made some remarks on the judiciary - suggesting it could undergo changes if the ANC was returned to power.
These would not infringe its independence, but enable it to work more quickly, Mr Zuma said.
The Independent Electoral Commission (IEC) reports that more than 23 million people, including 16,000 of the South African diaspora, have registered to vote in what is being seen as the most important election since the end of apartheid.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.