His body was found with facial and head injuries on a bed, after he was apparently attacked with a machete and a wooden club.
Mr Terreblanche came to prominence in the early 1980s, campaigning for a separate white homeland and championing a tiny minority determined to preserve apartheid.
The BBC's Karen Allen in Johannesburg says the murder comes amid growing anxiety about crime in South Africa.
Opposition politicians linked the death to racially inflammatory sentiments from a minority of the ruling ANC party.
During a news conference in Ventersdorp on Sunday, South African Police Minister Nathi Mthethwa cautioned against stoking tensions.
"We call on all South Africans across whatever divide - across the racial divide, across the political divide - to desist from making any inflammatory statements which are not going to help anyway on the case we are dealing with," he said.
Meanwhile, relatives and friends of Mr Terreblanche gathered near his home to pay their respects.
His Afrikaner Weerstandsbeweging (Afrikaner Resistance Movement - AWB) echoed Mr Zuma's call for calm.
AWB spokesman Andre Visagie said: "We will decide upon the action we are going to take to avenge Mr Terreblanche's death," adding that next steps would await a party meeting in May.
He earlier blamed the killing on the recent singing of an apartheid-era song - "kill the Boer, kill the farmer" - by a firebrand ANC leader.
Martin Plaut, Africa editor
For most South Africans, Eugene Terreblanche was a throwback to another era. But his death is a blow to the country's image of racial tolerance, fostered so carefully by Nelson Mandela.
Some are likely to believe that the fact that his alleged attackers were arrested so rapidly smacks of a cover-up. Others, on the minority far-right fringe, will see his death as a vindication of their assertion that whites cannot live under black rule.
It is a tragic fact that more than 3,000 white farmers have been murdered since the end of apartheid in 1994. And it is possible that some people may seek retribution.
Mr Terreblanche's funeral could become a rallying point for such sentiment.
ANC spokesman Jackson Mthembu said he did not think Mr Terreblanche's murder was politically motivated.
"No one can bring any evidence... linking this song to the death of Mr Terreblanche," he told the BBC.
Last week, South Africa's High Court banned ANC Youth League leader Julius Malema from singing "kill the Boer". It ruled the song was hate speech, but the ANC is appealing.
Boer is Afrikaans for a farmer, but is sometimes used as a disparaging term for any white in South Africa.
A spokeswoman for the opposition Democratic Alliance party, Juanita Terblanche, no relation, said: "[Mr Terreblanche's killing] happened in a province where racial tension in the rural farming community is increasingly being fuelled by irresponsible racist utterances."
The minority party Freedom Front Plus called on people to refrain from reacting emotionally.
"The murder creates an explosive situation and is condemned in the strongest possible terms," party spokesman Pieter Groenewald said.
More than 3,000 white farmers are estimated to have been murdered since the end of apartheid in 1994.
A committee of inquiry found in 2003 only 2% of farm attacks had a political or racial motive, although critics said this figure was far too low.
Mr Terreblanche had founded the white supremacist AWB in 1973, to oppose what he regarded as the liberal policies of the then-South African government.
His party tried terrorist tactics and threatened civil war in the run-up to South Africa's first democratic elections, before sliding into relative obscurity.
Mr Terreblanche served three years in jail after being convicted in 2001 of the attempted murder of a farm worker.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.