The first two sections cover people deported to forced labour camps in Germany.
The DM10bn ($5bn) fund, financed equally by the German government and industry, will provide compensation for hundreds of thousands of Jews, eastern Europeans and former prisoners of war.
Moral responsibility
The agreement follows more than a year of negotiations between Germany, the United States and representatives of the victims.
German Foreign Minister Joschka Fischer said the fund meant his country was facing up to its moral responsibility towards Nazi victims.
But US deputy treasury secretary, Stuart Eizenstat, who played a major part in negotiating the settlement, said the fund would not end Germany's moral responsibility for the holocaust.
"Nothing can erase the memory of those who died or the suffering of those who survived," he stated, going on to highlight the cultural impact of the forced labour system.
The fund divides victims into three categories and awards different amounts to different groups.
The first two sections cover people deported to forced labour camps in Germany.
Payment is according to the conditions survivors were kept under, which varied from ghettoes to concentration camps.
The other group includes people who had property stolen by the Nazis, or were victims of racial persecution.
The latter group also includes Jewish businesses forcibly taken over by German firms as part of a process of 'Aryanisation.'
Payments to survivors in Poland, Russia, the Czech Republic, Ukraine and Belarus, as well as the Jewish Claims conference, might begin this year.
Representatives of those countries and the head of the Jewish Claims Conference, were among those who attended a signing ceremony in Berlin on Monday.
Shortfall
Joschka Fischer described the creation of the fund - named Remembrance - an "historic day" not only for the survivors, but for Germans as well.
"Germany learned many things about itself and its past during the negotiating process, and openly discussed what had been suppressed all too willingly and successfully for a long time," the foreign minister said.
One survivor of a Nazi-era forced labour camp present at the ceremony, Karel Horak, 79, hailed the fund as a moral and financial victory for survivors.
The German Government's chief representative at the compensation talks, Otto Lambsdorff, has appealed for more German companies to pledge compensation money.
The reluctance of many German firms to contribute to the fund means there is still a shortfall of about DM2bn ($1bn).
Mr Lambsdorff said German industry had a collective responsibility for having turned many people into slaves for the benefit of the Nazi war effort.