Another 32 people have been rescued alive and taken to local hospitals, but there is now little hope for another 40 missing miners.
Survivors said there were two mudslides at the mine on Thursday. The second buried miners trying to rescue those trapped by the first avalanche.
Correspondents say the victims were poor people who were trying to scrape gold deposits from the mine near Filadelfia some 300 kilometres (160 miles) northwest of the capital Bogota, which had been officially closed due to poor safety conditions.
Many people who have been displaced by the country's 37-year civil conflict live in this area.
Friends buried
"Suddenly the hill came down upon us and we had no time to do anything," Jairo Bedoya, a miner who managed to escape unharmed, told local Caracol Radio.
"My feet were trapped between rocks, but I managed to free myself, however other friends remained buried," he added.
Civil defence and Red Cross workers who rushed to the mine near the town of Filadelfia, in the department of Caldas, have been using heavy digging equipment to try to reach those buried, but difficult conditions continue to hampering the work.
"Rescue efforts are going to be very difficult and could take days. It rained hard for days here and it is a swampy area," Natalia Gaviria, a spokeswoman for the Civil Defence agency in the capital of Caldas Manizales said.
Distraught relatives of those still missing have waited for hours near the mine for news.
"I told my wife not to go to the mine because it is very dangerous, but she didn't listen," a sobbing Jesus Uria Romero said.
Landslides, often triggered by heavy rain, are common in Colombia. In April, seven miners were killed by a landslide in an open-pit gold mine in the Choco department.