The two countries currently have no formal diplomatic ties
Protesters greeted Turkish President Abdullah Gul on a landmark visit to Armenia that he said "promises hope for the future".
Alongside his Armenian counterpart, he attended a World Cup qualifying match between the two national teams, which had never played each other before.
The visit signals a thaw in relations between the two countries.
But President Gul was met by angry protesters carrying flags and signs reading: "Recognise the genocide."
'Lack of trust'
This is the first time a Turkish leader has set foot in the country, following Armenian President Serzh Sargsyan's invitation to attend the match, which Turkey won 2-0.
But the game was held amid tight security - 5,000 police were deployed - and Armenian fans reportedly booed and hissed as President Gul took his seat behind a special bullet-proof area at Yerevan's Hrazdan stadium.
The two leaders met before the match in the Armenian capital, Yerevan.
"I was happy to see that we were unanimous with the Armenian side on the need for mutual dialogue to remove barriers to improving bilateral ties," said President Gul when he arrived back in Turkey.
"I underlined that there is no problem that dialogue cannot solve and that dialogue will help alleviate the deep lack of trust between regional countries," he added.
French President Nicolas Sarkozy, who currently holds the European Union's rotating presidency, praised the visit as "courageous and historic".
"It allows hope for progress soon in establishing normal relations between Turkey and Armenia," he said.
Visit 'a betrayal'
The two countries have waged a war of words over Armenia's attempts to label as genocide a mass killing of Armenian civilians by Ottoman forces during World War I.
Protesters gathered outside the palace where the two presidents met
Before leaving Ankara, Mr Gul said he hoped the match would help lift the barriers that divided the two nations, which have no diplomatic ties.
However, the invitation has already sparked a major debate in Turkey, with some nationalists regarding the fact that the president took it up as a betrayal of the country's national interests.
More than a dozen countries, various international bodies and many Western historians have recognised the civilian killings as genocide.
Turkey admits that many Armenians were killed but it denies any genocide, saying the deaths were a part of the world war.
The two countries have had no diplomatic ties since Armenia became independent from the Soviet Union in 1991.
Their common border has been closed since the war between Armenia and Turkey's ally, Azerbaijan, in the 1990s over the disputed territory of Nagorno-Karabakh.