Japanese officials, who were given a warm welcome by North Korean Prime Minister Hong Song-nam, said negotiations had been "frank".
But the BBC's Charles Scanlon, in Tokyo, says there has been little sign of real progress on the first of two days of talks.
The head of the Japanese delegation, Hitoshi Tanaka, had said that a key issue would be the resolution of Tokyo's demands for the return of at least 11 Japanese citizens it says were kidnapped by North Korea in the 1970s and 1980s.
On Sunday, at the talks, Japanese delegates stressed that they would give no ground on this contentious issue.
For its part, Pyongyang reiterated its demand that Tokyo apologise and pay reparations for its colonisation of Korea between 1910 and 1945.
These issues have hampered negotiations between the two countries on establishing formal diplomatic relations.
'Desperation'
The talks come amid growing signs that North Korea, one of the world's most isolationist countries, may be seeking to improve ties with the rest of the world.
In recent weeks, North Korean officials have been spearheading a diplomatic offensive aimed at improving ties with the US and Washington's two key Asian allies, South Korea and Japan.
The three countries meet regularly to fine-tune and co-ordinate their policies towards the North, which was branded by US President George W Bush as part of an "axis of evil".
Last month, Washington said it was ready to send a high-level envoy to Pyongyang but would first wait to see the outcome of resumed inter-Korean dialogue as well as the current talks between Japan and North Korea.
Analysts believe the North's new diplomatic drive is prompted by economic desperation and its need for foreign help.
Kidnap controversy
Japan believes at least 11 people kidnapped by North Korea
Says they were used to teach Japanese to N Korean spies
Spies then said to have entered S Korea posing as Japanese tourists
Japan used to be one of the largest donors of food aid to North Korea through the World Food Programme, but this year Tokyo has not given any food aid and has made clear that future help will depend on progress made on the issue of the alleged kidnappings of its nationals.
Pyongyang has repeatedly denied that it was involved in the abduction of Japanese citizens, though it has shown some signs of flexibility.
Last week it allowed Japanese Red Cross officials to meet police who are said to be conducting an investigation. It also gave details of other Japanese who have settled in North Korea.
But Japanese foreign ministry officials are warning against expecting big advances as there are still major sticking points between the two sides.