Anthony John Allen made a statement to police in May 1976, saying he had met his wife Patricia in 1967 about the same time he faked his own death.
Mr Allen, of Poole, Dorset, also said in the statement he had been through a bigamous marriage with Patricia before marrying her in a legal ceremony in 1972.
The 68-year-old denies murdering his wife and their children, Jonathan, seven, and Victoria, five, who all disappeared in Salcombe in May 1975.
Mr Allen's past emerged at Exeter Crown Court when statements he made to the police after the disappearance of his family were read to the court by the prosecution.
It was also revealed he had been imprisoned for 13 months in 1973 over a mail delivery business and told police his wife "did not want me from the time I came out".
In a witness statement made to the police in October 1975, Allen said when he moved to Salcombe earlier that year as restaurant manager of the Marine Hotel, his marriage was "under strain".
He said they were trying to make a new start, but were "drifting apart".
The statement added in May 1975 he and Patricia had an argument after going to a carnival with the children and that day or the next, he found her packing her bags and leaving in her Volkswagen car.
'Little holiday'
Mr Allen said in the statement she reappeared the next day, saying she could not live without the children, and that she was considering emigrating.
The children were then woken up, clothes and toys were packed, and they were told they were going on a "little holiday".
Mr Allen said he signed a piece of paper, written by his wife, agreeing to hand the children into her custody and she could take them anywhere she liked.
The court heard he told the police he never saw or heard from them again after she drove off in her Volkswagen car.
She had said she would leave it at Totnes railway station, but it was not there when he went later to pick it up, and he was told four or five weeks later it was in a Salcombe car park.
The trial continues.