Solicitors for investors swindled by a financial consultant who left a £7m IOU in a safe say they could face a long battle to win their money back.
Police say Graham Price has lost much of the money gambling
They lost large sums to Graham Price, 58, a former Halifax Bank agent at the branch in Gowerton, Swansea, awaiting sentence after admitting theft.
John Collins and Partners said their clients had to consider legal action.
But the Halifax said it was not responsible and all had invested with Price in his personal capacity.
The former Halifax agent is on bail after he pleaded guilty at Swansea Crown Court to 43 charges and asked for a further 263 offences to be taken into account.
His crime was uncovered when a cash audit revealed three empty boxes and a signed IOU note from Price which said: "I have borrowed £7m from the Halifax".
He stole £10m in total, including money from the Halifax. Price disappeared after the discovery but was traced and arrested, and police said he seemed to have "lost" much of the money, with a lot going on horse racing.
Some of the 85 victims have taken on solicitors to reclaim their money, and Llyr Davies, of John Collins, said his firm had been contacted by several.
"These individuals have invested considerable sums of money with Mr Price, believing that they would be legitimately invested, our clients believing they would be invested with the Halifax.
"It's since turned out that the investments were not made with the Halifax and Mr Price used the money himself and obviously our clients are now faced with the position whereby they've got to consider legal ways of recovering the money."
The agency for the Halifax in Gowerton where Price worked
Mr Davies said the case would involve complicated legal argument.
"Mr Price was trading from premises that were branded with the Halifax logo," he told BBC Radio Wales.
"The Halifax are denying that they are liable to compensate these individuals and therefore they are faced with having to run very complex arguments as to why in their view the Halifax should be held liable."
Mr Davies said Price's assets had been frozen and the Halifax had lost a large sum, but his clients were "innocent investors".
He urged anyone else affected to come forward. "There is an argument that there is strength in numbers here and although they are faced with a David and Goliath scenario it's important for them to come out and consider any action together."
He said it was impossible to tell how long legal action would take, and much depended on whether victims pursued it individually or together.
A Halifax spokesman said everything involved in the investment had made clear that it was done in Price's personal capacity, and nothing suggested the people who had lost money were putting it into the bank.
He said Price had not taken undertaken the investment "under the auspices of the Halifax," and had recruited them through his own business.