However, one bright point for Toshack was the return of Cardiff's Paul Parry to a Wales shirt after an 18 month-gap because of personal problems.
The Georgians had overcome considerable difficulties to actually make it to the Liberty Stadium due to the recent conflict with Russia over the disputed territory of South Ossetia.
The visitors wore black armbands and linked hands - held aloft - when their anthem was played.
Zurab Khizanishvili and Levan Mchedlidze both had headers go close in the opening minutes.
But Wales worked their way into the game and Simon Davies fired over from five yards after Parry's fine cross.
Wales then took the lead after 16 minutes when Koumas curled in a speculative 25-yard shot that seemed destined for the midriff of Loria.
But the Georgia goalkeeper allowed the ball to bounce out of his grasp and over the line.
Loria gained some redemption when he was forced into a plunging save to stop a Parry drive, before the Cardiff man went close again from a Simon Davies pull-back.
Jason Koumas scored Wales' opener but had to settle for defeat
Neal Eardley and Davies combined well down the right after 26 minutes to produce a delivery that evaded Loria and saw Koumas fire into the side-netting.
But Georgia enjoyed their own moments and Myhill needed to be alert to palm out a curling corner from David Mujiri.
Georgia started the second period with a quicker tempo and substitute Aleksidze saw a header go just wide, with Kenia firing over soon after.
Wales were being pressured on both flanks and Georgia deserved their equaliser after 66 minutes.
Aleksidze's ball cut through the centre of the Wales defence to allow Kenia the opportunity to slide the ball past Myhill.
Davies almost found an instant reply as he broke from midfield to round the on-rushing Loria.
But the Georgian keeper forced the Fulham man wide and Davies' goal-bound shot was too weak to beat the covering defence.
Then Carl Fletcher sent Wales sub Rob Earnshaw away with a pass down the right inside channel.
The Forest striker cut inside to make a better angle for a left-foot shot but had the ball nicked off his toes.
The match appeared to be heading for a draw but Georgia punished Wales with an injury-time winner when Gotsiridze took advantage of a mix-up between Ashley Williams and Myhill to head home.
Wales team: Boaz Myhill (Hull), Neil Eardley (Oldham), Sam Ricketts (Hull), Carl Robinson (Toronto), Craig Morgan (Peterborough), Ashley Williams (Swansea), Carl Fletcher (Crystal Palace), Simon Davies (Fulham, capt), Paul Parry (Cardiff), Jason Koumas (Wigan), Freddie Eastwood (Coventry). Subs: Rhys Taylor (Chelsea), Steve Evans (Wrexham), David Cotterill (Sheff Utd), Rob Earnshaw (Nottingham Forest), Owain Tudur Jones (Swansea), David Vaughan (Blackpool), Jermaine Easter (Plymouth), Andrew Crofts (Gillingham), Matthew Collins (Swansea).
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.