Allen's two tries were not quite enough for Gloucester
Gloucester: (14) 26
Tries: Allen (2), Azam
Pens: Lamb (3)
Agen: (15) 32
Tries: Mignardi, Van Niekerk, Soucaze, Stoltz
Cons: Miquel (3)
Gloucester face an uphill battle to qualify for the knock-out stages of the Heineken Cup after falling to Agen.
The home side failed to sparkle, trailing 15-14 at the break after Arnaud Mignardi and Pieter Van Niekerk scored for the French side.
Agen went further ahead through Thomas Soucaze and Conrad Stolz, although a superb try from Anthony Allen, who also crossed in the first half, cut the gap.
Olivier Azam scored on 71 minutes but an Agen drop-goal clinched their win.
After losing their Heineken Cup opener to Leinster last week, Gloucester have given themselves a mountain to climb by losing the first home match of their European campaign.
But Gloucester can only blame themselves after a display lacking the fluency which had characterised their season, with Agen's outstanding Fijian winger Rupeni Caucaunibuca providing the most attacking threat on the pitch.
Fly-half Ryan Lamb kicked two early penalties to get Gloucester off to a strong start, but they failed to build on it.
An excellent Caucaunibuca break created the inroads from which Mignardi eventually crashed over, and Van Niekerk's try on 17 minutes confirmed Agen's dominance.
Gloucester received a boost with Agen winger Dave Vainqueur's 34th-minute sin-binning and soon after Allen - in his first match since his England call-up - charged over.
After the break, Agen re-established their lead through an impressive Soucaze move and they claimed a bonus point after another Caucaunibuca break ended in Stolz's try.
The in-form Allen dragged his side back into the match with an individual try, and a late effort from Azam, driven over by his fellow forwards, gave Gloucester hope.
But replacement fly-half Francois Gelez knocked over a 74th-minute drop-goal to take Agen six points clear, and the French side held off Gloucester's late attacks to take the win.
Gloucester coach Dean Ryan
"If we want to compete on that level our players need to recognise what's in the opposition team and that's why this was a very good reality check for us.
"All I've heard for the last three or four weeks is the hype surrounding this group, and this group is coming to terms with that.
"We're a young side that lacks physicality at this level and we've a lot to learn."
Gloucester: Morgan; Bailey, Adams, Allen, Balshaw; Lamb,
Richards; Califano, M Davies, Nieto; Pendlebury, Bortolami; Buxton, Boer, Narraway.
Replacements: Walker for Morgan (49), Keil for Adams (56), Collazo for Califano (53), Azam for M Davies (53), Eustace for Bortolami (55), Balding for Boer (39).
Not Used: Lawson.
Agen: Elhorga; Vainqueur, Ahotaeiloa, Mignardi, Caucaunibuca; Miquel, Morales; Van Niekerk, Tiatia, Galasso; Badenhorst, Socol; Som, Yukes, Soucaze.
Replacements: Guinazu for Vainqueur (59), Stoltz for Ahotaeiloa (47), Gelez for Miquel (63), Stolz for Badenhorst (47), Culine for Yukes (34).
Not Used: Cabarry, Fulton.
Sin Bin: Vainqueur (33).
Referee: Malcolm Changleng (Scotland).