Berwick progressed to the second round of the Challenge Cup following a penalty shoot-out at Cliftonhill.
Albion Rovers dominated for long spells, but Berwick took the lead in the 73rd minute through Danny Swanson.
Rovers found a deserved equaliser in injury-time when Jamie McKenzie's shot was deflected past Flockhart.
The extra-time period ended goalless. McKenzie and Ciaran Donnelly missed penalties, before Gary Wood scored the winning spot kick for Berwick.
Albion Rovers: Scott (Thompson 45), Graham Watson (Dimilta 81), Reid, Donnelly, Benton, Donald (Friel 45), Chisholm, McKenzie, Hunter, Patrick Walker, Wright. Subs Not Used: Smith, Peter Watson.
Booked: Graham Watson, McKenzie.
Goals: McKenzie 90.
Berwick Rangers: Flockhart, Lennox, Noble, Smith, Bolochoweckyj, Lucas, David Greenhill (Manson 64), McLeish, Swanson (Diack 86), Wood, McMullan (Haynes 82). Subs Not Used: Gemmill, O'Connor.
Booked: Lennox, Lucas.
Goals: Swanson 73.
Berwick win 4-3 on penalties
Ref: J McKendrick