The man injured two people and caused up to £200,000 worth of damage in what has been described as a "frenzied" assault.
Tanya Jones said she saw a man walking up the road "calmly attacking" everything in his way with the axes.
"
He was shouting but it was hard to hear what he said over the noise of him
hitting things
"
Canon Martin Webster
The attacker set upon a car with a woman and her son inside, by the riverside cafe near the abbey, "smashing all the windows", Ms Jones added.
The glass shattered over the occupants and the woman's son had a cut arm while the woman was clearly suffering from shock," she said.
Ms Jones said it was a "long time" after the drama began that the attacker went on to the church, where there was such destruction.
Wrecked church
Canon Martin Webster, vicar of the abbey, said the "frenzied attack" started when the man became involved in a dispute some distance from the abbey.
"
He was clearly in an agitated state but quite determined
"
Canon Martin Webster
After damaging vehicles he burst into the church, "running amok" with an axe in either hand for some five to six minutes.
"He got into the sanctuary and really had a go at the heart of our church, that's what is most upsetting," Mr Webster said.
"He was shouting but it was hard to hear what he said over the noise of him hitting things.
"He was clearly in an agitated state but quite determined.
"It was upsetting, but at the time we were more concerned with making sure staff and members of the public had left the building.
"But when he started to tackle the organ I did want to say to him, `Please don't do that'."