New recruits actively target the Yemeni regime and its supporters like the US
The deadly car bombing outside the US embassy in Yemen represents an escalation in attacks against Western targets and shows al Qaeda-inspired jihadis are growing in ability and determination.
Islamic Jihad has claimed responsibility for the attack, which killed at least 16 people, but it is possible that other groups will come forward in the next few days.
There is a complex network of over-lapping splinter cells and claims of rival leadership within Yemen.
Extremist violence in Yemen has been on the rise since February 2006, when 23 prominent militants tunnelled their way out of a high-security jail.
Ten Europeans and four Yemenis have died in attacks on tourist convoys in the past 15 months.
In March, a misfired mortar strike hit a girls' school next door to the US embassy by mistake.
A subsequent bombing campaign in the capital - against an expatriate residential compound and oil company offices - prompted the US state department to evacuate all non-essential embassy staff from Yemen.
US employees had just started to return to their embassy desks at the end of August - so the timing of the latest attack is significant.
During July, Yemeni security forces killed five al-Qaeda suspects, disrupted a second cell and arrested more than 30 suspected al-Qaeda members.
In August, a prominent Islamic Jihad figure was arrested.
But this attack shows that effective leadership remains intact and operational capacity has not been disrupted.
Two Saudi passports were found among documents seized in the July raids and interrogations were said to have uncovered plans to launch attacks in Saudi Arabia and Yemen. Yemen subsequently extradited eight Saudi nationals to Riyadh.
The raids underlined the importance to Saudi Arabia of Yemen's internal security. But Yemen is also paying the price for the northern kingdom's muscular clampdown on its own insurgents.
In March, a Saudi militant fundraiser said al-Qaeda had been defeated in Saudi Arabia and he called on his remaining associates to flee to Yemen to escape capture or assassination by the Saudi authorities.
The current migration of Saudi jihadis to Yemen coincides with the emergence of a transnational structure calling itself al-Qaeda in the South of the Arabian Peninsula.
Yemen's mountainous terrain and the weak presence of state structures outside Sanaa have long fostered close ties between jihadis in these neighbouring states.
Cash-strapped Yemen lacks the financial resources to tackle terrorism in the same robust manner as the Saudis; its per capita gross domestic product of $2,300 is dwarfed by the $23,200 seen across the northern border.
In recent years, the Yemeni government has pioneered a dialogue programme and poetry recitals to influence violent jihadis and tribesmen.
The most recent initiative is a two-hour feature film intended to educate the public about Islamic extremism.
The film, called The Losing Bet, follows two Yemeni jihadis who return home after being radicalised abroad.
They are directed by an al-Qaeda mastermind to recruit new members and carry out a "martyrdom operation".
News footage from the aftermath of a real suicide bombing is edited into scenes of this creative new drama - written and produced by a popular Yemeni director.
The film was launched in August, at a five-star hotel that has previously been an intended target of foiled terrorist plots.
It comes as the government faces a new generation of violent Islamists who are blowing the old, inclusive consensus apart.
The young generation appears to be immune to the standard tactic of negotiation and compromise that President Ali Abdullah Saleh used with the Yemeni mujahideen who returned home at the end of Afghanistan's war against the Soviet Union.
The Afghan veterans supported the northern tribes against the former socialist South Yemen during the 1994 civil war in return for a reputed "covenant of security" deal - where the government guaranteed protection inside Yemen as long as violence occurred outside the boundaries of the state.
But new recruits are actively targeting President Saleh's regime, citing as provocation the torture and humiliation of captive al-Qaeda members.
In July, a suicide bomber blew himself up outside a police station in Hadramaut. In a subsequent statement, a splinter cell pledged to continue attacks against security and intelligence structures.
Such an explicit declaration means there is no longer scope for dialogue, according to Ahmed Saif, director of the Sheba Centre for Security Studies.
"The government is moving to a policy of direct confrontation with the younger generation," he says.