Customs officers are so thin on the ground that they are "no discouragement to terrorists", the UK's official reviewer of terror laws has warned.
Customs officer 'too thinly stretched'
Lord Carlile, in an annual report, said adequate staffing at "ports of entry of all kinds is an important matter".
He also urged less use of police stop and search powers under the act.
Home Office figures, included in Lord Carlile's report, showed 266 people were arrested under the Terrorism Act in 2005 - up 64% on the year before.
Of those, 35 were charged, 27 under the act.
The majority of the 2005 arrests - 212 out of the 266 - were made after the 7 July bomb attacks.
And he expressed concern, as he has done previously, about the risk of hijacking executive jets.
In his report on the operation of the act in 2005, Lord Carlile described the close working relationship between the police, customs and immigration officers as "excellent where the facilities and staff are adequate".
"However, I remain of the view that customs officers, in particular, are thinly spread," he said.
He told how in March 2006 he himself tried to declare some gifts on arrival at Heathrow from South Asia, but there were no customs officers and the phone provided to do this failed to operate.
"This kind of manpower weakness is no discouragement to terrorists," he said.
"This is still a chorus complaint by special branch officers. The adequacy of staffing at HM Customs and Excise at and for ports of entry of all kinds is an important matter."
Home Secretary John Reid said customs were used in a targeted way.
In his response to the peer's report, Mr Reid said officers were deployed according to risk with strike forces deployed to meet specific risks or where there were no permanent staff.
"The time has long gone when small numbers of staff, carrying out the same and similar tasks each day in fixed locations can provide either the detection or the deterrent capability needed," he said.
"This targeted, risk based, approach is also the rationale behind the use of telephones in the red channels rather than have staff dealing with the occasional passenger with goods to declare."
Customs was looking at the use of "more advanced technology to upgrade this service", he added.
On the stop and search powers Lord Carlisle said there was "little or no evidence" that the powers had the potential to prevent an act of terrorism, compared with normal police stop-and-search powers.
Under Section 44, police can declare a geographical area where searches can be carried out which do not have to be founded on reasonable suspicion.
Lord Carlisle noted that no Scottish force had used the powers, but said he doubted whether Scotland was less at risk from terrorism than other parts of the country.
In response to this criticism, the home secretary accepts there is "some inconsistency" in the way stop and search powers under the act are authorised.
In his letter to Lord Carlile, Mr Reid says "these powers are an important tool in the on-going fight against terrorism. As part of a structured anti-terrorist operation, the powers help to deter and disrupt terrorist activity by creating a hostile environment for would-be terrorists to operate in".
He adds: "The decision to authorise the powers is ultimately one for the Chief Constable of the police force concerned.
"We do, however, acknowledge your observations that there is some inconsistency between forces in the authorisation of the powers and a programme of work will be shortly undertaken by the Association of Chief Police Officers to consider and address these issues."