Police have been advised to use the anti-terrorism powers sparingly
The UK's terror law watchdog says people are being stopped and searched to racially balance official figures.
In his annual report, Lord Carlile QC warned of "poor or unnecessary" use of the special Section 44 powers.
He said there was ample anecdotal evidence of officers searching people not connected to terrorism.
Lord Carlile also warned there was little or no evidence that blanket use of Section 44 searches had the potential to prevent a terror attack.
Section 44 of the Terrorism Act 2000 allows police to stop and search someone without suspicion that an offence has occurred.
The powers can only be used in specific areas on the orders of a police chief, with later approval by the home secretary.
Supporters say such powers can make it harder for extremists to carry out reconnaissance in public areas, such as near important tourist attractions.
'Poor or unnecessary'
The Metropolitan Police used them 170,000 times during 2008, according to figures previously obtained by BBC London.
But in the strongest criticisms Lord Carlile has ever made of the use of the powers, the peer said that none of the stops in London or elsewhere had led to a conviction.
"The damage to community relations if they are used incorrectly can be considerable," he said.
"Examples of poor or unnecessary use of Section 44 abound.
"I have evidence of cases where the person stopped is so obviously far from any known terrorism profile that, realistically, there is not the slightest possibility of him/her being a terrorist, and no other feature to justify the stop."
Lord Carlile said it was "totally wrong" for anyone to be stopped under Section 44 just to produce a racial balance in the statistics.
"I can well understand the concerns of the police that they should be free from allegations of prejudice," he continued.
"But it is not a good use of precious resources if they waste them on self-evidently unmerited searches.
"It is also an invasion of the civil liberties of the person who has been stopped, simply to 'balance' the statistics."
Former British diplomat Sir Edward Clay told BBC Radio 4's The World Tonight that he was stopped and searched on his way to work.
"I'm 63, I'm a grey-to-brown-haired white male, I'm 5ft 10in tall, looking extremely conventional," he said.
"I found this frankly slightly intimidating. I found it mildly sinister. I was quite shocked really to think that it could happen in broad daylight at 8.30 in the morning on a London street."
Lord Carlile said that Section 44 was causing concern at all levels of the police and that he did not know why it was being used at some sites but not others which were strikingly similar.
However, he did say the power was needed and had been used effectively in the past.
But he added: "I now feel a sense of frustration that the Metropolitan Police still does not limit their section 44 authorisations to some boroughs only, or parts of boroughs, rather than to the entire force area.
"I cannot see a justification for the whole of the Greater London area being covered permanently, and the intention of the section was not to place London under permanent special search powers."
Home Secretary Alan Johnson said the Metropolitan Police had already begun to review how Section 44 was used across the whole of the capital, including a pilot of its more restricted use.
"I welcome this initiative and, like you, look forward to reviewing the results of this project," he said in a response to Lord Carlile's review.