Europe South Asia Asia Pacific Americas Middle East Africa BBC Homepage World Service Education

Front Page



UK Politics







Talking Point

In Depth

On Air

Low Graphics

Tuesday, August 3, 1999 Published at 13:48 GMT 14:48 UK


Search engine seeks billion URLs

The world's biggest search engine being installed (Fast photo)

By Internet Correspondent Chris Nuttall

A search engine that promises to index all of the World Wide Web has been officially launched. says it already holds 200m unique Web addresses in its database, or 25% of the 800m URLs (Uniform Resource Locators) estimated to be out there.

A study published in last month's Nature calculated that the leading search engines indexed no more than 16% of the Web. Northern Light achieved this figure, with AltaVista covering 15.5% and Hotbot, 11.3%.

Powered by Norwegian algorithms

The Norwegian company, Fast Search and Transfer ASA, is behind alltheweb. It has already made a name for itself and created a court case over its MP3 search engine used by Lycos.

Fast says its new search engine "is the result of more than a decade of research into optimising search algorithms and architectures in a project initiated at the Norwegian Institute of Technology in Trondheim."

It plans to catalogue the entire Web in the space of a year, by which time it should be more than one billion pages in size at current rates of growth.

The "Fast Search" architecture is based on Dell PowerEdge servers and PowerVault storage subsystems, operating in parallel to distribute user queries, search the catalogue and spider the Web.

Fast says a typical query can search all 200m documents in less than a second and the parallel systems mean a document index can be built in only 12 hours, compared to several days or even weeks for some of the other search engines.

Fast targets Inktomi

Fast's strategy will be similar to that of its major rival Inktomi - reselling the system to major portals, search engines, ISPs and content sites.

"Smaller search engines will miss a number of web site 'gems' that would have been ranked in the first one or two pages - if only the search engine had those sites in its catalogue", says Espen Brodin, president and CEO of Fast Search and Transfer.

"The massive size of Fast Search allows it to find more of these highly rated gems and place them on the first or second page of search results, resulting in a better and more satisfying search experience."

Big ain't necessarily better

But size is not everything and while alltheweb seemed lightning fast when I tried it, it did not appear to match the relevancy of a much smaller search engine now gaining popularity,

"There's always going to be people who want really comprehensive results so there's certainly every reason to go for alltheweb now," says Danny Sullivan of Search Engine Watch.

"But you just can't pull relevancy out of all that. This is a technology showcase for them so they can power other people like Inktomi.

Google has links edge

"Google is a perfect example of a search engine that is attracting a lot of loyal users and people are saying is a good service.

"That's because it's providing very good results by looking at the amount of links pointing to another page and deducing that page must be more relevant than one with fewer links."

"But people should go and try all the tools out there and stick with the ones they feel comfortable with."

Advanced options | Search tips

Back to top | BBC News Home | BBC Homepage | ©

Sci/Tech Contents

Relevant Stories

07 Jul 99 | Sci/Tech
Web engines could do better

03 Aug 99 | Sci/Tech
Pressure mounts on MP3

10 Jul 98 | The Company File
Inktomi searches for Net profits in Europe

03 Apr 98 | Sci/Tech
World Wide Web is 320 million and growing

Internet Links

NEC study on search engines

Search Engine Watch

FAST Search


The BBC is not responsible for the content of external internet sites.

In this section

World's smallest transistor

Scientists join forces to study Arctic ozone

Mathematicians crack big puzzle

From Business
The growing threat of internet fraud

Who watches the pilots?

From Health
Cold 'cure' comes one step closer