Tuesday, August 3, 1999 Published at 13:48 GMT 14:48 UK
Search engine seeks billion URLs
The world's biggest search engine being installed (Fast photo)
By Internet Correspondent Chris Nuttall
A search engine that promises to index all of the World Wide Web has been officially launched.
Alltheweb.com says it already holds 200m unique Web addresses in its database, or 25% of the 800m URLs (Uniform Resource Locators) estimated to be out there.
A study published in last month's Nature calculated that the leading search engines indexed no more than 16% of the Web. Northern Light achieved this figure, with AltaVista covering 15.5% and Hotbot, 11.3%.
Powered by Norwegian algorithms
The Norwegian company, Fast Search and Transfer ASA, is behind alltheweb. It has already made a name for itself and created a court case over its MP3 search engine used by Lycos.
Fast says its new search engine "is the result of more than a decade of research into optimising search algorithms and architectures in a project initiated at the Norwegian Institute of Technology in Trondheim."
It plans to catalogue the entire Web in the space of a year, by which time it should be more than one billion pages in size at current rates of growth.
The "Fast Search" architecture is based on Dell PowerEdge servers and PowerVault storage subsystems, operating in parallel to distribute user queries, search the catalogue and spider the Web.
Fast says a typical query can search all 200m documents in less than a second and the parallel systems mean a document index can be built in only 12 hours, compared to several days or even weeks for some of the other search engines.
Fast targets Inktomi
Fast's strategy will be similar to that of its major rival Inktomi - reselling the system to major portals, search engines, ISPs and content sites.
"Smaller search engines will miss a number of web site 'gems' that would have been ranked in the first one or two pages - if only the search engine had those sites in its catalogue", says Espen Brodin, president and CEO of Fast Search and Transfer.
"The massive size of Fast Search allows it to find more of these highly rated gems and place them on the first or second page of search results, resulting in a better and more satisfying search experience."
Big ain't necessarily better
But size is not everything and while alltheweb seemed lightning fast when I tried it, it did not appear to match the relevancy of a much smaller search engine now gaining popularity, google.com.
"There's always going to be people who want really comprehensive results so there's certainly every reason to go for alltheweb now," says Danny Sullivan of Search Engine Watch.
"But you just can't pull relevancy out of all that. This is a technology showcase for them so they can power other people like Inktomi.
Google has links edge
"Google is a perfect example of a search engine that is attracting a lot of loyal users and people are saying is a good service.
"That's because it's providing very good results by looking at the amount of links pointing to another page and deducing that page must be more relevant than one with fewer links."
"But people should go and try all the tools out there and stick with the ones they feel comfortable with."