Link to BBC Homepage

Front Page







World News in Audio

On Air


Talking Point


Low Graphics


Site Map

Friday, April 3, 1998 Published at 16:02 GMT 17:02 UK


World Wide Web is 320 million and growing
image: [ Where do you want to go today? Search engines may not have the answer. ]
Where do you want to go today? Search engines may not have the answer.

Computer Science researchers have confirmed what most Internet users have guessed already: the World Wide Web is bigger than anybody imagines and search engines are only looking at a fraction of pages.

Friday's edition of Science Magazine features a paper by Steve Lawrence and Lee Giles of the NEC Research Institute in Princeton, New Jersey. Their comparative study of the performance of search engines has led them to the conclusion that the Web is now at least 320 million pages wide.

This is a significantly higher estimate (and the researchers believe it could still be a serious underestimate) than previously made:

  • Forrester Research estimates the Web stretches to 75 million pages
  • Altavista says 100-150 million pages
  • Wired Digital says 175 milion

    Their report concludes that search engines cover from 3 per cent (Lycos) to 34 per cent (Hotbot) of the indexable Web.

    Combining the results of six engines covered about 3.5 times as much of the Web as one, so using multiple engine sites, such as Metacrawler, is recommended.

    The rankings for the six studied went Hotbot 34 per cent of indexable pages, AltaVista 28 per cent, Northern Light 20 per cent, Excite 14 per cent, Infoseek 10 per cent, Lycos 3 per cent. Yahoo was not included because of its more general directory structure.

    The report says search engines may be limited by network bandwidth, disk storage, computational power or a combination of all three, despite their claims that they can continue to index close to the entire Web as it grows.

    Refine your search

    "This study is a good thing because people are going to become more aware of the problem, but it's not a new thing," said Danny Sullivan of Search Engine Watch, speaking to News online from a search engine conference in America.

    "Of the search engines, Hotbot says its system is scalable and it can index 300 million pages if it wanted to."

    "But the average user is not complaining about not getting enough information, they're concerned about not finding exactly what they want."

    "That's why you're going to see more specialised search engines with people wanting to crawl only a few sites relevant to a particular topic, home DIY in Britain for example. Some of the big sites are already putting in filters to do this and then there are specialist ones such as which just finds web developments for you."

    Back to top | BBC News Home | BBC Homepage

    Link to BBC Homepage

  • [an error occurred while processing this directive] [an error occurred while processing this directive]
    Internet Links

    Science magazine

    NEC Research Institute


    Search Engine Watch

    The BBC is not responsible for the content of external internet sites.