When its Initial Public Offering was made on June 10, search provider Inktomi's opening price of $18 had leapt to $36 by the close. Within a month, it had doubled again, and then some, to $76.

Eric Brewer on the Inktomi success story

If its growth pattern is mirroring that of the Internet, Inktomi's co-founder and Chief Scientist Eric Brewer has no doubt the company can cope. His buzzword is scalability - whatever the expanding Internet can throw at the search engine, Inktomi can easily add to its capacity to skim through the cream of the Web.

Inktomi knows no barriers

In contrast, he told News online on a visit to London, search engines Infoseek and Lycos "hit the wall" some time ago and AltaVista is only staying in contention through brute-force investment.

While its rivals are now turning to the portal model of striking partnerships with content providers to keep visitors coming, Inktomi is staying in the background as a pure search engine for well-known companies.

Its success in making Hotbot the number one rated search engine has led to Microsoft adopting it for its Start portal, Yahoo! choosing it as a replacement for AltaVista and Disney making it the workhorse for its new family-content search facility DIG.

Clustering computers gathers the most data

Search enquiries are fed through Inktomi's banks of computers in Virginia and Santa Clara, California. There are 166 linked computers in Santa Clara, most with dual processors.

This creates a single super computer with the indexed database spread across the nodes. Adding nodes allows the database to be expanded further or extends the capacity for more users.

Dr Brewer says AltaVista and Infoseek use SMP (Symmetric Multi-Processor) machines. "For AltaVista, that means putting ten processors in one big box, it's very expensive and the whole database has to fit in there.

"When they want to add more users, they add more boxes. They have to throw a lot of money at it. We are more cost-effective and unique in that our clustering technology spreads the database across machines."

Inktomi tops search results

Inktomi is staying ahead of the competition by investing heavily in further research. Almost half of its 130 staff is working on development, many of them having PhD's in parallel processing.

Inktomi stems from the US government funding Dr Brewer in 1994 to look into cheaper forms of supercomputing than traditional proprietary massively parallel approaches. He clustered ordinary workstations and produced a prototype search engine to test the technology.

The Inktomi Corporation, named after the spider in Lakota Indian legend known for his ability to defeat larger adversaries through wit and cunning, was founded ion 1996 and developed the HotBot search engine with Wired Ventures.

Much of Web still unmapped

Scientists at the NEC Research Institute estimated in April that the Web extended to at least 320 million pages. HotBot was ranked as finding 34 per cent of indexable pages, AltaVista had 28 per cent, with Infoseek and Lycos 10 per cent and three per cent respectively.

Dr Brewer says it would be possible to index 100 per cent of the Web but the company's partners do not view it as cost-effective nor necessary.

"It really comes down to what is the marginal benefit to the user. As long as we're the largest database and have a good set of relevant documents we'll find the stuff that people are looking for."

Trafficking faster data

Inktomi's second area of research has been network caching and has led to its Traffic Server product.

"It still uses the same coupled cluster technology. In this case, it's using it to make a single local store of Internet content. This allows users to get data from the cache rather than go across the Internet which can be slow at times, thus improving their performance."

Inktomi estimates Traffic Server can save Internet Service Providers and other companies more than 30 per cent in bandwidth costs. Dr Brewer anticipates strong demand for the product in Europe and Asia where bandwidth and telecom costs are up to ten times higher than those in the US. Inktomi opened a London office in February.

His company has no plans to enter the portal race directly. "We'll stay focussed on technology, that really is our strength. We have plenty of partners that can do a great job of building and managing portals and attracting users so we'll leave that to them and make sure their products are successful through the use of the best possible technology.

"I would expect you will see a continuous flow of major corporations and media companies into the Internet and portal sector over the next two years. But that's actually good for us because we focus on having many different partners.