Expedition to the lost net

Often the wires weaving the web go nowhere

The global reach of the net is one of its greatest charms.

But there are some parts of the net that no-one can reach, parts that are cut off from even the most dedicated net user either by accident or design.

A study of these lost sites by US researchers has investigated the extent of the "dark net" and just what causes a net address to fall off the map.

The study found that up to 5% of the net - potentially 100 million hosts - is completely unreachable.

Return to sender

The internet works because all the computers on it have an individual address.

Routers, devices that shuffle data around the net, know where to send information because they maintain lists of these addresses.

But occasionally some of these addresses disappear and researchers at Arbor Networks have spent three years finding out why.

The researchers found that the number of sites an individual surfer can see depends on their starting point.

Contractual wrangles between net service providers can mean that customers of the rowing firms are cut off from portions of the web that the rival firm gives access to.

The infrastructure is increasingly unwieldy and vulnerable

Craig Labovitz, Arbor Networks

More often though, the researchers found that net sites are cut off because of wrongly configured routers or malicious hackers and computer vandals abusing loopholes in net software.

The study reveals that all parts of the net are not equally connected to every other part. By contrast, some networks inadvertently act as gatekeepers for sections of the net that lie beyond them.

Hijacked routers

In their study, the three researchers, Craig Labovitz, Abha Ahuja and Michael Bailey, reported that many of the lost net sites flare into life briefly when being used to send spam (unsolicited commercial e-mail messages) or to launch attacks on other parts of the web.

The Computer Emergency Response Team, which monitors security problems online, has already issued a warning about malicious hackers hijacking routers to use as a launch pad for attacks.

The specifications that govern the way routers co-ordinate to ensure the net keeps running are unfortunately susceptible to subversion by those who are determined enough.

Routers can be made to pose as particular net addresses and be used to launch barrages of data at other target sites.

"The sheer quantity of routing information, coupled with the lack of security both in routers and the routing protocol itself create an infrastructure that is increasingly unwieldy and vulnerable," said Craig Labovitz, lead researcher on the Arbor Network study.

Other parts of the internet, mainly sites used by the US military, are lost because they use old addresses that no router references anymore.