Also known as the 'invisible web', the 'deep web' is a vast repository of web pages, usually generated by database-driven websites, that are available to web users yet hidden from traditional search engines. The 'spiders' used by these search engines to crawl the web cannot reach most of the pages created on-the-fly in dynamic sites such as ecommerce, news and major content sites.

According to a study by Bright Planet, the deep web is estimated to be up to 550 times larger than the 'surface web' accessible through traditional search engines and over 200,000 database-driven websites are affected by the problem. Chris Sherman, associate editor of Search Engine Watch, estimates the amount of quality pages in the deep web to be 3-4 times more than those pages accessible through search engine like Google. While the actual figures are debatable, it is clear that the deep web is far bigger than the surface web, and it is growing at a much faster pace.

more articles about the deep web »

While search engines like Google, AltaVista and FAST claim to 'index the web', they are actually indexing a very small portion of it. Traditional search engines, using 'spiders' and 'crawlers', were designed to index simple HTML pages that have incoming links from other pages on the web. But modern websites, operating databases to generate pages on-the-fly, are "too sophisticated" for these search engines to index their pages. So while these search engines do a very good job at indexing small sites and personal home pages, they cannot provide sophisticated sites such as eBay, Library of Congress and iMDB (the Internet Movie Database) the means to expose their pages to searchers.

Websites using dynamic pages generated by applications like CGI or ASP, are usually part of the deep web. The pages in these sites, while available to any human web user, are practically invisible to search engines. Affected sites therefore sustain a significant loss in targeted traffic.

Naturally, a search engine that does not index most of the web is providing the user with a partial service. But the deep web is a bigger problem than that; The quality of information in the deep web is usually very high (compared to the surface web), because the deep web consists of the major, authorotive websites available on the net.

There are several companies offering solutions to the deep web problem. Quigo Technologies, Inc. offers search engine exposure technologies for large ecommerce and content sites, and cutting-edge search, retrieval, and Information Extraction (IE) platform for exposing hidden information in the Web (quigoint.com).
Other deep solutions include LexiBot by BrightPlanet (lexibot.com), InvisibleWeb.com by IntelliSeek (invisibleweb.com) and Copernic (copernic.com).

more articles about the deep web »

Fill in this form for more deep web information: