The first crawler was created back in 1993. It was called the World Wide Worm. It crawled from one site to the next and indexed all pages by saving the content of the web pages into a huge database. Crawlers or spiders visit a web page, read it, and then follow links to other pages within the site and even follow links to other sites. Web crawlers return to each site on a regular basis, such as every month or two, to look for changes. Everything a crawler finds goes into a database, which people are able to query.
The advantage of the web crawlers is that they have an extensive database with almost the complete Internet indexed in it. The disadvantage is that you get thousands of web pages as a response for almost any request.
Web directories work a little differently. First of all they contain a structured tree of information. All information entered into this tree is either entered by the webmaster who wants to announce his new web page or by the directory maintainer who looks at the web pages submitted. The directories where webmasters can submit both the URL (Uniform Resource Locator) and the description normally contain more misleading information.
Getting into most web directories is a combination of luck and quality. Although anyone is able to submit a web page, there is no guarantee that a certain page will be included. Some directories charge for submission, which is basically a rip-off.
Some crawlers maintain an associated directory. If your site is in the directory, the ranking of your web site normally will be higher, as it is already pre-classified. Many directories work with crawlers together, in order to deliver results, when nothing can be found in the directory. |
