Type of web bot: crawler

A web crawler is a type of bot, and also know as spider,bot or robot.

A crawlers used to automating maintenance tasks on a Web site,such as identifies links or checking validating HTML code

Is a program broweses the world wide web in methodical.

Googlebot discovers new and updated page to added to google index that is call crawling

How does web crawler works

It is a program that download seed from the world wide web, extracts the links contained in that page,

and take the page those links refer to, extracts the links in thse page.

Credits to:

http://research.microsoft.com/en-us/projects/msrbot/#webcrawler

http://www.boryi.com/web-bot-webcrawler/development.html

http://www.elrona.com/12/29/web-crawler-web-spiders-web-bots/