Googlebot
From Free net encyclopedia
Mathwiz2020 (Talk | contribs)
clean up and changing template as per [[Talk:Google]] using [[Wikipedia:AutoWikiBrowser|AWB]]
Next diff →
Current revision
A Googlebot is a search bot, used by Google. It collects documents from the web to build a searchable index for the Google search engine.
If a webmaster does not want to have any of the information on their site downloaded by a Googlebot, they can do so by inserting a robots.txt file, which can stop Googlebot and other web-crawling robots from caching their site.
Googlebot has two versions, deepbot and freshbot. Deepbot, the deep crawler, tries to follow every link on the web and download as many pages as it can to the Google indexers. Currently (March 2006), it completes this process about once a month. Freshbot crawls the web looking for fresh content. It visits websites that change frequently, according to how frequently they change. Ideally, freshbot would visit a daily newspaper's website every day and a weekly ezine would get crawled once every 7 days.
Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to from another known page on the web in order to be crawled and indexed.
External links
Template:Google servicesde:Googlebot fr:Googlebot pt:Googlebot