Google Documents Its Three Types Of Web Crawlers
Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers. I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. … Read more