Google has added two new crawlers to the Googlebot family of web crawlers:
- GoogleOther-Image
- GoogleOther-Video
Google said these two new crawlers are versions of GoogleOther optimized for fetching image and video bytes. GoogleOther was added in April 2023 to be used internally by Google teams to crawl the public web and free up some resources for the main Googlebot crawlers.
GoogleOther-Image. GoogleOther-Image, according to the documentation, is the version of GoogleOther optimized for fetching publicly accessible image URLs. It will go under the user agent tokens of GoogleOther-Image and GoogleOther and the full user agent string will be GoogleOther-Image/1.0.
GoogleOther-Video. GoogleOther-Video, according to the documentation, is the version of GoogleOther optimized for fetching publicly accessible video URLs. It will go under the user agent tokens of GoogleOther-Video and GoogleOther and the full user agent string will be GoogleOther-Video/1.0.
Why these new crawlers. Google said, “the new crawlers were launched to better support crawling of binary data that may be used for research and development.”
More on Google crawlers. The types of Googlebot crawlers include:
- Googlebot – The main crawler for Google’s search products. Google says this crawler always respects robots.txt rules.
- Special-case crawlers – Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules.
- User-triggered fetchers – Tools and product functions where the end-user triggers a fetch. For example, Google Site Verifier acts on the request of a user or some Google Search Console tools will send Google to fetch the page based on an action a user takes.
Google also has listed IP address ranges and reverse DNS mask for each type:
Why we care. Many of you check your crawling activities and bot activity on your website and in your log files. When you see this new GoogleOther crawlers, do not be alarmed. It is a real Googlebot.
New on Search Engine Land
About the author
Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.
Source link : Searchengineland.com
