Google’s Gary Illyes and John Mueller, along with Bing’s Fabrice Canel and probably other representatives from these search engines spent time in Dublin this week to attend the IETF 121 Dublin meeting with the goal of submitting ideas on how to improve the Robots Exclusion Protocol. This covered improving crawl efficiency, adding new AI controls … Read more
Google’s Search Advocate, John Mueller, shared insights on diagnosing widespread crawling issues. This guidance was shared in response to a disruption reported by Adrian Schmidt on LinkedIn. Google’s crawler stopped accessing several of his domains at the same time. Despite the interruption, Schmidt noted that live tests via Search Console continued to function without error … Read more
Google posted a new “Search Off The Record” podcast yesterday on the topic of crawling where John Mueller, Lizzi Sassman, and Gary Illyes spoke about how Google crawls, some ideas on how to make crawling more efficient and some misconceptions around crawl frequency and site quality. I don’t think anything in this podcast is particularly … Read more
According to a recent LinkedIn post by Gary Illyes, Analyst at Google, you should be cautious if Google starts aggressively crawling your website. While an uptick in crawling can be a good sign, Illyes says it may indicate underlying issues. Illyes cautions: “Don’t get happy prematurely when search engines unexpectedly start to crawl like crazy … Read more
Google’s Gary Illyes posted on LinkedIn with two common examples of when a spike in Googlebot activity, crawling, is a bad thing. The short answer is when Googlebot gets to crawling an infinite section of your site (like calendar pages that goes on forever) and when your site is hacked with a ton of new … Read more
Google posted a public service announcement saying you should disallow Googlebot from crawling your action URLs. Gary Illyes from Google posted on LinkedIn, “You should really disallow crawling of your action URLs. Crawlers will not buy that organic non-GMO scented candle, nor do they care for a wishlist.” I mean, this is not new advice. … Read more
Reading Time: 11 minutes As the predominant web crawler accounting for nearly 29% of bot hits, Googlebot plays a crucial role in determining how users access information online. By continuously scanning over millions of domains, Googlebot helps Google’s algorithms understand the vast landscape of the internet and maintain its dominance in search with over 92% … Read more
When it comes to crawl budget and Google not crawling your site too much or too little, Google takes into account all Googlebot activity across all verticals. So that includes crawling for web search, Merchant Center, ads, and all the various crawling activities. Google’s John Mueller was asked by Christian Radny, “Does merchant center crawling … Read more
In a recent statement on LinkedIn, Google Analyst Gary Illyes shared his mission for the year: to figure out how to crawl the web even less. This comes on the heels of a Reddit post discussing the perception that Google is crawling less than in previous years. While Illyes clarifies that Google is crawling roughly … Read more
Google’s John Mueller said on Reddit that disallowing URLs with UTM parameters in them won’t help you to improve crawling or rating with Google Search. He added that a site should try to keep its internal URLs clean and consistent, but over time, the canonical tags should help with external links that carry UTM parameters … Read more
421
229
714
826
543