Google’s Gary Illyes answered a question about the non-search features that the GoogleOther crawler supports, then added a caution about the consequences of blocking GoogleOther. What Is GoogleOther? GoogleOther is a generic crawler created by Google for the various purposes that fall outside of those of bots that specialize for Search, Ads, Video, Images, News, … Read more
Today being July 4th, I normally would avoid writing stories here but there are a lot of rumors out there that Reddit is blocking Google Search from crawling and indexing its content. It is not true – Reddit is not blocking Google Search. I mean, at first glance, it does look like Reddit’s robots.txt file … Read more
More than 250 websites now block Google-Extended, the “standalone product token” Google introduced Sept. 28 to let you block Bard, Vertex AI generative APIs and future generations of ****** from accessing your content. That’s according to research shared exclusively with Search Engine Land by the Detailed.com team. Why we care. There has been much debate and discussion … Read more
X, well, Twitter.com, is now blocking Bing Search, specifically Bingbot, from crawling and accessing content posted on Twitter.com, on the X platform. Twitter specifically added to its robots.txt file a directive to disallow Bingbot from crawling the content on its platform. Here is a screenshot of its robots.txt file: This was spotted first, I believe, … Read more
At least 26 of the top 100 most popular websites – and 242 of the top 1,000 – are now blocking GPTBot, the web crawler OpenAI introduced Aug. 7, according to an updated analysis. Why we care. To block or not to block ChatGPT? That has been a big question for many SEOs because ChatGPT does … Read more
January 27, 2015 | Posted by Shiri Berzack In the past, Google has warned against blocking Googlebot from crawling specific aspects of your website, and now the warning has been extended to mobile sites. Google’s Zineb Ait Bahajji announced on Google Plus, that blocking Googlebot for Smartphones will have a negative effect on … Read more
Google’s John Mueller answered a question in Reddit about whether Google indexes content placed before the main content as part of the post. John’s answer addressed part of the question but left the overall question unanswered. Nevertheless, there is a solution for the person’s question. Does GoogleBot “Collect” Content Before the Post? The person asking … Read more
I blocked two of our ranking pages using robots.txt. We lost a position here or there and all of the featured snippets for the pages. I expected a lot more impact, but the world didn’t end. Warning I don’t recommend doing this, and it’s entirely possible that your results may be different from ours. I was … Read more
421
229
714
826
543