Blocked or not? Wildcards, typos and other robots.txt ambiguities | TechnicalSEO.com
Older than most of the search engines we use today, the robots.txt file is a dinosaur of the web. Although there are many articles discussing the protocol and how to use the file, in this post we will cover some ambiguities, corner cases and other undocumented scenarios. Why should you care? The robots.txt is a … Read more
