Google Clarified Support For Robots.txt Fields
Google added eight or so words between brackets to clarify that fields that aren’t explicitly listed in the robots.txt documentation aren’t supported. Why did they do this? Google said because sometimes Google gets questions about it. The words Google added were added to the syntax section of the robots.txt documentation and it reads: (other fields … Read more
