Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies
Google is exploring alternatives or supplemental ways of controlling crawling and indexing beyond the 30 year standard of the robots.txt protocol. “We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases,” Google wrote. Engaging with the community. … Read more
