AI crawler URL matrix
Submit a domain and a list of URLs to see, for each AI bot, whether the live robots.txt allows or blocks the path — with the exact RFC 9309 rule that won.
Submit the form to populate the matrix.
Submit a domain and a list of URLs to see, for each AI bot, whether the live robots.txt allows or blocks the path — with the exact RFC 9309 rule that won.
Submit the form to populate the matrix.