← Back to the audit

AI crawler URL matrix

Submit a domain and a list of URLs to see, for each AI bot, whether the live robots.txt allows or blocks the path — with the exact RFC 9309 rule that won.

One per line, max 10. A path (/blog) or a full URL.

Intent profile

Used to interpret whether each access decision matches your crawler-governance intent.

Bots to test

Essential keeps the matrix compact. Extended adds user fetchers and secondary training crawlers. Advertising tests OAI-AdsBot and OAI-SearchBot for ChatGPT ads landing pages.

Adds the ChatGPT Ads checks for OAI-AdsBot and OAI-SearchBot. Scores do not change.

Submit the form to populate the matrix.