Test and optimize your robots.txt

Run a free test to spot configuration issues and find useful improvements. Enter your domain, review the report and see what to adjust.

Website URL Test my robots.txt for free

What should your site allow?

Choose the goal that best fits your site. The recommendations will use that goal.

AI search open, training restricted Allow AI search bots (ChatGPT, Claude, Perplexity), block training crawlers. Maximum AI visibility Allow all legitimate AI bots for maximum citation; aggressive scrapers handled elsewhere. Publisher protection Strict training control with selective AI search visibility; for media and editorial sites. WordPress safe default Recommended baseline for typical WordPress sites: AI search OK, training restricted. Strict crawler restriction Block all AI bots; keep only minimal traditional search visibility.

Help me choose my intent profile Four questions can recommend an existing profile before the audit.

Answer a few questions to find a suitable option. You can change it before running the test.

Should traditional search engines be able to crawl the site?

Yes, keep search crawling available Limit crawling as much as possible I am not sure

Do you want to declare a restriction against AI training on your content?

Declare an AI-training restriction Do not restrict AI training in this profile I am not sure

Do you want to allow crawlers used for AI answers or AI search when that supports discoverability?

Allow AI-search and user-request crawlers Restrict AI answer or user-request crawlers I am not sure

What is your main priority?

Balance discoverability and control Maximize discoverability Limit AI use as much as possible Protect editorial or sensitive content Keep a cautious WordPress baseline

This helper selects one of the available profiles. You can review and change the choice before testing.

Recommend a profile

Free. No sign-up. Full audit results are cached briefly by domain for performance. We keep the previous score for up to 30 days to show progress, plus short-lived diagnostic attempts for reliability. Long-term stats are aggregated and anonymised.