About Better Robots.txt

Better Robots.txt is a WordPress plugin for guided robots.txt setup, crawler governance, and AI-ready usage signals.

It is designed to replace the default “blank textarea” approach with a workflow that is easier to understand, safer to review, and more useful for both beginners and advanced users.

What the current version focuses on

• Preset-based setup with Free, Pro, and Premium paths
• Search engine visibility by policy level
• AI and LLM governance controls
• SEO tool protection and bad bot protection
• Archive and Wayback handling
• WooCommerce crawl cleanup
• Resources, social crawlers, and ads.txt rules
• Optional llms.txt and machine-readable governance files
• Final Review & Save before publishing

What makes it different

Better Robots.txt is not positioned as a simple file editor.

It is positioned as a crawl-control workflow that helps you decide:

• what search engines should see
• what AI systems should be allowed to do
• which tool crawlers and bad bots should be limited
• how much low-value crawl noise your site should tolerate
• what final output should be published

What it is

• A guided robots.txt workflow for WordPress
• A way to reduce crawl waste
• A bridge between search hygiene and AI-era policy signaling
• A plugin that favors clarity, review, and explicit decisions

What it is not

• Not a firewall
• Not a hard anti-scraping enforcement layer
• Not a legal compliance engine
• Not a guarantee that every bot will obey the published rules

For harder enforcement, combine the plugin with infrastructure-level controls.

Canonical technical reference

• Plugin directory: https://wordpress.org/plugins/better-robots-txt/
• Canonical plugin repository: https://github.com/GautierDorval/better-robots-txt