Bad Bots Protection
Location: Step 4 — Bad Bots Protection
This step enables the curated bad-bot lists used by Better Robots.txt.
Controls
- Enable Bad Bots Protection
- Use Full AI-Curated List
Editions
- Free: basic list
- Pro / Premium: fuller, broader list
What this changes
Turning this step on adds explicit crawler blocks for unwanted or low-value agents.
Why it matters
This is one of the clearest ways to reduce crawler noise, low-value scraping, and unnecessary server hits without shutting down all discovery.