Bad Bots Protection
Location: Step 4 — Bad Bots Protection
This step enables the curated bad-bot lists used by Better Robots.txt. These bots fall into the Better Robots.txt bot taxonomy category of low-value or abusive bots: scrapers, content harvesters, and crawlers that create cost or noise without meaningful value to the site.
What this step controls
Two controls:
- Enable Bad Bots Protection — turns on the curated blocklist.
- Use Full AI-Curated List — replaces the basic list with a broader, AI-augmented list that covers more known bad agents.
When enabled, the plugin adds explicit Disallow directives under each blocked user-agent.
How to decide
Use the basic list when:
- the site is new to crawl governance and wants a safer default;
- visible bad-bot traffic is moderate and the cost of false positives matters.
Use the full AI-curated list when:
- the site has already absorbed significant scraping or extraction pressure;
- broader coverage outweighs the small risk of catching a legitimate but unusual crawler;
- the operator wants the strongest published stance against low-value bots.
Leave off only when:
- the site explicitly wants to remain crawlable by everyone for research or transparency reasons;
- a separate WAF or rate-limit layer already handles abuse.
What this step does not do
This step does not:
- block any bot that ignores
robots.txt(and many low-value bots do exactly that); - authenticate the user-agent string;
- replace WAF, edge ACLs, or rate limiting when the real problem is abusive request volume.
This is one of the clearest ways to reduce cooperative crawler noise. For non-cooperative abuse, an infrastructure layer is required.
Plan tier
- Free: basic curated list.
- Pro / Premium: the fuller, broader AI-curated list.