Settings Overview
Better Robots.txt is organized as a guided sequence rather than a flat list of toggles. The wizard walks through 14 numbered steps, preceded by a Mode Selection screen, and each step covers one crawler-governance decision at a time.
This page is the index. Click any step to see what it controls, how to decide, what it does not do, and which plan tier it requires.
Before the wizard
- Mode Selection — pick Essential, AI-First, Fortress, or Custom as the starting preset.
The 14 numbered steps
| Step | Setting | Category |
|---|---|---|
| 1 | Search Engine Visibility | Baseline allow posture |
| 2 | AI and LLM Governance | Training, answer-retrieval, signals |
| 3 | SEO Tool Protection | SEO crawler categories |
| 4 | Bad Bots Protection | Low-value or abusive crawlers |
| 5 | Archive & Wayback Control | Archive bots |
| 6 | Global Settings | Sitemaps, type, core protection |
| 7 | Spam, Feeds & Crawl Traps | Cleanup of low-value URL patterns |
| 8 | E-commerce Optimization | WooCommerce-style cleanup |
| 9 | Resources & Assets | CSS, JS, and image allow rules |
| 10 | Social Media Crawlers | Preview-bot policy |
| 11 | Ads & Revenue | ads.txt / app-ads.txt access |
| 12 | llms.txt | Machine-readable AI guidance |
| 13 | Advanced Controls | Module-level overrides |
| 14 | Review & Save | Preview and publish |
Why this matters
A robots.txt policy is easier to manage when the operator can reason step by step instead of trying to write the entire file by hand. The numbered structure also makes governance auditable: each step maps to a documented decision in the Better Robots.txt bot taxonomy, which is the canonical source for category framing across the product.
Recommended path
- Pick a preset at Mode Selection.
- Walk through Steps 1 to 13 in order — do not skip ahead.
- Stop at Review & Save and inspect the generated
robots.txtbefore publishing. - Re-audit after publication with the live checkers (see Related).