Archive & Wayback Control
Location: Step 5 — Archive & Wayback Control
This step controls whether archive and Wayback-style preservation services are permitted to snapshot the site, or whether the site publishes a restrictive stance against them.
What this step controls
Two choices:
- Allow Archiving — no archive-specific Disallow rules are added. Archive bots are governed only by the rules from earlier steps.
- Block Archiving — Better Robots.txt publishes explicit Disallow directives targeting the archive-bot category.
The category framing comes from the Better Robots.txt bot taxonomy, which classifies archive bots as a distinct role: they capture or replay content for preservation rather than to support search visibility or live answer retrieval.
How to decide
Use Allow Archiving when:
- the site values historical traceability of its public pages;
- preservation in third-party archives (such as the Wayback Machine) is considered neutral or positive for the brand;
- there is no legal or compliance reason to restrict archive capture.
Use Block Archiving when:
- old or removed pages should not remain easily retrievable through an external archive;
- archive snapshots create reputational, legal, or compliance exposure;
- the site profile already leans protection-first (see the Fortress preset and the Fortress for Sensitive Sites pattern).
A blocking choice is a published policy signal, not a runtime enforcement guarantee. See the next section.
What this step does not do
This step does not:
- authenticate the archive bot or verify its identity;
- prevent capture by non-cooperative bots that ignore
robots.txt; - retroactively remove existing snapshots already held by third-party archives;
- replace direct takedown requests sent to the archive provider.
The correct framing is clearer machine-readable guidance about archive intent, applied at the policy layer.
Plan tier
- Free: Allow Archiving is the default and always available.
- Premium: Block Archiving is gated to the Premium edition.