SEO Tool Protection
Location: Step 3 — SEO Tool Protection
This step blocks groups of SEO analysis crawlers. SEO tools fall into the Better Robots.txt bot taxonomy category of SEO tool bots: they crawl for research, monitoring, and competitive intelligence rather than to support search discovery for end users.
What this step controls
Two toggles, grouped by crawler tier:
- Block Basic SEO Tools — covers the most common SEO crawlers, including
SemrushBotandDotBot. - Block Extra SEO Tools — covers a larger crawler group, including
AhrefsBot,MJ12bot,Serpstat,Screaming Frog, and others.
Both toggles emit explicit Disallow directives under each blocked user-agent.
How to decide
Use Block Basic SEO Tools when:
- the visible crawl load from common SEO tools is unwanted;
- competitive intelligence visibility is a concern;
- there is no business reason to remain crawlable by the basic tier.
Use Block Extra SEO Tools when:
- a stricter posture against deep-crawl SEO tools is justified;
- the site has experienced aggressive extraction or content scraping;
- the operator does not personally use the larger crawler group.
Leave the toggles off when:
- the site (or its agency) actively uses some of these tools for SEO research and needs the crawlers to remain functional;
- partial discoverability through SEO indexes brings value.
What this step does not do
This step does not:
- authenticate or verify SEO bot identity (a user-agent string can be spoofed);
- prevent crawl by tools that ignore
robots.txt; - replace WAF or rate-limit controls when the issue is server load.
The correct framing is a policy signal at the robots.txt layer. Vendors that respect robots.txt will honour it; the rest is an infrastructure concern.
Plan tier
- Free: SEO tool blocking is not exposed.
- Pro / Premium: both toggles are available, with the larger crawler group on higher tiers.