Search Engine Visibility
Location: Step 1 — Search Engine Visibility
This step controls how openly the site is exposed to search-engine crawlers. It sets the baseline allow posture used by every subsequent step in the wizard.
What this step controls
Search-engine access follows the selected preset:
- Minimal Visibility (
minimal) — covers a small, conservative set of major search engines such as Google, Bing, and DuckDuckGo. Best when the site only needs the most established search products. - Recommended Visibility (
recommended) — extends the minimal set with major worldwide search engines. This is the default starting point for most sites. - Extended Visibility (
extended) — adds a broader list that includes additional regional and privacy-focused engines. Useful when discoverability outside the dominant search products matters. - Custom Control — exposes the per-bot UI so search-engine access can be allowed or restricted one user-agent at a time.
The category framing comes from the Better Robots.txt bot taxonomy: search crawlers are a distinct role from training, answer-retrieval, archive, or SEO-tool bots, and they are controlled here only.
How to decide
Use Minimal when:
- the audience is concentrated in markets served by the dominant engines;
- crawl budget concerns justify keeping the allow list short.
Use Recommended when:
- the site has no specific reason to restrict search visibility;
- the goal is a sensible default that covers most worldwide traffic.
Use Extended when:
- regional or privacy-focused engines (such as Qwant, Yandex, Mojeek, Naver, Brave Search) are part of the discovery strategy.
Use Custom Control when:
- an agency or operator wants module-level overrides per bot;
- an explicit allowlist or blocklist must be documented for audit reasons.
What this step does not do
This step does not:
- authenticate search-engine bots or verify they are who they claim to be;
- govern training crawlers, answer-retrieval bots, or archive bots (those are handled in later steps);
- guarantee inclusion in any specific search index — visibility depends on each engine's own ranking and discovery logic.
Plan tier
- Free:
MinimalandRecommendedare available. - Pro / Premium:
ExtendedandCustom Controlare unlocked.