Social Media Crawlers
Location: Step 10 — Social Media Crawlers
This step controls whether social-media preview crawlers — the bots that build the link-preview card when a page is shared on a social platform — are allowed to fetch the site.
What this step controls
One toggle:
- Block Social Media Crawlers — turn it on to disallow social-media preview fetchers. Leave it off to let them fetch normally.
When enabled, the plugin emits Disallow directives under the relevant preview-bot user-agents (the family that includes the major social platforms' link-unfurling fetchers).
How to decide
Leave the toggle off when:
- pages are routinely shared on social platforms;
- having a proper image, title, and description in the link preview matters for traffic;
- the marketing or editorial team expects social previews to look polished.
Turn the toggle on when:
- the site is intentionally private or not promoted on social platforms;
- link previews are considered an unwanted secondary cache of the content;
- a stricter posture against preview crawlers is part of a broader protection-first stance.
What this step does not do
This step does not:
- prevent users from sharing the URL on social platforms — the link is shareable either way, only the preview enrichment is affected;
- block the same platforms' ads, analytics, or other crawl paths;
- guarantee the absence of a preview if the platform caches an older successful fetch.
Plan tier
This toggle is available on every tier.
Related
- Resources & Assets — open-graph images still need to be fetchable for previews to work.
- Bot Taxonomy