Advanced Settings

Location: Step 13 - Advanced Settings

This is the expert step for output shaping and manual directives.

Available Controls

• Crawl Delay (seconds)
• Consolidate User-agents
• Custom Rules

What These Settings Do

• Crawl Delay adds a delay value for bots that respect it
• Custom Rules lets you add your own directives
• Consolidate User-agents groups rules by User-agent

Plan Notes

• Crawl Delay is available on Pro and Premium