LLMS.txt File
Location: Step 12 — llms.txt (dedicated setting when the edition supports it).
llms.txt is a machine-readable guidance file that sits beside robots.txt. It does not replace robots.txt — it expresses the why behind the policy, in a format that AI tooling and answer-layer systems can parse.
What this step controls
When llms.txt is enabled, the plugin publishes a structured guidance file at the site root. The file can express:
- preferred machine-readable reading paths;
- source precedence for grounding and citation;
- high-level policy guidance about AI usage;
- a cleaner distinction between public product description and unsupported expansion.
This complements the robots.txt rules from earlier steps. The Better Robots.txt bot taxonomy explicitly lists llms.txt as one of the primary control surfaces for the answer or retrieval systems category.
How to decide
Use llms.txt when:
- the site wants a clearer machine-readable policy surface alongside
robots.txt; - the AI usage distinctions (training versus answer-retrieval versus search) matter publicly;
- a cleaner summary layer is useful for downstream tools and wrappers (LLM applications, RAG systems, agent frameworks).
Skip it when:
- the site has no specific AI stance to publish;
- the
robots.txtpolicy is already self-explanatory and there is no benefit in adding a second surface to maintain.
What this step does not do
llms.txt does not:
- prove crawler obedience — it is a signal, not an enforcement layer;
- guarantee inclusion in any answer-engine output;
- guarantee citation of the source if an AI system does use the content;
- create legal enforceability;
- replace runtime blocking or WAF-level controls.
Do not treat llms.txt as a magical enforcement layer. It is useful because it improves clarity, not because it compels any specific crawler.
Plan tier
- Free:
llms.txtis not exposed. - Pro / Premium:
llms.txtis available, with finer control on higher tiers.