Google Search, Lighthouse, llms.txt, and WordPress

The current llms.txt debate is easy to misread.

One side will say:

Google says llms.txt is not needed, so it is useless.

The other side will say:

Lighthouse checks llms.txt, so it must be a ranking factor.

Both readings are too weak.

The stronger interpretation is that Search visibility and agentic browsing readiness are different layers.

Google Search guidance says you do not need to create llms.txt, AI-specific files, Markdown versions, or special AI markup to appear in generative AI Search features. Chrome Lighthouse’s experimental Agentic Browsing documentation, however, checks for llms.txt as a machine-readable summary at the domain root.

That does not create a simple contradiction. It creates a product and audit distinction:

• Search asks whether Google can crawl, index, retrieve, rank, and surface useful content.
• Agentic Browsing asks whether a site is structured for machine interaction.

WordPress teams need to understand both.

What Google Search is saying

Google Search is pushing back against shallow “AI optimization” hacks.

The message is basically:

• do not invent special AI markup just to appear in generative Search features;
• do not assume llms.txt gives special treatment;
• do not rewrite content only for AI systems;
• keep doing strong SEO work;
• make pages crawlable, useful, well-structured, and accessible.

That guidance is important because it prevents a market of fake shortcuts.

A WordPress site should not add llms.txt because someone promised direct ranking gains in AI Overviews.

What Chrome Lighthouse is saying

Chrome Lighthouse is asking a different question.

Its Agentic Browsing category is not a classic Search ranking document. It is an experimental audit surface that checks whether a site has properties useful for machine interaction.

One of those properties is a machine-readable site summary, usually exposed through llms.txt.

That does not mean:

• the file ranks pages;
• the file enforces crawler behavior;
• every AI system consumes it;
• every WordPress site must have it;
• passing the check means the site is agent-ready.

It means the file is now visible inside a mainstream developer audit context.

That matters.

The WordPress problem

WordPress is exactly where this distinction becomes operational.

Many WordPress sites have:

• plugin-generated metadata;
• multiple SEO extensions over time;
• auto-generated archives;
• messy tag pages;
• multilingual alternates;
• cache/CDN transformations;
• builders that produce complex markup;
• forms that are hard for machines to interpret;
• unclear distinctions between search bots, training bots, and user-triggered fetchers.

Search may still crawl the site. That does not mean the site is easy for agents to interpret or act on.

A clean robots.txt helps with access. A good llms.txt helps with orientation. A policy file helps with interpretation. Source pages help with retrieval. Accessibility and stable UI help with interaction.

None of those layers replaces the others.

How Better Robots.txt should be positioned

Better Robots.txt should not be positioned as:

Install this plugin and rank better in AI because Lighthouse checks llms.txt.

That would be an overclaim.

The stronger position is:

Better Robots.txt helps WordPress teams publish a coherent crawl-governance and machine-guidance layer: robots.txt, AI crawler segmentation, optional llms.txt, policy signals, and audit interpretation. That layer contributes to agentic readiness, but it does not replace source-page quality, accessibility, stable UI, logs, or runtime enforcement.

That is both more accurate and more defensible.

The layered model

Layer	Main question	Better Robots.txt role
Search crawl	Can search crawlers reach the right pages?	Strong support through robots.txt, sitemap awareness, and crawl hygiene
AI crawler posture	Are crawler purposes separated?	Strong support through AI crawler rules and audit guidance
Machine summary	Can machines find the priority pages quickly?	Support through llms.txt guidance and checker pages
Governance	Is the site’s policy coherent?	Support through AI usage signals and governance-file awareness
Source-page quality	Are there stable pages worth citing?	Indirect support through routing and documentation, not content creation
Interaction readiness	Can agents use forms, buttons, and workflows safely?	Limited support; requires accessibility, UX, front-end, and QA work

What WordPress teams should do now

1. Do not panic-add llms.txt

A placeholder file is weak. Add llms.txt only when it is accurate, maintained, and connected to real source pages.

Read: llms.txt and Lighthouse audit for WordPress.

2. Do not block AI blindly

Blocking every AI-looking crawler may damage search discovery or user-triggered retrieval while failing to enforce the behavior you actually care about.

Read: AI training vs AI search crawlers.

3. Fix the crawl baseline first

Before agentic readiness, make sure the site is not breaking classic crawlability.

Read: Robots.txt checker for AI crawlers.

4. Build real source pages

The best llms.txt file cannot make weak pages useful. If the site wants to be cited, it needs pages that answer concrete questions.

Read: AI visibility.

5. Audit interaction readiness separately

A site can pass robots.txt and llms.txt checks but still be hard for agents to use.

Read: Lighthouse Agentic Browsing for WordPress and Agentic readiness checklist for WordPress.

The strategic conclusion

llms.txt is not the revolution by itself.

The important signal is that machine readability is moving from theory into developer tooling.

WordPress sites will increasingly need to answer 4 separate questions:

1. Are we visible to Search?
2. Are we clear to AI retrieval systems?
3. Are we governed for machine access and use?
4. Are we usable by agents?

Better Robots.txt should own the WordPress crawl-governance and machine-guidance layer in that stack.

That is a stronger position than pretending one file controls the future of AI search.

Read next

• Agentic readiness for WordPress
• Lighthouse Agentic Browsing for WordPress
• llms.txt and Lighthouse audit for WordPress
• robots.txt, llms.txt, and WebMCP
• The machine governance file stack

Official references

This interpretation is anchored in three public sources:

• Google Search generative AI optimization guidance: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
• Chrome Lighthouse llms.txt audit documentation: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
• Chrome Lighthouse Agentic Browsing scoring documentation: https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring