URL × bot matrix explained

The URL × bot matrix is the part of /check that answers a precise question:

According to this site’s robots.txt, can this bot access this specific URL?

This is different from a global score. It is a URL-level access test.

What the matrix tests

The matrix tests selected URLs against selected crawler user-agents, such as:

• Googlebot;
• Bingbot;
• GPTBot;
• OAI-SearchBot;
• ClaudeBot;
• Claude-SearchBot;
• PerplexityBot;
• Google-Extended.

For each combination, the matrix reports whether access is allowed or blocked according to robots.txt matching.

What makes it different from a presence check

A presence check only asks:

Is GPTBot mentioned somewhere?

The matrix asks:

For this URL, which group matched?
Which rule won?
Was access allowed by default, explicitly allowed, or blocked?

That distinction matters because a site can mention a bot while still allowing or blocking different paths through wildcard groups and more specific rules.

Winning rule explanation

A good URL × bot result should show:

Bot: GPTBot
URL: /blog/example/
Decision: blocked
Matched group: User-agent: GPTBot
Winning rule: Disallow: /
Line: robots.txt line number when available

This makes the audit explainable, not just colourful.

What the matrix does not prove

The matrix does not prove that a bot actually visited the URL.

It does not prove crawler obedience.

It does not replace server logs.

It does not replace bot verification.

It shows the declared access decision according to the public robots.txt rules available at scan time.

How it fits with Better Robots profiles

The matrix answers the access question.

The selected governance profile answers the intent question.

Together, they allow Better Robots to say:

This bot can access this URL.
This access is aligned, misaligned or neutral for the selected profile.

That is what separates Better Robots from a flat robots.txt validator.