Check your OAI-AdsBot policy
Check the robots.txt rule that applies to OAI-AdsBot for your landing page. Enter your site and the page used in your ad to see the matching group, the deciding rule and the declared crawl-policy result.
“Allowed by robots.txt” describes the instructions published for cooperative crawlers. It does not prove that a firewall will accept the request. This checker does not send any request to your landing page, so HTTP access remains unverified.
Prefer to start from the tester itself? Open the advertising URL tester and fill in your site there.
What the checker tests
The form and the link open the URL tester of the free Better Robots audit with the advertising set of user agents. It tests two of them:
OAI-AdsBot, the crawler OpenAI uses to review landing pages submitted as ads on ChatGPT;OAI-SearchBot, which OpenAI’s ad creation guide names next to OAI-AdsBot among the user agents landing pages must not block.
GPTBot is not part of this set. It is OpenAI’s training crawler, and its rule is a separate choice.
For each path, the tester fetches /robots.txt from the host you enter, selects the group that applies to each crawler and applies the robots.txt standard, RFC 9309: rules are compared with the path and its query string, the longest matching rule wins, and Allow wins a tie. The result gives the decision for each crawler and path, the rule that decided it, and where that rule came from.
Use the final host of your landing page. www.example.com and example.com can serve different robots.txt files, and when one redirects to the other, the crawler ends up reading the second. If you need to share the result, note the date and time of your check.
How to read the result
| Result | What it means | What to do |
|---|---|---|
| Allowed by a group that names OAI-AdsBot | Your robots.txt has a group for OAI-AdsBot, and its rules allow this path. | Nothing for robots.txt. If the ad is still refused, look at the HTTP layer. |
Allowed by the general * group | No group names OAI-AdsBot, so the crawler follows User-agent: *, which allows this path. This is not a block. | Optional: choose Allow public pages, respecting site exclusions in Better Robots.txt if you want an explicit rule that mirrors your general rules. |
| Allowed because no group applies | The file has neither a group for OAI-AdsBot nor a User-agent: * group, so no rule applies. | Nothing for robots.txt, unless you meant to publish rules. |
| Disallowed | A Disallow rule wins for this path, either in the group that names OAI-AdsBot or, when no group names it, in the general group. The deciding rule is shown. | Change the rule if this is the page used in your ad. See what to do if OAI-AdsBot is disallowed. |
| Indeterminate | The tester could not read a complete robots.txt: the file was unavailable, unreachable, redirected too often or to a destination the tester does not follow, or larger than the amount it reads. | Fix the delivery of /robots.txt, then check again. The details are below. |
| HTTP response: not verified | The tester does not request your landing page. | Run an HTTP test yourself with the ChatGPT Ads troubleshooting guide. |
| Access by a verified OpenAI crawler: not verified | No public tool can confirm that the real OpenAI crawler reaches your page. | Look at your server or CDN logs if you need that evidence. |
None of these results confirms access by OpenAI or the approval of an ad. They describe the rules your site publishes.
When the result is indeterminate
The robots.txt standard says what a crawler should do when it cannot read the file. That fallback is never a rule you published, and it says nothing about how your landing page answers.
What happened to /robots.txt | What RFC 9309 says a crawler does | How to read it |
|---|---|---|
| A 4xx status such as 403 or 404: the file is “unavailable” | The crawler may access any resource on the site. | No policy was read. A 403 on /robots.txt often means the same protection also blocks your pages. |
| A 5xx status or a network failure: the file is “unreachable” | The crawler must assume complete disallow. | Treat it as a blocker until the file loads again. |
| More than five consecutive redirects | The crawler may treat the file as unavailable. | Serve /robots.txt from its final address, or with fewer redirects. |
| A redirect the tester stopped following | No rules were read. | Check where /robots.txt redirects and why. |
| A file longer than the tester reads (about 500 KiB) | A crawler that limits how much it parses must read at least 500 kibibytes; content beyond its limit can be ignored. | Shorten the file: rules after the limit were not evaluated. |
For developers: the same result in JSON
Add format=json to the tester address to receive the matrix as JSON. Each cell reports where its decision came from in policy_source (explicit_group, wildcard_fallback, default or unavailable) and the kind of group that matched in matched_group_kind. A robots_fetch object records how /robots.txt was retrieved and which RFC 9309 fallback applies. Its landing_page_http and verified_openai_access fields always read not_verified.
If OAI-AdsBot is disallowed
On WordPress with Better Robots.txt 3.1.8 or later:
- Open Step 11, Ads & Revenue, section ChatGPT Ads.
- Choose Allow public pages, respecting site exclusions. The plugin adds an OAI-AdsBot group that copies your general rules, without opening the paths you exclude for every crawler.
- If the plugin reports a conflict, open the custom rule that names OAI-AdsBot, often in Step 13, Advanced Settings, and resolve it there.
- Read the preview in Step 14, Review & Save, save, then run this check again.
If the deciding rule comes from your general group, for example Disallow: /offers/, the landing page is excluded for every crawler. Decide whether that exclusion is still wanted: the ChatGPT Ads setting will not override it.
On another platform, edit robots.txt by hand. Remove or narrow the group that disallows OAI-AdsBot, or give it the same rules as your general group. Avoid a bare Allow: /, which would open everything the general group closes. The guide to ChatGPT Ads and robots.txt explains why with a worked example.
Better Robots.txt Pro, from version 3.1.8, can also import an audit recommendation that allows OAI-AdsBot. You see a preview before anything is applied, and the audit never recommends disallowing OAI-AdsBot.
Check the rule by hand
You can reproduce the decision with the file itself. Open your site’s robots.txt, for example https://example.com/robots.txt, and follow three steps.
- Find every group whose
User-agentline namesOAI-AdsBot, in any letter case. Several groups for the same crawler count as one. If there is none, use theUser-agent: *group. If there is neither, no rule applies and the path is allowed. - In that group only, find the
AllowandDisallowrules whose path matches the beginning of your URL path, query string included.*matches any sequence of characters, and$marks the end of the address. - The longest matching rule wins. When an
Allowrule and aDisallowrule are equally long,Allowwins. No matching rule means the path is allowed.
A common trap, shown in a fictional file:
User-agent: *
Disallow: /private/
User-agent: OAI-AdsBot
Allow: /offers/| Path | Decision for OAI-AdsBot | Why |
|---|---|---|
/offers/spring-sale/ | Allowed | Allow: /offers/ matches |
/private/report/ | Allowed | The OAI-AdsBot group has no rule for this path, and it inherits nothing from User-agent: * |
/ | Allowed | No rule matches |
The author probably wanted /private/ to stay closed. Because a group that names a crawler replaces the general group for that crawler, /private/ is open to OAI-AdsBot. This is why Better Robots.txt copies your general rules into the OAI-AdsBot group instead of writing a single Allow line.
What the checker sends
The tester requests only the public /robots.txt of the host you enter. It does not present itself as an OpenAI crawler, and it does not visit your landing page. The audit’s data policy describes what it records.
Better Robots.txt is not affiliated with or endorsed by OpenAI. OpenAI and ChatGPT are named here only to identify the crawlers being tested.
Related pages
- ChatGPT Ads and robots.txt: the three options of the plugin and a before-and-after example.
- ChatGPT Ads website unavailable: when the rules allow the crawler but the server refuses it.
- OpenAI crawlers compared: advertising, search, training and on-demand access.
- OAI-SearchBot checker: the search crawler named next to OAI-AdsBot.
- URL × bot matrix explained: how the tester decides for each path.
- Ads & Revenue setting: Step 11 of the plugin.