ChatGPT Ads cannot access your website: diagnose the cause
A landing page can open in your browser while remaining inaccessible to an advertising crawler. To find the cause, examine the complete destination URL, its robots.txt policy and the HTTP response received by a diagnostic request separately.
Start with the exact URL used in the ad, including parameters and redirects. Record the response status, test time and User-Agent presented. A permissive crawl policy alongside an HTTP 403 response points to an access restriction for your hosting or security team to investigate.
The steps below help identify the layer that needs attention. A plugin setting cannot override a request rejected by a server or firewall. All examples use the fictional domain example.com.
Start here: what you see and who fixes it
| What you observe | What it establishes | Evidence to collect | Who usually fixes it |
|---|---|---|---|
| robots.txt disallows the ad’s path for OAI-AdsBot | A declared refusal for this URL | The checker result and the deciding rule | Whoever manages robots.txt. In Better Robots.txt: Step 11 (ChatGPT Ads), the custom rule in Step 13, or, in Pro, the Custom AI Crawlers list in Step 2 |
| No group names OAI-AdsBot | No dedicated rule; the general * group applies | The checker result for the exact path | Nobody, unless the general group disallows the path |
| robots.txt allows the path, but a test request gets HTTP 403 | A refusal at the HTTP layer, despite the policy, seen from your test location | Status, time and time zone, User-Agent, request ID, title of the error page | Your host, CDN or security team |
| HTTP 401, or a login page served with status 200 | The page requires authentication | Final URL and a copy of the page received | You or your developer: staging protection, membership or maintenance plugin |
| HTTP 200 with a challenge page or an empty body | A response arrived, but the expected content was not established | Page title, response size, any challenge reference | Your CDN or bot-protection provider |
| HTTP 404 at the final URL | The destination does not exist for this request | The full redirect chain | You or your marketing team: fix the URL or the redirect |
| HTTP 429 | Rate limiting was observed | Time, number of requests, Retry-After header if present | Your host or CDN rate-limiting rules |
| A redirect to the home page, a login page or another country site | The crawler may never reach the advertised content | Each Location header in the chain | You, your CDN or the plugin that redirects |
/robots.txt returns 5xx or does not answer | Crawlers must then assume the whole site is disallowed (RFC 9309) | Status of /robots.txt and the time of the test | Your host |
| Timeout or network error during your test | The test is incomplete | Time, error message, the network you tested from | Retry later; your host if it persists |
1. Pin down the exact ad URL
Test the address the ad really uses, not your home page.
- The full destination URL, with its parameters. OpenAI’s ad creation guide lets advertisers add tracking parameters such as UTM tags to the destination URL, and robots.txt rules can match query strings.
- The final host after redirects.
httptohttps,example.comtowww.example.com, or a regional site: robots.txt applies per host, so read the file of the host that serves the page. - Each redirect. A chain that ends on a login page, a 404 or another country site can keep the crawler away from the advertised content.
- The resources the page needs. Stylesheets, scripts and images can sit on another path or another host, with different rules.
Each ad links to a single destination, so one URL is enough to start.
2. Check the robots.txt rule for that URL
Run the OAI-AdsBot checker with your host and the path of the ad. It shows the rule that applies to OAI-AdsBot and to OAI-SearchBot, the two user agents OpenAI’s ad creation guide names.
- Disallowed: fix the rule first. With Better Robots.txt, the guide to ChatGPT Ads and robots.txt explains the three options of Step 11.
- Allowed, by a group that names OAI-AdsBot or by the general group: robots.txt is not the obstacle. Continue with step 3.
- Indeterminate:
/robots.txtcould not be read in full. Fix its delivery first. A 5xx answer on that file makes crawlers assume the whole site is disallowed.
3. Compare explicit GET requests
Send the same request twice from a terminal: once as an ordinary client, once presenting the OAI-AdsBot User-Agent string that OpenAI publishes. Test only sites you are responsible for. The commands below use bash syntax.
URL="https://example.com/offers/spring-sale/?utm_source=chatgpt"
UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot"
# 1. Ordinary request: status, number of redirects, final address
curl -sS -L -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" "$URL"
# 2. Same request, presenting the OAI-AdsBot User-Agent
curl -sS -L -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" -A "$UA" "$URL"
# 3. Response headers for that request (look for a request ID)
curl -sS -L -D - -o /dev/null -A "$UA" "$URL"
# 4. robots.txt with the same User-Agent
curl -sS -o /dev/null -w "%{http_code}\n" -A "$UA" "https://example.com/robots.txt"For each request, write down the time and your time zone, the status code, the final address, and any request identifier shown in the headers or on the error page.
A User-Agent string does not prove identity. Your test is not OpenAI, even with the same User-Agent. A 403 that appears only with the OAI-AdsBot string strongly suggests a rule that matches that User-Agent. A 200 does not prove that the real crawler gets through: a firewall can also filter by IP address, and services that verify crawlers treat a request from your computer differently. OpenAI publishes the IP ranges of OAI-AdsBot at openai.com/adsbot.json. The list can change, so an allowlist should read it from the source rather than copy it once.
4. Read the response
| Response | What it usually means for the crawler |
|---|---|
| 403 Forbidden | The server, CDN, firewall or a security plugin refused the request. An error page titled with a server name, such as nginx or Apache, tells you which software produced the page, not which rule or which component made the decision. |
| 401 Unauthorized | The page asks for credentials: staging protection, basic authentication or a members-only area. |
| 404 Not Found | The URL, or one of its redirects, points to a page that does not exist. |
| 429 Too Many Requests | A rate limit applied. Look for a Retry-After header. |
| 301, 302, 307 or 308 | A redirect. Follow it to the end: the content at the final address is what counts. |
| 200 with a challenge | A JavaScript or CAPTCHA challenge replaced the page. The request was answered, but not with your content. |
| 200 with an empty or unrelated body | The status looks fine, but the expected content was not delivered. Compare the size and title with what a browser receives. |
| 5xx | The server failed. On /robots.txt, crawlers must then assume complete disallow. |
5. Find the layer that refused the request
The refusal usually comes from one of four layers. The examples below are clues, not configurations to copy: each product names and orders its rules differently, and a rule pasted from elsewhere can open more than you intend.
- WordPress. A security or firewall plugin with “bad bot” or AI-crawler lists, a maintenance or “coming soon” mode, a membership plugin, a geolocation redirect. For robots.txt itself, check whether WordPress is set to discourage search engines and whether a physical
robots.txtfile sits at the web root, where it takes precedence over the file WordPress generates. - Web server. Rules that deny requests by User-Agent pattern, a web application firewall module, or access rules in an
.htaccessfile or the server configuration. - CDN or firewall service. Bot management, managed challenges, options that block AI crawlers, rate limiting, country rules.
- Hosting platform. Platform-wide bot protection or resource limits applied before your site sees the request.
When you ask for an exception, prefer your provider’s verified-crawler features, or OpenAI’s published IP list kept up to date, over a rule that trusts the User-Agent string alone. Anyone can copy a User-Agent.
6. Send your host a precise request
Copy this template and fill in each line. A precise request saves a round trip.
Subject: Requests presenting the OAI-AdsBot User-Agent receive HTTP 403
Domain: example.com
Exact URL tested: https://example.com/offers/spring-sale/?utm_source=chatgpt
Date, time and time zone: 2026-11-20 14:05 (UTC-5)
User-Agent presented: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
Status received: 403 (an ordinary browser request receives 200 at the same time)
Request ID or reference: (from the response headers or the error page)
Rule involved, if known: (for example, a bot-protection or User-Agent rule)
robots.txt: allows this path for OAI-AdsBot (checked with the OAI-AdsBot checker)
Request: please identify the rule that refused this request and allow
OAI-AdsBot on this landing page, ideally through verified-crawler
features or OpenAI's published IP list (https://openai.com/adsbot.json)
rather than through the User-Agent alone.7. Retest, then ask for a new review
- Once the change is made, repeat the requests from step 3 and the robots.txt check from step 2.
- In OpenAI’s Ads Manager, check the status of the ad. OpenAI’s troubleshooting guide explains that the reason appears when you hover over an ad marked “Not serving”.
- Edit the ad and resubmit it for review, or request another review if you believe the ad was rejected in error.
- If you contact OpenAI support, its troubleshooting guide lists useful details: the landing URL, the disapproval code or message, and the time you received the notice.
OpenAI’s crawler documentation gives no delay for OAI-AdsBot. Resubmitting the ad, or requesting another review, is what asks OpenAI to look at the page again.
A fictional example: permissive rules, refused requests
This scenario is invented to illustrate the method. It does not describe a real site or a real case.
A fictional shop advertises https://example.com/offers/spring-sale/?utm_source=chatgpt, and the ad is rejected.
- Checker: OAI-AdsBot and OAI-SearchBot are allowed by the general
*group. robots.txt is not the cause. - Browser: the page opens normally.
- Test requests: the ordinary request returns 200. The request presenting the OAI-AdsBot User-Agent returns 403 with a generic “403 Forbidden” page, and the same User-Agent also receives 403 on
/robots.txt. - Reading: the published policy is permissive, but a rule in front of WordPress refuses requests that present this User-Agent. The error page shows which software produced it; it does not prove which component decided.
- Next steps: the shop sends the template from step 6 to its host. Once the host has changed the rule, the shop repeats the tests and resubmits the ad.
Changing a Better Robots.txt option would not fix this case: the refusal happens before WordPress is involved.
What Better Robots.txt can and cannot do here
It can:
- publish the robots.txt rules you choose for OAI-AdsBot, with a preview before saving;
- keep that choice separate from OAI-SearchBot, GPTBot and ChatGPT-User;
- show conflicts with your custom rules before they reach the public file;
- state plainly what it does not verify.
It cannot:
- change a response decided by your server, CDN, firewall or host;
- send test requests to your pages: the plugin sends no HTTP probe;
- confirm that the real OpenAI crawler reached your page, or that an ad will be approved.
Sources
Checked on November 20, 2026.
- OpenAI: Overview of OpenAI crawlers: the OAI-AdsBot User-Agent string and its published IP list.
- OpenAI Help Center: Creating ads for ChatGPT Ads: one destination URL per ad, tracking parameters, and landing pages that must not block OAI-AdsBot and OAI-SearchBot.
- OpenAI Help Center: Troubleshooting common issues: ad status, common rejection causes, resubmission and review requests.
- OpenAI Help Center: Advertiser guidance for allowing OpenAI web crawlers: further reading.
- RFC 9309, Robots Exclusion Protocol: what crawlers do when robots.txt is unavailable or unreachable, and why its rules are not access authorization.
Better Robots.txt is not affiliated with or endorsed by OpenAI. OpenAI and ChatGPT are named here only to identify the system concerned.
Related pages
- ChatGPT Ads and robots.txt: configure the OAI-AdsBot policy in WordPress.
- OAI-AdsBot checker: test the rule that applies to your landing page.
- robots.txt declaration vs technical blocking: why a rule and a firewall answer different questions.
- Ads & Revenue setting: the advertiser control and the seller declaration files.
- Troubleshooting: other common questions about Better Robots.txt.