Skip to main contentSkip to content

ChatGPT Ads cannot access your website: diagnose the cause ​

A landing page can open in your browser while remaining inaccessible to an advertising crawler. To find the cause, examine the complete destination URL, its robots.txt policy and the HTTP response received by a diagnostic request separately.

Start with the exact URL used in the ad, including parameters and redirects. Record the response status, test time and User-Agent presented. A permissive crawl policy alongside an HTTP 403 response points to an access restriction for your hosting or security team to investigate.

The steps below help identify the layer that needs attention. A plugin setting cannot override a request rejected by a server or firewall. All examples use the fictional domain example.com.

Start here: what you see and who fixes it ​

What you observeWhat it establishesEvidence to collectWho usually fixes it
robots.txt disallows the ad’s path for OAI-AdsBotA declared refusal for this URLThe checker result and the deciding ruleWhoever manages robots.txt. In Better Robots.txt: Step 11 (ChatGPT Ads), the custom rule in Step 13, or, in Pro, the Custom AI Crawlers list in Step 2
No group names OAI-AdsBotNo dedicated rule; the general * group appliesThe checker result for the exact pathNobody, unless the general group disallows the path
robots.txt allows the path, but a test request gets HTTP 403A refusal at the HTTP layer, despite the policy, seen from your test locationStatus, time and time zone, User-Agent, request ID, title of the error pageYour host, CDN or security team
HTTP 401, or a login page served with status 200The page requires authenticationFinal URL and a copy of the page receivedYou or your developer: staging protection, membership or maintenance plugin
HTTP 200 with a challenge page or an empty bodyA response arrived, but the expected content was not establishedPage title, response size, any challenge referenceYour CDN or bot-protection provider
HTTP 404 at the final URLThe destination does not exist for this requestThe full redirect chainYou or your marketing team: fix the URL or the redirect
HTTP 429Rate limiting was observedTime, number of requests, Retry-After header if presentYour host or CDN rate-limiting rules
A redirect to the home page, a login page or another country siteThe crawler may never reach the advertised contentEach Location header in the chainYou, your CDN or the plugin that redirects
/robots.txt returns 5xx or does not answerCrawlers must then assume the whole site is disallowed (RFC 9309)Status of /robots.txt and the time of the testYour host
Timeout or network error during your testThe test is incompleteTime, error message, the network you tested fromRetry later; your host if it persists

1. Pin down the exact ad URL ​

Test the address the ad really uses, not your home page.

  • The full destination URL, with its parameters. OpenAI’s ad creation guide lets advertisers add tracking parameters such as UTM tags to the destination URL, and robots.txt rules can match query strings.
  • The final host after redirects. http to https, example.com to www.example.com, or a regional site: robots.txt applies per host, so read the file of the host that serves the page.
  • Each redirect. A chain that ends on a login page, a 404 or another country site can keep the crawler away from the advertised content.
  • The resources the page needs. Stylesheets, scripts and images can sit on another path or another host, with different rules.

Each ad links to a single destination, so one URL is enough to start.

2. Check the robots.txt rule for that URL ​

Run the OAI-AdsBot checker with your host and the path of the ad. It shows the rule that applies to OAI-AdsBot and to OAI-SearchBot, the two user agents OpenAI’s ad creation guide names.

  • Disallowed: fix the rule first. With Better Robots.txt, the guide to ChatGPT Ads and robots.txt explains the three options of Step 11.
  • Allowed, by a group that names OAI-AdsBot or by the general group: robots.txt is not the obstacle. Continue with step 3.
  • Indeterminate: /robots.txt could not be read in full. Fix its delivery first. A 5xx answer on that file makes crawlers assume the whole site is disallowed.

3. Compare explicit GET requests ​

Send the same request twice from a terminal: once as an ordinary client, once presenting the OAI-AdsBot User-Agent string that OpenAI publishes. Test only sites you are responsible for. The commands below use bash syntax.

bash
URL="https://example.com/offers/spring-sale/?utm_source=chatgpt"
UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot"

# 1. Ordinary request: status, number of redirects, final address
curl -sS -L -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" "$URL"

# 2. Same request, presenting the OAI-AdsBot User-Agent
curl -sS -L -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" -A "$UA" "$URL"

# 3. Response headers for that request (look for a request ID)
curl -sS -L -D - -o /dev/null -A "$UA" "$URL"

# 4. robots.txt with the same User-Agent
curl -sS -o /dev/null -w "%{http_code}\n" -A "$UA" "https://example.com/robots.txt"

For each request, write down the time and your time zone, the status code, the final address, and any request identifier shown in the headers or on the error page.

A User-Agent string does not prove identity. Your test is not OpenAI, even with the same User-Agent. A 403 that appears only with the OAI-AdsBot string strongly suggests a rule that matches that User-Agent. A 200 does not prove that the real crawler gets through: a firewall can also filter by IP address, and services that verify crawlers treat a request from your computer differently. OpenAI publishes the IP ranges of OAI-AdsBot at openai.com/adsbot.json. The list can change, so an allowlist should read it from the source rather than copy it once.

4. Read the response ​

ResponseWhat it usually means for the crawler
403 ForbiddenThe server, CDN, firewall or a security plugin refused the request. An error page titled with a server name, such as nginx or Apache, tells you which software produced the page, not which rule or which component made the decision.
401 UnauthorizedThe page asks for credentials: staging protection, basic authentication or a members-only area.
404 Not FoundThe URL, or one of its redirects, points to a page that does not exist.
429 Too Many RequestsA rate limit applied. Look for a Retry-After header.
301, 302, 307 or 308A redirect. Follow it to the end: the content at the final address is what counts.
200 with a challengeA JavaScript or CAPTCHA challenge replaced the page. The request was answered, but not with your content.
200 with an empty or unrelated bodyThe status looks fine, but the expected content was not delivered. Compare the size and title with what a browser receives.
5xxThe server failed. On /robots.txt, crawlers must then assume complete disallow.

5. Find the layer that refused the request ​

The refusal usually comes from one of four layers. The examples below are clues, not configurations to copy: each product names and orders its rules differently, and a rule pasted from elsewhere can open more than you intend.

  • WordPress. A security or firewall plugin with “bad bot” or AI-crawler lists, a maintenance or “coming soon” mode, a membership plugin, a geolocation redirect. For robots.txt itself, check whether WordPress is set to discourage search engines and whether a physical robots.txt file sits at the web root, where it takes precedence over the file WordPress generates.
  • Web server. Rules that deny requests by User-Agent pattern, a web application firewall module, or access rules in an .htaccess file or the server configuration.
  • CDN or firewall service. Bot management, managed challenges, options that block AI crawlers, rate limiting, country rules.
  • Hosting platform. Platform-wide bot protection or resource limits applied before your site sees the request.

When you ask for an exception, prefer your provider’s verified-crawler features, or OpenAI’s published IP list kept up to date, over a rule that trusts the User-Agent string alone. Anyone can copy a User-Agent.

6. Send your host a precise request ​

Copy this template and fill in each line. A precise request saves a round trip.

txt
Subject: Requests presenting the OAI-AdsBot User-Agent receive HTTP 403

Domain: example.com
Exact URL tested: https://example.com/offers/spring-sale/?utm_source=chatgpt
Date, time and time zone: 2026-11-20 14:05 (UTC-5)
User-Agent presented: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
Status received: 403 (an ordinary browser request receives 200 at the same time)
Request ID or reference: (from the response headers or the error page)
Rule involved, if known: (for example, a bot-protection or User-Agent rule)
robots.txt: allows this path for OAI-AdsBot (checked with the OAI-AdsBot checker)

Request: please identify the rule that refused this request and allow
OAI-AdsBot on this landing page, ideally through verified-crawler
features or OpenAI's published IP list (https://openai.com/adsbot.json)
rather than through the User-Agent alone.

7. Retest, then ask for a new review ​

  1. Once the change is made, repeat the requests from step 3 and the robots.txt check from step 2.
  2. In OpenAI’s Ads Manager, check the status of the ad. OpenAI’s troubleshooting guide explains that the reason appears when you hover over an ad marked “Not serving”.
  3. Edit the ad and resubmit it for review, or request another review if you believe the ad was rejected in error.
  4. If you contact OpenAI support, its troubleshooting guide lists useful details: the landing URL, the disapproval code or message, and the time you received the notice.

OpenAI’s crawler documentation gives no delay for OAI-AdsBot. Resubmitting the ad, or requesting another review, is what asks OpenAI to look at the page again.

A fictional example: permissive rules, refused requests ​

This scenario is invented to illustrate the method. It does not describe a real site or a real case.

A fictional shop advertises https://example.com/offers/spring-sale/?utm_source=chatgpt, and the ad is rejected.

  1. Checker: OAI-AdsBot and OAI-SearchBot are allowed by the general * group. robots.txt is not the cause.
  2. Browser: the page opens normally.
  3. Test requests: the ordinary request returns 200. The request presenting the OAI-AdsBot User-Agent returns 403 with a generic “403 Forbidden” page, and the same User-Agent also receives 403 on /robots.txt.
  4. Reading: the published policy is permissive, but a rule in front of WordPress refuses requests that present this User-Agent. The error page shows which software produced it; it does not prove which component decided.
  5. Next steps: the shop sends the template from step 6 to its host. Once the host has changed the rule, the shop repeats the tests and resubmits the ad.

Changing a Better Robots.txt option would not fix this case: the refusal happens before WordPress is involved.

What Better Robots.txt can and cannot do here ​

It can:

  • publish the robots.txt rules you choose for OAI-AdsBot, with a preview before saving;
  • keep that choice separate from OAI-SearchBot, GPTBot and ChatGPT-User;
  • show conflicts with your custom rules before they reach the public file;
  • state plainly what it does not verify.

It cannot:

  • change a response decided by your server, CDN, firewall or host;
  • send test requests to your pages: the plugin sends no HTTP probe;
  • confirm that the real OpenAI crawler reached your page, or that an ad will be approved.

Sources ​

Checked on November 20, 2026.

Better Robots.txt is not affiliated with or endorsed by OpenAI. OpenAI and ChatGPT are named here only to identify the system concerned.

Finding these guides useful?

Add to my Google sources