Cloudflare–OpenAI Discovery Pilot: How to Get Found in ChatGPT Search

Cloudflare and OpenAI published early results from their web-discovery research pilot on October 15, 2026. Using signals from a participating cohort of about 1% of Cloudflare's network, the pilot helped OpenAI discover and index hundreds of millions of previously unknown pages.

The result matters to publishers because it shows that AI-search discovery is becoming more signal-driven. A page can be useful even when it has few links, but a crawler still needs to find the URL, distinguish it from soft 404s and duplicates, notice meaningful changes, and access the content without being blocked by a CDN or firewall.

Fact versus recommendation: the dates, cohort size, discovery result, crawler identities and product behavior below come from Cloudflare and OpenAI. The audit workflow, priority order and measurement framework are Rubic8 recommendations. Neither company has published a ranking formula, traffic uplift, citation rate or revenue benchmark for participating sites.

What the Cloudflare–OpenAI Pilot Actually Found

Cloudflare announced the research pilot on July 8, 2026. Its stated goal was to test whether network-level information—such as content freshness, traffic quality and actual page changes—could help AI search systems crawl and index the open web more efficiently.

On October 15, Cloudflare and OpenAI reported the first results. Signals from roughly 1% of Cloudflare's network helped OpenAI discover and index hundreds of millions of new pages. Cloudflare says these pages give ChatGPT Search more timely and relevant sources from which to answer questions.

That number is a pilot-wide result, not a promise for an individual website. The announcement does not say that every participating page was indexed, cited or sent referral traffic. OpenAI independently evaluates and ranks discovered pages according to relevance and helpfulness.

The companies also identified several sources of crawl waste:

• requests for URLs that do not exist;
• sites that redirect arbitrary URLs to the homepage;
• duplicate copies of the same content;
• unchanged pages fetched repeatedly;
• and isolated compromised pages that may be unsafe.

The next phase will improve page selection and change signals and examine what happens after discovery. Cloudflare explicitly says the partners are still working to understand how discovery translates into business outcomes.

Discovery, Indexing, Ranking and Citation Are Different

A practical AI-search strategy starts by separating four stages that are often treated as one:

This distinction prevents an easy mistake: allowing a bot makes a page eligible for discovery, but does not make it authoritative, useful or relevant. Technical access is necessary; content quality and query fit remain separate problems.

Which OpenAI Crawler Controls ChatGPT Search?

OpenAI's official crawler documentation describes three relevant user agents with different purposes:

• OAI-SearchBot supports automatic discovery for ChatGPT Search. Blocking it prevents page content from appearing in search answers, although a title and navigational link may still surface in limited circumstances.
• GPTBot controls whether content may be crawled for training OpenAI's generative foundation models.
• ChatGPT-User can fetch pages for certain user-initiated actions. It is not the automatic search crawler and is not the control for ChatGPT Search inclusion.

OpenAI says the OAI-SearchBot and GPTBot preferences are independent. A publisher can allow search discovery while disallowing training. OpenAI also notes that robots.txt changes may take about 24 hours to be reflected.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

This example expresses a policy; it does not prove that the CDN, WAF or origin actually permits the crawler. Use Rubic8's Robots.txt Generator & Tester to review the rule logic, then verify the live response and security logs.

A Publisher Checklist for Better ChatGPT Search Discovery

1. Test every layer that can block OAI-SearchBot

Start with the live robots.txt file, then check CDN rules, bot protection, rate limits, application middleware and the origin server. A correct robots rule cannot compensate for a 403 response at the edge. OpenAI recommends allowing both OAI-SearchBot and its published IP ranges.

Do not trust a user-agent string alone in high-risk environments. It can be spoofed. Use OpenAI's published IP data or a verified-bot signal from your CDN. Cloudflare's bot reference lists separate detection IDs for GPTBot, ChatGPT-User and OAI-SearchBot; enterprise Bot Management can use those IDs, while free-plan identification relies more heavily on self-declared user agents.

2. Return honest HTTP responses

A missing URL should return 404 or 410, not a 200 homepage. A moved page should use one intentional redirect to its replacement. Repeated redirect chains, blanket homepage redirects and soft 404s make automated traffic look important while providing no useful source.

Audit parameter combinations, internal search URLs, pagination and obsolete media paths. Infinite or low-value URL spaces can consume crawl capacity and make the valuable pages harder to distinguish.

3. Consolidate duplicates

Choose one canonical URL for each piece of content. Redirect retired variants where appropriate, use a self-referencing canonical on the preferred page, and keep internal links consistent. Do not publish the same article under multiple language, tag or query-string URLs without a clear canonical strategy.

The pilot's filtering work reinforces a familiar SEO principle: duplicate discovery is not additional visibility. It is additional ambiguity.

4. Make meaningful changes visible

The pilot is testing signals that indicate when a page is worth revisiting. Publishers should support that goal with accurate metadata and visible editorial practice:

• update the page when facts, prices, specifications or procedures change;
• show a truthful reviewed or updated date;
• update XML sitemap lastmod only after a substantive change;
• purge stale CDN content after important corrections;
• and keep the canonical URL stable when the topic remains the same.

Changing a date without improving the content is not a freshness strategy. It creates a noisy signal and can reduce reader trust.

5. Make deep pages discoverable

Cloudflare notes that a specific manual, current local menu or exact government requirement can be useful even when few sites link to it. Help crawlers reach such pages through contextual internal links, category pages, breadcrumbs and a current XML sitemap.

A URL that appears only after client-side interaction, requires an internal search form or sits in an orphaned archive is a fragile discovery target. Important content should have a stable URL and at least one crawlable path from the site structure.

6. Publish information worth citing

Discovery cannot manufacture usefulness. Pages are stronger candidates when they complete a real task or provide information that is difficult to replace: original measurements, first-hand experience, exact compatibility notes, current local details, transparent calculations, screenshots, working examples or a useful interactive tool.

Use descriptive titles and headings, answer the main question early, identify the publisher and author, and cite primary sources for claims that can change. Avoid hiding essential text behind images or JavaScript-only interfaces.

7. Separate search access from training policy

Decide what the business wants before changing crawler settings. Search discovery, model training and user-triggered agents are different uses. A blanket block on every AI-related request can remove a discovery channel that the publisher intended to keep.

Rubic8's Cloudflare AI Crawler Controls guide explains the Search, Training and Agent categories in detail. The new pilot does not override those choices. Cloudflare states that a discovery signal does not grant permission to access a page and that the pilot uses web metadata for search discovery rather than sharing page content for model training.

8. Monitor crawler and referral evidence separately

Cloudflare AI Crawl Control can report requests by crawler, operator, category, hostname, path and response status. OpenAI referrals may arrive from domains such as chatgpt.com or openai.com. Track both datasets, but do not treat them as interchangeable.

A crawler request shows access activity. A referral shows that a user followed a link. Neither alone proves that a page ranked prominently or caused a conversion.

Create a baseline before changing policy. Measure:

• verified OAI-SearchBot requests and response codes;
• pages crawled versus pages receiving referrals;
• referral sessions from AI platforms;
• engagement and conversions from those sessions;
• server load and repeated fetches;
• and the freshness of cited information when your pages do appear.

How to Audit One Important Page

Choose a high-value page instead of trying to optimize the entire site at once.

1. Request the canonical URL without cookies and confirm a clean HTTP 200 response.
2. Check that robots.txt allows OAI-SearchBot for the page and its required assets.
3. Verify that the CDN or WAF recognizes legitimate OpenAI traffic rather than challenging it.
4. Confirm a unique title, one clear H1, descriptive headings and useful text in the returned HTML.
5. Check canonical, robots meta, language and structured data for consistency with the visible page.
6. Ensure the page has crawlable internal links and appears in the correct XML sitemap.
7. Remove soft-404 behavior, duplicate variants and unnecessary redirect hops.
8. Record the baseline crawler activity and AI referral traffic before making changes.

Rubic8's SEO & AI Visibility Checker can inspect many of these static signals on a public page. Its result is a technical audit, not a ChatGPT ranking or citation prediction.

What Publishers Should Not Conclude

• Cloudflare is not announcing a ChatGPT ranking boost. It is providing discovery signals; OpenAI ranks independently.
• The pilot does not guarantee citations or traffic. Its first reported outcome is broader discovery and indexing.
• Freshness does not mean changing timestamps daily. Meaningful content changes are the useful signal.
• Allowing OAI-SearchBot does not opt a site into GPTBot training. OpenAI documents separate controls.
• The pilot does not make traditional crawlability obsolete. Stable URLs, honest status codes, internal links, canonicalization and sitemaps still reduce ambiguity.
• This is not evidence that an llms.txt file improves visibility. The announcement focuses on network and page-change signals, not a new publisher file format.

For the broader distinction between experiments and verified search guidance, see Rubic8's llms.txt and AI SEO guide.

Frequently Asked Questions

Do I need Cloudflare to appear in ChatGPT Search?

No. OpenAI says any public website can appear in ChatGPT Search. The Cloudflare program is a research pilot designed to improve discovery efficiency; it is not an eligibility requirement.

Does allowing OAI-SearchBot guarantee indexing?

No. It makes the content eligible to be crawled for search. OpenAI says placement is not guaranteed and ranking uses multiple relevance and reliability factors.

Can I block training and still allow ChatGPT Search?

Yes. Allow OAI-SearchBot and set a separate preference for GPTBot. Also inspect CDN and WAF rules so a broad AI-bot rule does not override the intended robots policy.

Should I update every page to look fresh?

No. Update pages when the content materially changes. Accurate review dates and sitemap lastmod values are more useful than cosmetic date changes.

Will the pilot reduce crawler load?

That is the design goal: avoid repeatedly fetching unchanged, invalid or duplicate pages. However, Cloudflare notes that discovery of new pages can initially increase crawling while those pages are evaluated.

Final Recommendation

The Cloudflare–OpenAI pilot is important because it moves AI-search discovery closer to real web changes rather than relying only on conventional link discovery and repeated crawling. Its early scale—hundreds of millions of newly discovered pages from a limited cohort—is notable, but it should not be converted into an individual traffic promise.

For publishers, the practical response is straightforward: allow the correct search crawler, return honest status codes, remove duplicate and low-value URL spaces, expose substantive updates clearly, publish information worth citing and measure verified referrals separately from bot activity.

Fix those foundations before looking for an AI-SEO shortcut. Better discovery can open the door; usefulness, relevance and trust determine whether a page deserves to be used.