Cloudflare says its network signals helped OpenAI index hundreds of millions of new pages for ChatGPT search.

Cloudflare and OpenAI say a research pilot has helped OpenAI discover and index hundreds of millions of new web pages. The pilot used a participating cohort of about 1% of Cloudflare’s total network.

The companies shared those early results on Oct. 15 in a post on Cloudflare’s blog coauthored by two OpenAI staff. Cloudflare announced the pilot on July 8.

The post says finding new pages can initially increase crawling, and it doesn’t report how often the new pages were cited in ChatGPT or how many visits they sent.

Cloudflare’s network signals are used by OpenAI to help its search crawler find web pages with value. The blog post gives several examples of these pages, such as the user guide of a particular washing machine, the most recent information regarding passport requirements and fees and a restaurant’s menu and hours. It states that these pages “can be valuable even if few other sites link to them.”

The July 8 announcement stated some of the new signals that will be utilized, including how recently a page has been updated, the quality of traffic to a website and what updates have actually been made to a page. Additionally, the announcement states that this information comes from participating websites. However, neither announcement explains who participated, nor whether the new pages are on their sites.

The Oct. 15 post states that Cloudflare’s signals will also assist in helping OpenAI’s crawlers avoid unnecessary requests.

Source link