For several years, the prevailing notion among SEO professionals was that ChatGPT operated through a straightforward process: it relayed a query to Bing, obtained results, and then created a response based on that data.
However, a recent report from Peec AI, a company specializing in both geographical and AI search analytics, has dispelled this misconception.
It has been revealed that ChatGPT has quietly established its own internal retrieval index, dubbed Labrador, a development that has been ongoing for several years.
This groundbreaking research, conducted by Tomek Rudzki of Peec AI, elucidates the mechanisms by which ChatGPT acquires information.
The findings are derived from a confluence of leaked technical data, OpenAI’s job postings, and sworn testimonies from legal proceedings. The resulting narrative is significantly more intricate and ambitious than previously anticipated.
Labrador: A Diverse Family of Indices
Between May 21 and July 21, 2026, server-side events from ChatGPT included a ‘result_source’ field that explicitly identified the source of information utilized in generating responses.
This field offered just four possible values: Labrador, Bright, Oxylabs, and SERP. While Labrador is the proprietary index of OpenAI, the remaining three are third-party scraping providers.
Research indicates that Labrador is not merely a single type of index; it comprises various categories, mirroring how Google has structured its search framework over the past two decades with a primary web index augmented by specialized verticals.
The categories identified include general web pages, PDFs, YouTube, news (differentiated by recency), arXiv, Wikipedia, local finance (complete with a PDF-specific index), legal, healthcare, shopping, and images.
Each category employs similar indexing data comprising page text, web crawl dates, and publication timelines.
Job Postings Reflect Long-Term Commitment
Peec AI has also referenced numerous OpenAI job listings as corroborative evidence for its research findings.
A position for a Software Engineer specializing in Foundations Search indicates that selected candidates would be assessed on their experience with “indexing systems, retrieval pipelines, and serving layers.”
Another listing for an Engineering Manager within OpenAI’s Online Data Systems team outlines responsibilities involving the construction and maintenance of databases and indexing services that underpin OpenAI applications, such as ChatGPT, at what the advertisement describes as an exabyte scale.
An exabyte equates to approximately one billion gigabytes, suggesting that such infrastructure cannot be supported by a company solely reliant on external indices.
Additionally, another job posting emphasizing embedding retrieval requires candidates to have a solid background in formulating new embedding training objectives, architecting vector stores, and developing dynamic indexing methodologies, alongside expertise in sparse and hybrid representation techniques.
Sparse representation, as utilized by early search engines like Google, employs lexical ranking methods such as TF-IDF and BM25.
When combined with dense vector embeddings, this creates a method known as Reciprocal Rank Fusion, which amalgamates these two distinct result types, a concept previously noted by Peec AI researcher Metehan Yesilyurt regarding ChatGPT’s operational mechanisms.
Connections to the 2024 Antitrust Hearing
Peec AI connects these developments to the antitrust proceedings involving Google, during which Nick Turley, head of ChatGPT, provided sworn testimony spotlighting OpenAI’s search strategy as a crucial topic.
Turley reportedly indicated that OpenAI encountered significant quality concerns using search data from non-Google API partners and expressed a desire to partner directly with Google, which ultimately rejected the proposal.
Evidence presented during the court hearings suggests that OpenAI’s transition to its own indexing framework was partially prompted by this refusal.
Turley asserted that OpenAI commenced its independent index development in 2023, initially aiming to address 80% of customer queries through this index by the end of the year.
He also noted that, even with Google’s comprehensive index data, assessing the feasibility of solely using their own index for all queries would be a five-year endeavor.
Thus, achieving complete independence concerning data sources is a protracted and methodical process, entailing thorough testing across various query categories one at a time.
Product Search: The Most Visible Implementation
Peec AI observes that the most conspicuous live application of ChatGPT’s indexing system emerges in the realm of product search results.
In mid-August 2026, an A/B test labeled “prefer-index-over-serp-v3” demonstrated operation in roughly 8% of conversational interactions.
As of September 2, 2026, at least five distinct shopping experiments were ongoing, each assessing various combinations of OpenAI’s internal data sources against external search data.
A detailed study accessed by experts follows a triad of major steps: an initial lexicon search employing the BM25 algorithm that narrows candidates to ten sources; a ranking phase encompassing 400 potential products; and an approximate nearest neighbor vector search executed across two dimensions.
This multi-stage ranking methodology applied to live shopping data implies that the OpenAI team actively engages in validation processes to ensure the index’s readiness for trial production.
ChatGPT’s Caching Capabilities
Live retrieval necessitates that the system accesses the most current version of a website whenever requested.
Given the scale at which ChatGPT operates, obtaining every webpage live on demand is virtually unattainable, as many pages require over a second to load.
Peec AI posits a convincing argument suggesting that ChatGPT maintains a cache of frequently visited webpages, a concept paralleling strategies proposed in Google’s own patents.
Researchers from Peec AI published evidence via a test website encompassing one billion pages to analyze ChatGPT’s web crawling behavior.
By September 2026, approximately 6 million pages had been crawled, representing a modest fraction of the total, yet the spider operates at a remarkable speed of 35,000 pages per hour.
Furthermore, during testing of the “lockdown mode” setting— which restricts live surfing while allowing access to cached or offline results—researchers discovered that ChatGPT could present seemingly current homepage content from leading SEO publishers even in the lockdown state, indicating this information likely stemmed from a cached source rather than newly downloaded content.
The Ongoing Role of Google and Microsoft
Despite its capacity to create an independent index, ChatGPT has not completely severed ties with external search engines.
For instance, a Peec AI experiment revealed that when a website lacking organic traffic was interrogated through ChatGPT over successive days, an observable uptick in the site’s traffic was noted via Google Search Console, suggesting ChatGPT queries Google directly to bolster its responses.
Simultaneously, Bing remains a notable presence, particularly during Deep Research sessions, where it is explicitly acknowledged as a data source in ChatGPT’s server-side events.
Microsoft has also introduced its Web IQ grounding platform, which purportedly offers sub-165 millisecond latency, reportedly powering ChatGPT products such as Microsoft Copilot and other enterprise-level customers, including Nasdaq.
Peec AI asserts that at least eight supplementary data providers, alongside ChatGPT’s crawler, supply content to its results.
These include Bright Data, Oxylabs, a dedicated SERP-scraping channel, Yelp, TripAdvisor, two internal data conduits for business listings, Google Maps data, and Microsoft’s Web IQ.
Implications for Website Owners
The crux of this study underscores the unreliability of employing Bing’s visibility as a proxy for ChatGPT’s visibility.
ChatGPT is now developing its crawler and index while also leveraging signals from various external sources; it subsequently tests different combinations with live traffic and incrementally adjusts towards its proprietary systems.
For online retailers, where consumers can compare prices, the consequences of this transition are particularly pronounced, as OpenAI engages in the most rigorous experimentation in this arena.
Conversely, businesses reliant on local visibility are directly impacted by their listings on review platforms like Yelp and TripAdvisor, since these reviews are treated as data licenses, while Google Maps data is incorporated into ChatGPT indiscriminately.
Furthermore, assessing a website’s representation within ChatGPT’s lockdown mode may provide initial insights into its visibility.
Nevertheless, developments are progressing rapidly. Peec AI has noted ongoing adjustments to various aspects of this system on a weekly basis, rendering some details potentially outdated by the time of reading.
Q 1: What is Labrador?
Labrador is the codename for ChatGPT’s proprietary search index, crafted by OpenAI to enhance its search functionalities and reduce reliance on Google and Bing.
Q 2: Does ChatGPT depend on Google and Bing?
Yes, evidence from Peec AI’s investigation indicates that ChatGPT directly accesses Google and Bing, with the latter utilized intensively during Deep Research sessions.
Q 3: What types of data does Labrador index?
Labrador encompasses various data categories, including general web pages, PDFs, YouTube videos, news articles, arXiv papers, Wikipedia entries, local listings, financial data, legal documents, healthcare information, shopping data, and images.
Q 4: When did OpenAI initiate the development of its own index?
Testimony from Nick Turley, head of ChatGPT, confirms that OpenAI embarked on building its independent index in 2023.
Q 5: Why did OpenAI undertake the construction of its own search index?
Frustrated by the subpar quality of search results from third-party platforms, OpenAI aimed to establish a direct collaboration with Google, which was ultimately declined.
Q 6: Where is ChatGPT’s self-hosted index observable?
It is most evident in shopping results, where multiple A/B tests have been executed to evaluate the effectiveness of OpenAI-generated results compared to external indexes.
Q 7: Does ChatGPT cache web pages instead of fetching them live?
Research findings suggest that ChatGPT likely caches web pages, as indicated by the observed crawling activity on test websites and results retrieved during lockdown mode.
Q 8: What other data providers does ChatGPT utilize?
In addition to Google and Bing, ChatGPT employs content from at least eight different data suppliers, including Bright Data, Oxylabs, Yelp, TripAdvisor, and Microsoft’s Web IQ.
Source link: Almcorp.com.
