Recently, Google unveiled a research paper indicating that frontier large language models (LLMs) successfully encode between 95% and 98% of the factual information tested.
However, these models struggle to directly recall 26% to 34% of this information when posed with queries.
The study identifies that the challenge of recall is compounded when the sequence of subject and object entities is inverted compared to the initial order encountered during training.
Parametric Information
Parametric information refers to the data embedded within LLMs throughout the training process. This data encompasses a vast array of sources, including web pages, song lyrics, literature, directives, programming code, and various other materials utilized during training.
The researchers aimed to explore the underlying reasons for the LLMs’ failure to retrieve certain pieces of information.
Whereas it was previously conjectured that a lack of sufficient training data could be responsible, the findings indicated this is not universally applicable to frontier LLMs.
According to the research, the encoding process has reached saturation. This denotes that the requisite information for responding to inquiries is predominantly present within the LLMs.
The authors articulate:
Encoding is saturated; recall is not. For frontier LLMs like Gemini-3-Pro and GPT-5, factual encoding is nearly saturated, with 95-98% of facts stored.
Nonetheless, these models fail to directly recall 26–34% of the facts, and even 11–12% on instances requiring cognitive engagement.
“Consequently, failures in recall constitute over 70% of errors for GPT-5.2 and a more significant proportion in advanced models, indicating recall truly represents a bottleneck.”
This implies that the limitation does not lie in the quantity of facts encoded, but rather in the retrieval mechanism accessing that embedded information.
Subject and Object Entities
A noteworthy revelation from the research is that difficulties in recalling specific facts stem from the sequential learning of subject and object entities.
When queries reverse the order of these entities, the LLMs exhibit increased difficulty in retrieving the corresponding facts due to the differing sequences in which they were initially processed.
The research paper elucidates the definitions of subject and object entities:
“The roles of subject and object are ascertained by the source text from which the fact was derived (e.g., a Wikipedia entry); the subject is the entity mentioned first, while the object follows.”
This leads to a clarification regarding the reversal of subject and object:
“A query where the answer is the object is termed a direct question, whereas one whose answer is the subject is categorized as a reverse question.”
To demonstrate this concept, Google provides the following example:
“Oasis performed their inaugural gig at the Boardwalk club.”
In this instance, “Oasis” serves as the subject entity, whereas “the Boardwalk club” acts as the object entity.
Thus, when the pair consistently appears with “Oasis” positioned first, the LLM finds it challenging to recall this fact when queried with a reversed subject/object arrangement.
Intriguingly, the model shows aptitude in recognizing the fact when presented with alternative options in a multiple-choice format.
The researchers did not elaborate on why the LLM can identify an answer when framed within a multiple-choice context; however, this suggests that the information is encoded and identifiable within the model.
Phrasing of the Question Had Insignificant Impact on Recall
The researchers investigated if rephrasing questions influenced frontier LLMs’ capacity to recall facts. Their findings revealed that rewording had minimal impact; the primary factor affecting recall was the inversion of the subject/object order.
Challenges with Long-Tail Facts
Another compelling finding indicates that frontier LLMs face significant challenges with what the researchers term “long-tail facts,” or rare information.
Although the disparity between the encoding of common and uncommon facts is marginal, the recall gap is more pronounced.
The inability to retrieve these rarer facts is often not a reflection of inadequate learning but rather a bottleneck encountered during the recall process.
Tested Solution: Enhanced Cognitive Processing
In their examination, the researchers assessed the efficacy of increased cognitive processing for fact retrieval and discovered that LLMs could subsequently recall 40% to 65% of the facts that had previously eluded direct retrieval.
However, this enhanced cognitive engagement is computationally intensive, with an additional concern regarding the optimal timing for invoking such cognitive efforts.
Scaling LLM Training Is Not a Solution
Finally, the researchers concluded that merely scaling frontier LLMs does not rectify the underlying recall issues.
SEO and Subject/Object Entity Pairs
The intuition surrounding the arrangement of subject and object entity pairs posits that aligning them according to their most conventional query order may yield benefits.
While this theory is not substantiated within the research paper, it offers a plausible hypothesis from an SEO perspective regarding potential impacts on LLM functionality.
Source link: Searchenginejournal.com.
