AI Missteps Exposed: A Brand That Never Existed Labeled as Trustworthy
During the recent CCTV 3.15 Gala in China, a shocking revelation surfaced: an AI agent endorsed a fictitious brand as “recommendable.”
This unsettling incident has brought to light the troubling phenomenon of “AI poisoning” and highlighted the murky underground industry known as Generative Engine Optimization (GEO).
Many users, who once held AI in reverence, are now grappling with the realization that these systems can be easily manipulated.
In light of this dilemma, a leading name in the search engine realm has opted to intervene. According to recent insights shared by The Verge, Google has amended its spam policy to encompass practices that distort AI model results.
As reported, Google intends to impose punitive actions, including diminished search rankings or outright deletions of manipulated search results.
Interestingly, GEO has emerged as a formidable adversary akin to its predecessor, Search Engine Optimization (SEO), presenting a significant challenge for search engines.
Notably, GEO began to gain traction as early as the fall of 2024, quickly regarded as a promising frontier by global digital marketers.
The pressing question remains: why has Google only now chosen to address it? The answer lies in the fact that GEO’s methodologies have evolved to an insufferable extent.
Rooted in the AI-driven marketing landscape, GEO seeks to deceive AI into featuring a client’s product within the outputs generated by models such as ChatGPT, Gemini, Doubao, and DeepSeek.
For instance, when prompted with “Which brand of makeup remover is recommended?” GEO’s objective is to have the AI reference the client’s offering.
The operational basis of GEO is predicated on a phenomenon referred to as “authority worship.” In an era where society largely perceives AI as the vanguard of the “Fourth Industrial Revolution,” these systems have become coveted repositories of knowledge.
Expressions like “ChatGPT asserts” or “DeepSeek elucidates” have infiltrated everyday dialogue. Recognizing this societal trust, GEO service providers exploit the naivety of users by covertly embedding advertisements into AI-generated content.
This subterfuge is particularly insidious, as such advertisements typically surface only in response to user inquiries.
The potency of GEO’s influence over substantial AI models can be distilled into a straightforward concept. AIGC (AI-Generated Content) fundamentally relies on probability-based models.
Instead of conjuring information from thin air, it aligns data with the most pertinent information available in its repositories.
By strategically adjusting content structure and semantic alignments, it becomes feasible to elevate the prominence of certain brands within AI-generated responses.
Addressing the challenge of data poisoning, merely disabling online search capabilities will not suffice. The training datasets utilized by influential AI models can still fall prey to manipulation.
Until recently, optimism prevailed in the AI community that larger model architectures would inherently resist such attacks.
This notion was based on the assumption that as model parameters expanded from terabytes to petabytes, the limited impact of “poison” would be overshadowed by the sheer volume of accurate data.
However, a study released by Anthropic has cast doubt on this premise. It demonstrated that the quantity of “poison” required to compromise data integrity is largely independent of the training dataset’s scale.
Remarkably, researchers were able to debilitate a model with 13 billion parameters using just 250 malicious documents, significantly diminishing the model’s output accuracy.
The old adage rings true: using falsehoods to validate other falsehoods will yield only more falsehoods. Consequently, flawed parameters within the dataset lead to erroneous outcomes.
Given the intrinsic nature of AI models, they are not immune to the effects of external forces; true neutrality remains an elusive ideal.
GEO service providers counteract this inherent vulnerability by deliberately crafting content that aligns with the inclinations of major AI models.
Initially, GEO’s strategy involved self-promotion. To illustrate, in our makeup remover example, a GEO provider might disseminate fabricated claims such as, “Our product, tested by the XXXX Association, resulted in 97.3% of users showing improved skin conditions in just 28 days.”
None of these assertions would hold factual merit; yet, AI systems do not authenticate claims, but rather propagate them. If the same text proliferates online, AI is likely to regard it as truth.
As tactical approaches evolve, so too does AI capability. Fact-checking functionalities are now staple features within AI assistants such as Doubao and DeepSeek.
Facing filtration of their self-aggrandizing claims, GEO practitioners have pivoted towards creating a façade of legitimacy. Recently, an investigative report unveiled a burgeoning industry dedicated to fabricating “pseudo-media systems.”
With a nominal investment of US$10 for a domain name, AI can orchestrate the creation of a “local news website” within a mere quarter-hour, complete with launch proclamations, editorial teams, and bona fide-looking journalistic content.
Investigations reveal that these websites are often run by entities assisting clients in crafting favorable online personas while obscuring unwanted search results.
These digital marketing professionals have astutely noted a paradigm shift in search engines’ content preferences.
In the age of AI-driven searches, platforms like Baidu and Google are now more concerned with the authenticity of expertise rather than sheer content volume.
The outdated model of “content saturation + keyword manipulation + external linking” has become obsolete. Ironically, an abundance of content with scattered focus may lead to decreased trustworthiness in the eyes of search engines.
How, then, does one cultivate an image of expertise before search engines? Some GEO specialists have unearthed a “meta solution”: masquerading as news media.
The legacy of news organizations, lauded as the “fourth estate,” imbues them with a degree of credibility that persists despite the encroachment of self-publishing on the internet. While traditional media authority has waned, it retains an aura of authenticity in the eyes of search engines.
Crucially, GEO agents do not fabricate established media outlets but rather focus on assembling lesser-known “local media networks.”
These entities occupy a nuanced position in the digital landscape, possessing a residue of credibility while being compelled to engage in “borderline news” that is often lackluster.
However, search engines still necessitate news media to uphold content integrity. When confronted with seemingly legitimate “local media,” AI systems can become disoriented.
Faced with this conundrum, Google has been compelled to enact decisive measures to thwart such manipulations before they escalate.
Source link: English.dotdotnews.com.
