CEO of NewzDash uncovered a significant development that many publishers, who threaten to restrict Google’s AI, seem unaware of.
Google has begun integrating entire Top Stories carousels within AI Overviews, a maneuver for which the typical robots.txt directive utilized by publishers proves ineffectual.
Shehata noted on LinkedIn that approximately one in six trending U.S. news queries now incorporates Top Stories seamlessly within AI Overviews.
He accompanied this revelation with a detailed data analysis on the NewzDash SEO for News blog. Screenshot from LinkedIn, July 2026
Historical Precedents from Google
The issue at hand is not novel in substance; however, its methodological approach is. In 2006, John and I participated in a panel discussion titled “Vertical Creep Into Regular Search Results” at the Search Engine Strategies New York conference.
We examined how Google was subtly incorporating elements from Google News into its primary search results, a precursor to the formal advent of Universal Search.
Danny Sullivan heralded the May 2007 rollout as the most transformative alteration ever introduced to Google’s results, merging videos, images, books, and news into a cohesive, ranked list rather than presenting ten isolated blue links.
Google’s own announcement mirrored this sentiment, emphasizing the transition to an integrated results methodology rather than isolated categories.
Nineteen years later, Google is revisiting this strategy. This time, the objective is not a composite results page, but rather an AI-generated response.
Insights from NewzDash’s Analysis
Data from NewzDash reveals that, among trending news queries where Google displays Top Stories, 15.5% in the U.S. and 17.46% in the UK now feature the carousel embedded within the AI Overview, rather than as a distinct module beneath it.
Entertainment questions dominate in both regions, achieving over 35% in the U.S. and 31.5% in the UK. World News nearly reaches 32% in the U.S., while Health and Science inquiries remain minimal.
Additionally, Shehata’s findings indicate that these two placements are mutually exclusive—when Top Stories resides within the AI Overview, Google does not activate a separate carousel further down the page for the identical query.
This distinction is more significant than it may initially appear, as it reframes the ongoing discourse publishers have been engaged in since the introduction of AI Overviews.
Misconceptions Surrounding Google-Extended
Many publishers wishing to opt out direct their robots.txt towards Google-Extended and consider their intentions fulfilled.
This oversight is critical. Google’s own crawler documentation explicitly states that Google-Extended pertains to specific AI training and grounding applications, such as preparing future Gemini models or supporting certain Vertex AI reactions, and does not influence a site’s inclusion in Google Search or serve as a ranking factor.
Consequently, blocking it does not exempt a publisher from appearing in AI Overviews, AI Mode, traditional Top Stories, or Top Stories embedded within an AI Overview.
Shehata is astutely emphasizing this point, as the confusion surrounding it is not merely fringe but rather an industry-wide default assumption.
Understanding the Mechanisms Affecting AI Overviews
In June, Google commenced testing a generative AI exclusion feature within Search Console, albeit restricted to a select group of UK site owners.
This enables qualifying publishers to withdraw their links and content from AI Overviews, AI Mode, and generative Discover functionalities without compromising their eligibility for traditional Search outcomes.
Google’s support documentation asserts that content from an excluded site will neither appear in these features nor serve as an input for generating responses.
By opting for this setting, and according to Shehata’s interpretation, publishers almost certainly forfeit access to the embedded Top Stories carousel—since it merely comprises publisher links within one of the covered surfaces.
The speculation surrounding the subsequent layout alterations remains unconfirmed. Will the AI Overview continue to display an embedded carousel composed of links from publishers who opted in?
Will Google revert to a standalone Top Stories module? Presently, no one outside of Mountain View possesses clarity on these questions.
Shehata prudently categorizes this as a high-confidence interpretation rather than a validated outcome.
Such caution is unfortunately rare in current AI SEO discourse, which significantly contributes to the credibility of his insights amid the prevailing cacophony on the topic.
Trust: The Underlying Narrative
The broader narrative here transcends the mechanics of Google-Extended versus the Search Console controls; however, it is imperative that publishers grasp this distinction before making any adjustments.
The crux of the issue lies in trust rather than mere visibility within AI Overviews. Users presently lack a reliable mechanism to discern whether an AI-generated answer is rooted in credible news sources or derived from less substantive materials.
Integrating Top Stories—accompanied by named publishers, authentic bylines, and direct links—within the AI Overview rather than relegating it beneath a flood of generated text constitutes a substantial, albeit incomplete, response to this trust deficiency.
My observations of Google’s transitioning news presentation since the days of “vertical creep” in 2006 suggest that this indicates a novel effort to restore trust rather than merely curtailing traffic to external websites.
Nevertheless, this observation is tempered by a caveat. A move in the right direction is not synonymous with a comprehensive solution, and Google’s reluctance to clarify the implications for publishers who opt out epitomizes the kind of uncertainty that undermines the trust this initiative aims to cultivate.
Three Actionable Steps for the Week
For SEO specialists managing news clients or handling in-house newsroom sites, several crucial actions should be undertaken this week rather than awaiting further clarification from Google regarding layout structures.
First, delineate your controls prior to making any adjustments. Conduct an audit to ascertain whether your site blocks Google-Extended, employs the new Search Console generative AI exclusion, or operates under neither condition.
Document which features each option governs. Misconstruing these controls as interchangeable could inadvertently lead a site to lose visibility within AI Overviews while mistakenly believing it has solely opted out of training data.
Second, if you possess access to the Search Console exclusion, trial it on a URL-prefix property or a specific section before applying it across the entire site.
Google’s control supports inheriting settings between parent and child properties, allowing news publishers to experiment with an exclusion on, say, an opinion section and observe its effects on that segment’s Top Stories eligibility prior to implementing it sitewide.
Third, begin retrieving Google’s generative AI performance reports in Search Console now, and pair them with a quantitative analysis tool like NewzDash, which tracks the frequency at which your URLs emerge within Top Stories carousels versus embedded AI Overview placements.
A well-informed opt-out decision requires a baseline assessment of the visibility at stake, which must be established before modifications are made to settings.
Two decades ago, “vertical creep” compelled publishers to navigate how a blended results page would treat their headlines. Today, the challenge lies in deciphering how an AI-generated answer interprets their content.
Although the mechanism has evolved, the imperative for publishers to fully understand the ramifications of their choices—both in opting in and out—remains unchanged.
John Shehata is diligently documenting this transition in real-time, and until Google communicates otherwise, his insights represent the closest approximation to objective truth within the industry.
Source link: Searchenginejournal.com.
