
A common robots.txt edit goes like this: add a Google-Extended block and expect your pages to leave Google’s AI answers. They don’t. Googlebot is the crawler behind Google Search, and AI Overviews and AI Mode are Search features. Google-Extended is a robots.txt token with a narrower job. It controls whether Google may use your content for Gemini training and grounding.
Googlebot is the generic name for Google Search’s two main crawlers. Googlebot Smartphone simulates a mobile user, and Googlebot Desktop simulates a desktop user. Google indexes the mobile version of most sites, so most requests come from the smartphone crawler.
Both answer to the same robots.txt token, Googlebot. Google’s crawler documentation says Googlebot rules affect Google Search, including Discover and all Search features. AI Overviews and AI Mode are among those features.
The desktop user agent looks like this:
Two crawl details matter for AI visibility. Google renders pages with a recent version of Chrome, so it sees content that JavaScript loads. Googlebot also reads only the first 2MB of an HTML file, so main content and markup belong near the top.
Google-Extended is a standalone product token for robots.txt. It has no user agent of its own. Google crawls with its existing user agents, and the token tells Google what it may do with the content afterward.
Google lists two uses it controls:
Grounding means passing live content to a model at answer time, so the reply stays factual and can cite sources. Google states that the token doesn’t affect a site’s inclusion in Google Search. It isn’t a Search ranking signal either.
| Googlebot | Google-Extended | |
|---|---|---|
| What it is | A crawler | A robots.txt token |
| Own user agent in logs | Yes | No |
| What it controls | Crawling for Google Search | Use of content for Gemini training and grounding |
| AI Overviews and AI Mode | Covered, as Search features | No effect |
| Search ranking | Pages must be crawled to rank | Not a ranking signal, per Google |
No. Google’s guide to AI features in Search says a page must be indexed and eligible to show with a snippet. It adds that there are no extra technical requirements. Those are Googlebot and Search rules, so a Google-Extended block changes nothing here.
To limit what Google AI Overviews and AI Mode show from a page, Google points to its snippet controls:
These controls apply to classic results too, so the cost is real. A nosnippet page loses its text preview in regular results as well as in AI features. A noindex removes the page from Search entirely.
Give the token its own group in your robots.txt file:
No crawler stops visiting, because there isn’t a Google-Extended crawler. Googlebot keeps crawling as before. The rule tells Google not to use your content for the Gemini purposes the token covers.
To keep Search and opt out of Gemini use, make the split explicit:
The grounding part is easy to overlook. By Google’s description, a Google-Extended block also opts your content out of grounding in Gemini Apps. When Gemini searches the web for an answer, your pages shouldn’t be used to ground it.
Gemini cites generously, which raises the stakes. In Ansvisor tracking data, September 15 to October 14, 2026, Gemini answers carried about 12.3 links each. Only Google AI Overviews carried more, at about 13.7.
A publisher protecting paid content may accept that trade. A brand that wants Gemini to recommend it usually shouldn’t.
Not every AI token in robots.txt names a crawler. Some name a bot that fetches pages. Others only govern how already-crawled content gets used.
| Token | Owner | Crawls pages? | Controls |
|---|---|---|---|
| Google-Extended | No, Googlebot crawls | Gemini training and grounding | |
| Applebot-Extended | Apple | No, Applebot crawls | Training Apple’s foundation models |
| GPTBot | OpenAI | Yes | Content that may train OpenAI models |
| ClaudeBot | Anthropic | Yes | Content that may train Claude models |
The difference shows up in logs. You can find GPTBot requests by user agent, but you’ll never find a Google-Extended request.
A "Googlebot" user agent proves nothing on its own. Google’s verification guide gives two methods:
There’s no Google-Extended traffic to verify. A request that calls itself Google-Extended didn’t come from Google’s documented crawlers.
Start with the rules that reach Googlebot, including the wildcard group. Make the Google-Extended call on its own merits. Then confirm no stray nosnippet tags sit on pages you want cited. The firewall and log checks are the same ones covered under AI crawler accessibility.
For the Search side, our walkthrough on tracking your website in Google AI Overviews shows what to measure once access is right.
Crawler rules decide eligibility, and Ansvisor shows what happens next. It tracks whether Google AI Overviews, AI Mode and Gemini mention and cite you across your prompts, markets and languages. After a Google-Extended change, the Gemini visibility tracker is where an effect would show, since the token covers Gemini grounding. Site Audit flags robots.txt and AI bot access problems on any page.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
Googlebot is the crawler Google uses for Search and its features, including AI Overviews and AI Mode. Google-Extended is a robots.txt token with no user agent of its own. It controls whether Google may use crawled content for Gemini training and grounding.
No. AI Overviews and AI Mode are Google Search features, and Google says Google-Extended doesn’t affect Search inclusion. To limit what AI features show from a page, use nosnippet, data-nosnippet, max-snippet or noindex. Those also affect regular results.
No. Google says Google-Extended has no separate HTTP user agent string. Google crawls with its existing user agents and reads the token from robots.txt to decide how that content may be used for Gemini.
No. Google states that Google-Extended is not used as a ranking signal in Google Search and does not affect inclusion. It can reduce your presence in Gemini Apps, though, because the token also covers grounding there.
Add a robots.txt group with “User-agent: Google-Extended” and “Disallow: /”, and leave Googlebot allowed. Googlebot keeps crawling for Search, AI Overviews and AI Mode. Google is told not to use the content for Gemini training and grounding.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.