AI & Infrastructure
Googlebot / Google-Extended

Googlebot / Google-Extended

Googlebot is Google’s crawler for Search, including AI Overviews and AI Mode. Google-Extended is a robots.txt token, not a crawler, that controls whether Google may use your content for Gemini training and grounding.
October 5, 2026
Cihan Geyik
Table of Content

A common robots.txt edit goes like this: add a Google-Extended block and expect your pages to leave Google’s AI answers. They don’t. Googlebot is the crawler behind Google Search, and AI Overviews and AI Mode are Search features. Google-Extended is a robots.txt token with a narrower job. It controls whether Google may use your content for Gemini training and grounding.

Google-Extended doesn’t touch AI Overviews or AI Mode. Both are part of Google Search, which Googlebot crawls. Google says the token has no effect on Search inclusion or ranking.

What is Googlebot?

Googlebot is the generic name for Google Search’s two main crawlers. Googlebot Smartphone simulates a mobile user, and Googlebot Desktop simulates a desktop user. Google indexes the mobile version of most sites, so most requests come from the smartphone crawler.

Both answer to the same robots.txt token, Googlebot. Google’s crawler documentation says Googlebot rules affect Google Search, including Discover and all Search features. AI Overviews and AI Mode are among those features.

The desktop user agent looks like this:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Two crawl details matter for AI visibility. Google renders pages with a recent version of Chrome, so it sees content that JavaScript loads. Googlebot also reads only the first 2MB of an HTML file, so main content and markup belong near the top.

What is Google-Extended?

Google-Extended is a standalone product token for robots.txt. It has no user agent of its own. Google crawls with its existing user agents, and the token tells Google what it may do with the content afterward.

Google lists two uses it controls:

  • Training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini.
  • Grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

Grounding means passing live content to a model at answer time, so the reply stays factual and can cite sources. Google states that the token doesn’t affect a site’s inclusion in Google Search. It isn’t a Search ranking signal either.

Googlebot vs Google-Extended

GooglebotGoogle-Extended
What it isA crawlerA robots.txt token
Own user agent in logsYesNo
What it controlsCrawling for Google SearchUse of content for Gemini training and grounding
AI Overviews and AI ModeCovered, as Search featuresNo effect
Search rankingPages must be crawled to rankNot a ranking signal, per Google

Does Google-Extended control AI Overviews and AI Mode?

No. Google’s guide to AI features in Search says a page must be indexed and eligible to show with a snippet. It adds that there are no extra technical requirements. Those are Googlebot and Search rules, so a Google-Extended block changes nothing here.

To limit what Google AI Overviews and AI Mode show from a page, Google points to its snippet controls:

<meta name="robots" content="nosnippet">
<meta name="robots" content="max-snippet:50">
<div data-nosnippet>Text to keep out of snippets</div>

These controls apply to classic results too, so the cost is real. A nosnippet page loses its text preview in regular results as well as in AI features. A noindex removes the page from Search entirely.

How to block Google-Extended in robots.txt

Give the token its own group in your robots.txt file:

User-agent: Google-Extended
Disallow: /

No crawler stops visiting, because there isn’t a Google-Extended crawler. Googlebot keeps crawling as before. The rule tells Google not to use your content for the Gemini purposes the token covers.

To keep Search and opt out of Gemini use, make the split explicit:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /
Never block Googlebot to stop AI use. A Googlebot disallow stops crawling for all of Search, including regular results, AI Overviews and AI Mode.

What a Google-Extended block costs in Gemini

The grounding part is easy to overlook. By Google’s description, a Google-Extended block also opts your content out of grounding in Gemini Apps. When Gemini searches the web for an answer, your pages shouldn’t be used to ground it.

Gemini cites generously, which raises the stakes. In Ansvisor tracking data, September 15 to October 14, 2026, Gemini answers carried about 12.3 links each. Only Google AI Overviews carried more, at about 13.7.

A publisher protecting paid content may accept that trade. A brand that wants Gemini to recommend it usually shouldn’t.

How does Google-Extended compare with other AI controls?

Not every AI token in robots.txt names a crawler. Some name a bot that fetches pages. Others only govern how already-crawled content gets used.

TokenOwnerCrawls pages?Controls
Google-ExtendedGoogleNo, Googlebot crawlsGemini training and grounding
Applebot-ExtendedAppleNo, Applebot crawlsTraining Apple’s foundation models
GPTBotOpenAIYesContent that may train OpenAI models
ClaudeBotAnthropicYesContent that may train Claude models

The difference shows up in logs. You can find GPTBot requests by user agent, but you’ll never find a Google-Extended request.

How to verify Googlebot

A "Googlebot" user agent proves nothing on its own. Google’s verification guide gives two methods:

  1. Run a reverse DNS lookup on the IP and check that the host ends in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that host and confirm it returns the same IP.
  2. Match the IP against the ranges Google publishes in files such as common-crawlers.json.
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

There’s no Google-Extended traffic to verify. A request that calls itself Google-Extended didn’t come from Google’s documented crawlers.

Common mistakes

  • Blocking Google-Extended to leave AI Overviews. It has no effect there.
  • Blocking Googlebot to stop Gemini training, which removes pages from Search as well.
  • Searching logs for a Google-Extended user agent that doesn’t exist.
  • Forgetting that the token covers grounding in Gemini Apps as well as training.
  • Using noindex when you only meant to limit snippets. The nosnippet and max-snippet controls are narrower.
  • Trusting a Googlebot user agent without a DNS or IP check.

How to audit your Google crawler setup

Check the Googlebot group → Decide on Google-Extended → Review snippet controls → Verify Googlebot in logs → Track AI Overviews, AI Mode and Gemini

Start with the rules that reach Googlebot, including the wildcard group. Make the Google-Extended call on its own merits. Then confirm no stray nosnippet tags sit on pages you want cited. The firewall and log checks are the same ones covered under AI crawler accessibility.

For the Search side, our walkthrough on tracking your website in Google AI Overviews shows what to measure once access is right.

How Ansvisor helps

Crawler rules decide eligibility, and Ansvisor shows what happens next. It tracks whether Google AI Overviews, AI Mode and Gemini mention and cite you across your prompts, markets and languages. After a Google-Extended change, the Gemini visibility tracker is where an effect would show, since the token covers Gemini grounding. Site Audit flags robots.txt and AI bot access problems on any page.

Googlebot, Google-Extended, Google Extended, Google Crawler, Googlebot User Agent, Google-Extended robots.txt

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What is the difference between Googlebot and Google-Extended?

Googlebot is the crawler Google uses for Search and its features, including AI Overviews and AI Mode. Google-Extended is a robots.txt token with no user agent of its own. It controls whether Google may use crawled content for Gemini training and grounding.

Does blocking Google-Extended remove my site from AI Overviews?

No. AI Overviews and AI Mode are Google Search features, and Google says Google-Extended doesn’t affect Search inclusion. To limit what AI features show from a page, use nosnippet, data-nosnippet, max-snippet or noindex. Those also affect regular results.

Is Google-Extended a crawler?

No. Google says Google-Extended has no separate HTTP user agent string. Google crawls with its existing user agents and reads the token from robots.txt to decide how that content may be used for Gemini.

Does blocking Google-Extended hurt Google rankings?

No. Google states that Google-Extended is not used as a ranking signal in Google Search and does not affect inclusion. It can reduce your presence in Gemini Apps, though, because the token also covers grounding there.

How do I block Google-Extended but keep Google Search?

Add a robots.txt group with “User-agent: Google-Extended” and “Disallow: /”, and leave Googlebot allowed. Googlebot keeps crawling for Search, AI Overviews and AI Mode. Google is told not to use the content for Gemini training and grounding.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.