
A page can be live, linked and crawlable and still never appear in search. Indexability asks whether a search engine is allowed and able to add a specific URL to its index. One stray noindex tag, a canonical pointing elsewhere or a 404 is enough to keep a page out. It then drops out of every AI answer that draws on that index as well.
Crawlability is whether a crawler can fetch the page. Indexability is whether the search engine may store it once fetched. The two fail in different ways.
robots.txt shows the gap. A disallow stops crawling, yet Google can still index the bare URL if other pages link to it. Search Console reports that as "Indexed, though blocked by robots.txt", usually with a thin snippet. Our web crawlers page covers the fetching side.
| Signal | Indexable | Not indexable |
|---|---|---|
| HTTP status | 200 with real content | 404, 410, soft 404 or persistent 5xx |
| Robots meta tag or header | No noindex | noindex in a meta tag or X-Robots-Tag header |
| Canonical | Points to itself, or none set | Points to another URL, or Google picks another |
| Redirects | The final destination URL | The redirecting URL itself |
| robots.txt | Crawling allowed | Crawling blocked, so a noindex can’t be seen |
Status codes are the bluntest signal. Google removes already indexed URLs that start returning 4xx codes, and it drops URLs that keep returning server errors.
A noindex directive tells search engines to drop a page from results. You can set it in the HTML, or send it as an HTTP header, which also works for PDFs and other files:
One trap catches many sites. Google’s noindex documentation says the page must not be blocked by robots.txt, or the crawler never sees the directive. To remove a page, allow crawling and serve noindex.
When several URLs show the same content, search engines group them and index one version, the canonical. Google ranks the signals you can send. Redirects and rel="canonical" links are strong signals, and listing a URL in a sitemap is a weak one.
They’re signals, so Google can overrule them. The status "Duplicate, Google chose different canonical than user" means it did. That usually points to mixed signals, such as internal links or a sitemap that favor another version.
Google’s AI features sit on top of its index. To appear in AI Overviews or AI Mode, Google says a page must be indexed and eligible to show with a snippet. A noindex page can’t be a source there.
Other engines follow similar logic. Microsoft Copilot grounds its answers in Bing’s index, so Bing indexability matters too. ChatGPT search and Perplexity also run their own crawlers, so status codes and redirects count there as well.
When a brand is missing from AI answers, an indexing gap is one of the first things to rule out. Our post on why your brand doesn’t appear in ChatGPT and Google AI walks through the other causes.
The URL Inspection tool in Google Search Console shows whether a URL is indexed, which canonical Google chose and whether it found a noindex. The Page indexing report groups excluded URLs by reason. Common statuses include:
Not every URL needs to be indexed. Filtered listings and internal search pages are often better left out, as long as the canonical versions of your main pages are in. Once a page passes, web indexing takes over.
A software company relaunches its site. The staging build carried a site-wide noindex, and one template kept it after launch: pricing. Within weeks, the page drops out of Google results. It also stops appearing as a source when AI Overviews answers questions about the product’s cost.
URL Inspection shows "URL marked ‘noindex’". Removing the tag and requesting indexing brings the page back. Answer tracking then confirms when AI Overviews cites it again.
Ansvisor’s Site Audit checks canonical URLs, robots.txt and AI bot access, three signals that often block pages by accident. Citation tracking then shows which of your URLs each AI engine cites, so you notice when a page you care about never appears.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
Indexability is whether a search engine may store a URL in its index once it has fetched it. It depends on the HTTP status code, robots meta tags and headers, canonical signals and redirects. An indexable page can still be left out if the engine judges it low value.
Crawlability is whether a crawler can fetch a page. Indexability is whether the engine may store it. A page blocked in robots.txt isn’t crawlable but can still be indexed from links, while a crawlable page with noindex won’t be indexed.
The usual causes are a noindex meta tag or header, a 4xx status or soft 404, and persistent server errors. Redirects and canonicals that point to another URL also keep a page out. A robots.txt block hides any noindex tag from the crawler.
Use the URL Inspection tool in Google Search Console to see the index status, the canonical Google chose and any noindex it found. The Page indexing report lists excluded URLs by reason. Bing Webmaster Tools offers a similar URL Inspection tool.
Yes. Google says a page must be indexed and eligible for a snippet to appear in AI Overviews or AI Mode. Microsoft Copilot draws on Bing’s index. A page that can’t be indexed can’t be a source for answers built on those indexes.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.