Technical Concepts
Web Crawlers

Web Crawlers

A web crawler is a program that finds pages by following links and sitemaps, downloads them and passes them to a search index. Googlebot and Bingbot are the best-known examples.
June 28, 2026
Cihan Geyik
Table of Content

Before a page can rank in Google or show up in a Copilot answer, a program has to find it and download it. That program is a web crawler. Crawlers start from known URLs, follow links and sitemaps, and hand what they fetch to an index. If no crawler reaches a page, nothing built on that index can use it.

Crawling isn’t indexing. A crawler downloads a page. A separate step decides whether to store it, and Google says not every page it processes gets indexed.

How does a web crawler work?

Start from known URLs → Check robots.txt → Download the page → Extract links → Queue new URLs → Pass content to the index

The queue of URLs waiting to be fetched is called the crawl frontier. Crawlers order it by how important a page seems and how often it changes. They also pace themselves so they don’t overload a server.

Google’s overview of how Search works describes three stages: crawling, indexing and serving results. Not every page makes it through each one. Googlebot also renders pages with a recent version of Chrome, so it sees content that JavaScript adds.

The main web crawlers

CrawlerOperatorWhat it feeds
GooglebotGoogleGoogle Search, including AI Overviews and AI Mode
BingbotMicrosoftBing search and Microsoft Copilot answers
ApplebotAppleSpotlight, Siri and Safari features
CCBotCommon CrawlAn open archive of web data
SEO crawlersYour own teamSite audits, such as broken link checks

Bing’s crawler is Bingbot, and its user agent carries the token bingbot/2.0. Its index matters beyond Bing itself, because Microsoft Copilot grounds its answers in Bing results.

AI companies run crawlers of their own, such as GPTBot and PerplexityBot. They have different jobs and rules, covered on our AI crawlers page.

What is a crawler-based search engine?

A crawler-based search engine builds its index automatically, by crawling, rather than from listings people submit by hand. Google and Bing both work this way. Early web directories took the other route, with human editors sorting sites into categories.

Answer engines continue the pattern. Perplexity runs its own crawler, and ChatGPT search uses OpenAI’s crawler alongside search partners. In every case, a crawler decides which pages exist for the system before any answer gets written.

What decides how a crawler moves through your site?

  • Links. A page with no internal links pointing to it is hard to find, so link new pages from related ones.
  • Sitemaps. A sitemap lists the URLs you want crawled. Crawlers treat it as a hint.
  • robots.txt. It says which paths a crawler may fetch. It doesn’t keep a URL out of the index if other sites link to it.
  • Server health. Google says 5xx and 429 errors make its crawlers slow down for a while.
  • Duplicate URLs. Tracking parameters and session IDs can create thousands of copies that waste crawl visits.

Google’s numbers show how gentle crawling is meant to be. Its Googlebot documentation says the crawler shouldn’t visit most sites more than once every few seconds on average.

Web crawlers vs AI crawlers

Search crawlers build an index for ranking, and Googlebot renders the full page. Many AI crawlers fetch the raw HTML only. A page that looks complete to Googlebot can look empty to them if its main content loads through JavaScript.

The purpose can differ too. Some AI crawlers collect training data, and others fetch a single page because a user asked. For how the end results compare, see our side-by-side of AI search results and traditional Google.

How to test whether a crawler can reach your page

  1. Run the URL Inspection live test in Google Search Console. It shows whether Googlebot can fetch the page and what it renders.
  2. Use the URL Inspection tool in Bing Webmaster Tools to see the page as Bingbot does.
  3. Request the page with a crawler’s user agent and note the status code.
  4. Check your server logs for real crawler visits, and verify the IPs before you trust them.

Step three is a one-line terminal command:

curl -I -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" https://www.example.com/

A borrowed user agent tests your rules, not the crawler’s real path. Firewalls that check IPs treat the real bot differently, so the logs have the final word. Once a page is fetched, web indexing decides whether it’s stored.

How Ansvisor helps

Ansvisor’s Site Audit checks the signals crawlers meet first, including robots.txt, canonical URLs and AI bot access. Answer tracking then shows whether the pages crawlers can reach get cited by ChatGPT, Gemini, Copilot, Perplexity and Google’s AI features.

Web Crawler, Search Engine Crawler, Web Spider, Search Bot, Crawler Bot

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What is a web crawler?

A web crawler is a program that discovers pages by following links and sitemaps, downloads them and sends the content to a search index. Search engines such as Google and Bing run crawlers, and so do AI companies and SEO tools.

How does a web crawler work?

It starts from a list of known URLs, checks each site’s robots.txt, downloads allowed pages and extracts their links. New links join a queue for later visits. The downloaded content then goes to an indexing system, which decides what to store.

What is Bing’s web crawler called?

Bing’s crawler is Bingbot, which identifies itself with the bingbot/2.0 token. It builds the Bing index, which also grounds Microsoft Copilot answers. You can see how Bingbot reads a page with the URL Inspection tool in Bing Webmaster Tools.

What is the difference between crawling and indexing?

Crawling is finding and downloading a page. Indexing is analyzing that page and deciding whether to store it in the search index. Google says indexing isn’t guaranteed, so a crawled page may still be left out.

How do I test if a search engine crawler can access my page?

Use the URL Inspection live test in Google Search Console for Googlebot and the URL Inspection tool in Bing Webmaster Tools for Bingbot. Then confirm in server logs, with verified IPs, that real crawler requests get a 200 response.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.