Before an AI answer can cite a page, a crawler has to fetch it and make sense of it. Screaming Frog SEO Spider is a desktop crawler that checks exactly that, page by page, across a whole site. It doesn’t track AI answers. What it does well is show whether AI crawlers can reach, render and parse what you publish.
A clean crawl removes blockers. It can’t show whether AI answers mention or cite you. That takes answer-level tracking.
What is Screaming Frog SEO Spider?
The SEO Spider is a website crawler made by Screaming Frog, a UK search agency. You install it on Windows, macOS or Linux, enter a URL and it crawls the site the way a search bot would.
It collects status codes, titles, meta descriptions, canonicals, robots directives, hreflang, links and structured data. Problems land in issue reports. "SEO spider software" is the generic name for this kind of tool, and Screaming Frog’s SEO Spider is a widely used one.
What does the free version include?
The free version crawls up to 500 URLs. A paid licence lasts one year, is sold per user and removes the limit. Bulk discounts start at five licences.
| Feature | Free | Paid licence |
|---|
| Crawl limit | 500 URLs | Unlimited, within memory and storage |
| Broken links, titles, directives, hreflang, XML sitemaps | Yes | Yes |
| JavaScript rendering | No | Yes |
| Structured data validation | No | Yes |
| Custom robots.txt, scheduling, crawl comparison | No | Yes |
| AI provider integrations | No | Yes |
| Search Console and Google Analytics integration | No | Yes |
How does Screaming Frog help with AI search visibility?
Many site-side AI search problems are crawl problems in disguise. The SEO Spider finds them at scale.
- Crawl as an AI bot. The user-agent setting lets you crawl as Googlebot, Bingbot, AI crawlers or a custom agent.
- Test robots.txt rules. You can respect or ignore the live file, or test a custom version before you publish it.
- Compare raw and rendered HTML. If the main text appears only after JavaScript runs, crawlers that don’t render JavaScript miss it.
- Validate structured data and flag errors in schema markup.
- Find indexability problems: noindex tags, canonicals that point elsewhere, redirect chains and orphan pages.
What AI features does Screaming Frog have?
Recent versions bring AI into the crawl itself. Version 21.0, released in October 2024, added direct API connections to OpenAI, Gemini and Ollama. Anthropic is now supported as well.
You write prompts that run on every crawled page, for example to draft meta descriptions or label page intent. You pay the AI provider for API use, while Ollama runs models on your own machine.
Version 22.0, from February 2025, added semantic similarity analysis with vector embeddings, a semantic content cluster diagram and semantic search. It flags near-identical pages and off-topic content, which helps before you add more pages on the same subject.
Version 24.0, from February 2026, added an MCP server. Screaming Frog says assistants such as Claude or LM Studio can now run crawls and analyze or export data in plain language. Not every feature works through MCP yet.
One caution: when a prompt runs through OpenAI, Gemini or Anthropic, the crawled page content goes to that provider. For sites behind a login or under strict data rules, a local Ollama model keeps the text on your machine.
How do Search Console and GA4 data help a crawl?
The paid version can pull Search Console and Google Analytics data into the crawl through their APIs. Each URL then shows clicks, impressions and sessions next to its technical issues.
That changes priorities. A noindex tag on a page with no traffic is a cleanup task. The same tag on a page that earns clicks, or that AI answers used to cite, is urgent. Search Console and analytics data also reveal orphan pages, URLs that get visits but have no internal links pointing to them.
The URL Inspection API integration goes one step further and reports Google’s own index status for each URL, within Google’s daily quota.
How to audit AI crawler access with Screaming Frog
Pick core pages → Crawl as each AI bot → Compare status codes → Check rendered text → Fix blocks and recrawl
- List the pages you want cited: product, pricing, comparison pages and top guides.
- Crawl them in list mode as Googlebot, then again as an AI crawler such as GPTBot or OAI-SearchBot.
- Compare responses. A 403 or a challenge page for the AI bot but not for Googlebot usually points to a firewall or CDN rule.
- Turn on JavaScript rendering and compare word counts between raw and rendered HTML.
- Fix the cause in robots.txt, your CDN settings or the page template, then crawl again.
A robots.txt file can treat search and training crawlers differently. This example keeps ChatGPT search open while limiting OpenAI’s training crawler:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /private/
Our guide on checking whether AI crawlers can access your website covers more bots and edge cases.
Screaming Frog vs an AEO audit tool: do you need both?
Usually, yes. They answer different questions.
| Question | Screaming Frog | AEO audit and tracking tool |
|---|
| Can crawlers reach every page? | Yes, site-wide | Often page by page |
| Is the schema valid? | Yes | Usually |
| Is a page written to be quoted? | Partly, through custom AI prompts | Yes, scored against AEO signals |
| Do AI answers mention or cite us? | No | Yes |
| Which competitors and sources win? | No | Yes |
The scale explains why. In one month, AI engines cited 122,066 distinct URLs across the 103,427 answers we tracked. A crawl can confirm your pages are fit to join that list. Only answer tracking shows whether they did. Source: Ansvisor tracking data, September 15 to October 14, 2026.
For a page-level checklist, see our AI visibility audit checklist.
What are Screaming Frog’s limits for AI search?
- It’s a desktop app, so large crawls depend on your machine’s memory and storage.
- Each crawl is a snapshot. Scheduling and automatic crawl comparison help, but you have to set them up.
- AI prompt features add API costs on top of the licence.
- It sees your site only, never the AI answers.
How Ansvisor helps
Screaming Frog checks the technical health of the whole site. Ansvisor’s Site Audit scores individual pages against 47 weighted AEO and GEO signals. They cover structure, content, authority, E-E-A-T and trust, including AI bot access and llms.txt, and each issue comes with a suggested fix.
Answer Engine Insights and Citations Monitoring then show whether those pages are mentioned and cited in AI answers. Ansvisor tracks ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Copilot and Perplexity, with Claude on Enterprise plans. Use the crawl for site health and Ansvisor for what the answers say. Start a 14-day free trial.
