# CrawlRadar

> Know if AI can find and recommend your website: which AI crawlers visit, the traffic ChatGPT, Perplexity, Gemini and Claude send you, and what to fix.

CrawlRadar is AI search analytics for website owners. It has three parts:

- AI crawler tracking: a server-side collector identifies 56+ AI crawlers (OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot and others) and verifies each visit against the vendor's published IP ranges. Analytics tags never see these visits because crawlers do not run JavaScript.
- AI referral traffic: a cookieless tracking tag counts the visitors ChatGPT, Perplexity, Gemini, Claude and Copilot send, and the pages they land on.
- Page readiness: 14 checks score each page 0-100 on whether an answer engine can retrieve, trust and quote it, including robots.txt rules for 14 AI crawler tokens.

## Product

- [CrawlRadar](https://www.crawlradar.xyz/): what the product shows, with the live dashboard for crawlradar.xyz itself.
- [Live demo](https://www.crawlradar.xyz/demo): CrawlRadar's own dashboard, real data: which AI crawlers read crawlradar.xyz and the visits AI answers send.
- [Pricing](https://www.crawlradar.xyz/pricing): Starter $9 a month for 1 website, Growth $19 a month for 10 websites, both priced on human pageviews. AI crawler visits never count. 14-day free trial with no card, 30-day money-back guarantee. Launch offer: code PH20OFF takes 20% off the first 3 months.

## Setup

1. Add your website: enter the domain and sign in with Google or email.
2. Paste two snippets: a tracking tag for AI referral traffic, and a collector for AI crawlers (Cloudflare Worker, Next.js middleware or Node handler, or the WordPress plugin).
3. Watch and fix: AI crawls and AI visits appear as they happen, next to the fixes ranked by impact.

## Tools

- [GEO score checker](https://www.crawlradar.xyz/tools/geo-score-checker): Score any URL 0–100 on AI citation readiness and get the prioritized fixes, in about five seconds.
- [AI crawler checker](https://www.crawlradar.xyz/tools/ai-crawler-checker): Read a site’s robots.txt and show which AI crawlers are allowed or blocked, split into search crawlers that decide citations and training crawlers you can opt out of.
- [llms.txt generator](https://www.crawlradar.xyz/tools/llms-txt-generator): Generate a spec-compliant llms.txt from a site’s sitemap, or validate the one already published against the spec.
- [Schema generator](https://www.crawlradar.xyz/tools/schema-generator): Build valid FAQPage, Article or Organization JSON-LD tuned for citation, entirely in the browser.

## Reference

- [AI crawler directory](https://www.crawlradar.xyz/crawlers): the 14 AI crawler tokens CrawlRadar reads your robots.txt for, which are search crawlers and which only train models, and what blocking each one costs.
- [The 14 checks](https://www.crawlradar.xyz/checks): every check in the scoring model, its weight, and the question it answers.
- [Comparisons](https://www.crawlradar.xyz/vs): how CrawlRadar's AI search analytics differs from Ahrefs, Semrush, Search Console, Surfer and Screaming Frog.

## AI crawlers

- [Is GPTBot blocked on your site?](https://www.crawlradar.xyz/crawlers/gptbot): GPTBot is OpenAI's training token for OpenAI model training. Blocking it keeps your pages out of future OpenAI model training. It does not remove you from ChatGPT Search, which crawls under OAI-SearchBot.
- [Is OAI-SearchBot blocked on your site?](https://www.crawlradar.xyz/crawlers/oai-searchbot): OAI-SearchBot is OpenAI's search index token for ChatGPT Search. This is the crawler that decides whether ChatGPT Search can cite you. Blocking it removes you from that citation pool entirely.
- [Is ChatGPT-User blocked on your site?](https://www.crawlradar.xyz/crawlers/chatgpt-user): ChatGPT-User is OpenAI's live answer fetch token for ChatGPT conversations. It fetches your page while someone is waiting on an answer. Blocking it means ChatGPT may not open your page even when a user asks it to.
- [Is ClaudeBot blocked on your site?](https://www.crawlradar.xyz/crawlers/claudebot): ClaudeBot is Anthropic's training token for Anthropic model training. Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected.
- [Is Claude-SearchBot blocked on your site?](https://www.crawlradar.xyz/crawlers/claude-searchbot): Claude-SearchBot is Anthropic's search index token for Claude’s search answers. This is the crawler that indexes pages for Claude’s search answers. Blocking it can remove you from the results Claude cites.
- [Is Claude-User blocked on your site?](https://www.crawlradar.xyz/crawlers/claude-user): Claude-User is Anthropic's live answer fetch token for Claude conversations. It fetches your page when someone asks Claude a question that needs it. Blocking it means Claude cannot retrieve your page for that person.
- [Is anthropic-ai blocked on your site?](https://www.crawlradar.xyz/crawlers/anthropic-ai): anthropic-ai is Anthropic's training token for Anthropic (legacy). A legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one.
- [Is PerplexityBot blocked on your site?](https://www.crawlradar.xyz/crawlers/perplexitybot): PerplexityBot is Perplexity's search index token for Perplexity search. This is the crawler that builds Perplexity’s index. Blocking it takes you out of the pool its answers are drawn from.
- [Is Perplexity-User blocked on your site?](https://www.crawlradar.xyz/crawlers/perplexity-user): Perplexity-User is Perplexity's live answer fetch token for Perplexity answers. It fetches your page in response to a live question. Blocking it costs you the highest-intent visit a crawler can make.
- [Is Google-Extended blocked on your site?](https://www.crawlradar.xyz/crawlers/google-extended): Google-Extended is Google's policy directive token for Gemini training and grounding. It is a robots.txt directive, not a crawler: no request ever arrives under this name. Disallowing it opts you out of Gemini training and grounding. It does not affect Google Search, including AI Overviews.
- [Is CCBot blocked on your site?](https://www.crawlradar.xyz/crawlers/ccbot): CCBot is Common Crawl's training token for the Common Crawl archive. Common Crawl’s archive feeds many models indirectly. Blocking it is a broad training opt-out with a long tail.
- [Is Bytespider blocked on your site?](https://www.crawlradar.xyz/crawlers/bytespider): Bytespider is ByteDance's training token for ByteDance’s models. A training crawler with a reputation for crawling aggressively. Blocking it is a common choice and costs little in Western answer engines.
- [Is Amazonbot blocked on your site?](https://www.crawlradar.xyz/crawlers/amazonbot): Amazonbot is Amazon's training token for Alexa and Amazon’s AI models. Amazon uses it to improve its products, including Alexa’s answers, and says the content may be used to train its AI models.
- [Is Applebot-Extended blocked on your site?](https://www.crawlradar.xyz/crawlers/applebot-extended): Applebot-Extended is Apple's policy directive token for Apple Intelligence training. Like Google-Extended, this is a directive rather than a crawler: it opts you out of Apple’s model training while leaving plain Applebot, which powers Siri, Spotlight and Safari results, untouched.

## Scoring model

- [What is answer-first writing?](https://www.crawlradar.xyz/checks/answer-first): Answer-first writing means every section answers its own heading in the first sentence, before any preamble — and it is the heaviest single check in this model because an engine lifting a passage reads the top of a section, not the middle of it. Worth 13 of 100 points.
- [What makes a passage extractable by an AI engine?](https://www.crawlradar.xyz/checks/extractability): A passage is extractable when it survives being cut out of the page: it makes sense without the paragraph before it, carries its own subject rather than a pronoun, and fits the chunk size a retrieval system actually stores. Worth 10 of 100 points.
- [Do headings matter for AI search?](https://www.crawlradar.xyz/checks/headings): Yes — headings are the boundaries an engine chunks a page along, so a page with one H1, descriptive H2s and no skipped levels gets split where its author intended rather than mid-argument. Worth 9 of 100 points.
- [Does formatting change whether AI quotes your page?](https://www.crawlradar.xyz/checks/scannability): It does: lists and tables keep their row and column relationships when a page is chunked, and the same content written as flowing prose loses them the moment it is extracted. Worth 9 of 100 points.
- [Is your robots.txt blocking AI crawlers?](https://www.crawlradar.xyz/checks/crawler-access): This check reads your robots.txt the way each agent does — group by group, for the exact path scanned — and reports which of the 14 AI crawler tokens are allowed, because an engine that cannot fetch the page cannot cite it no matter how good the writing is. Worth 8 of 100 points.
- [Why do AI engines prefer pages with numbers in them?](https://www.crawlradar.xyz/checks/specifics): Because a specific is what makes a sentence worth quoting — a date, a figure, a version, a named entity gives an answer something checkable to carry, and this check measures their density per hundred words rather than their raw count. Worth 8 of 100 points.
- [Do AI crawlers run JavaScript?](https://www.crawlradar.xyz/checks/render-dependency): No — the AI crawlers fetch your HTML and read what is in it, so any content that only appears after hydration is invisible to them, and this check compares your raw HTML against your rendered text to find out how much of your page that is. Worth 7 of 100 points.
- [How does an AI engine decide who wrote a page?](https://www.crawlradar.xyz/checks/entity-clarity): From the signals it can machine-read — an author, an organisation, a sameAs pointing at a profile that exists — and a page with none of them is a source an engine has no way to attribute. Worth 7 of 100 points.
- [Does an FAQ section help you get cited?](https://www.crawlradar.xyz/checks/faq): A real question-and-answer section helps, because a question-shaped heading followed immediately by its answer is the exact shape a retrieval system is matching against — and it earns more when the visible questions and the FAQPage markup agree. Worth 7 of 100 points.
- [What stops a page from being indexed at all?](https://www.crawlradar.xyz/checks/indexability): A noindex tag, a canonical pointing somewhere else, a redirect chain or a non-200 status — the mechanical blockers that make every other improvement on the page irrelevant until they are cleared. Worth 6 of 100 points.
- [Does llms.txt actually do anything?](https://www.crawlradar.xyz/checks/llms-txt): Not much yet, and this check says so: it is worth a small share of the score because no answer engine has committed to reading it, and inflating a proposed standard into a ranking factor is how measurement tools lose their credibility. Worth 4 of 100 points.
- [Do AI engines care how old a page is?](https://www.crawlradar.xyz/checks/freshness): They care about whether the page says how old it is: a visible date backed by a matching dateModified is a signal an engine can use, and a page that quietly changes without one gives it nothing to weigh. Worth 4 of 100 points.
- [Why does an unfocused page get cited for nothing?](https://www.crawlradar.xyz/checks/topical-focus): Because retrieval matches a passage to a question, and a page whose title, headings and body pull in different directions has no passage that clearly belongs to any one of them. Worth 4 of 100 points.
- [Which schema types help with AI citation?](https://www.crawlradar.xyz/checks/schema): The ones that state what the page is and who stands behind it — Article, FAQPage, Organization — parsed and validated rather than merely present, because malformed JSON-LD is worth exactly as much as none. Worth 4 of 100 points.

## Comparisons

- [CrawlRadar vs Ahrefs](https://www.crawlradar.xyz/vs/ahrefs): Ahrefs is a backlink, keyword and rank-tracking suite, and tells you who links to you, what you rank for, and how those positions move — but not which AI crawlers visit you, how much traffic AI sends, and whether a page can be quoted.
- [CrawlRadar vs Semrush](https://www.crawlradar.xyz/vs/semrush): Semrush is an all-in-one search marketing suite, and tells you keyword research, competitive share of search, and campaign reporting — but not which AI crawlers fetch your own pages and how many visitors AI assistants send you.
- [CrawlRadar vs Google Search Console](https://www.crawlradar.xyz/vs/google-search-console): Google Search Console is Google's own reporting for its own index, and tells you impressions, clicks and average position for the queries Google shows you — but not what OpenAI, Anthropic and Perplexity crawlers do on your site, and the visits their assistants send.
- [CrawlRadar vs Surfer SEO](https://www.crawlradar.xyz/vs/surfer-seo): Surfer SEO is a content optimisation tool scored against the current search results, and tells you how your draft compares with the pages already ranking for a keyword — but not whether the draft is shaped so a passage can be extracted and attributed, and whether AI then visits it.
- [CrawlRadar vs Screaming Frog](https://www.crawlradar.xyz/vs/screaming-frog): Screaming Frog is a desktop crawler for technical SEO audits, and tells you site-wide technical issues: status codes, redirects, duplicate metadata — but not which real AI crawlers visited, and how a page reads to a system that wants to quote one paragraph of it.

## Guides

- [What is generative engine optimization (GEO)?](https://www.crawlradar.xyz/blog/what-is-generative-engine-optimization): GEO is optimising a page to be quoted inside an AI answer rather than ranked as a link. What changes, what carries over from SEO, and what to do first.
- [How to get cited by ChatGPT](https://www.crawlradar.xyz/blog/how-to-get-cited-by-chatgpt): Four things decide whether ChatGPT can cite your page: OAI-SearchBot access, server-rendered HTML, a clear publisher, and an answer-first passage.
- [Answer-first writing: the highest-leverage GEO fix](https://www.crawlradar.xyz/blog/answer-first-writing): Answer-first writing means each section answers its heading in the first sentence. Worth 13 of 100 points, more than schema, FAQ and llms.txt combined.
- [How to track ChatGPT referral traffic in GA4](https://www.crawlradar.xyz/blog/track-chatgpt-referral-traffic-ga4): ChatGPT visits appear in GA4 with the source chatgpt.com. How to filter for them, give AI assistants their own channel, and what GA4 still cannot see.
- [How to track AI crawlers on your website](https://www.crawlradar.xyz/blog/how-to-track-ai-crawlers): AI crawlers never run your analytics tag. How to find GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in server logs, and verify they are genuine.

## About

- [About CrawlRadar](https://www.crawlradar.xyz/about): built by Iliass El Barhoumi; what the product measures, how the scoring model works, and what it deliberately does not claim.
- [Contact](https://www.crawlradar.xyz/contact): email the founder for support and billing; privacy@crawlradar.xyz for data and account deletion.

## Notes

- Every page linked here is also Markdown: add .md to its URL, or request it with Accept: text/markdown. The home page's is https://www.crawlradar.xyz/index.md.
- The scoring engine is the same code in the Chrome extension and on this site; only the IO adapter differs.
- GPTBot, ClaudeBot and CCBot are training crawlers: blocking them is an opt-out from model training, not from citations. OAI-SearchBot, Claude-SearchBot and PerplexityBot decide citations. Google-Extended does not control Google AI Overviews.
- llms.txt is an emerging convention with no measured effect on citation. CrawlRadar scores it 4 points out of 100 for that reason, and says so on the tool page.