Allowlisted agents
20
Recent hits (≤1k)
0
Identified agents
0
Unclassified
0
| Agent | Vendor | Category | Recent hits | Ping target |
|---|---|---|---|---|
GPTBot OpenAI's primary training-data crawler for ChatGPT models. docs ↗ | OpenAI | llm | 0 | — |
OAI-SearchBot OpenAI's search index crawler for ChatGPT browsing & citations. docs ↗ | OpenAI | search | 0 | — |
ChatGPT-User On-demand fetcher when a ChatGPT user opens a link. | OpenAI | llm | 0 | — |
ClaudeBot Anthropic's crawler for Claude training & retrieval. docs ↗ | Anthropic | llm | 0 | — |
PerplexityBot Perplexity's answer-engine crawler. docs ↗ | Perplexity | search | 0 | — |
Grok xAI's Grok crawler. | xAI | llm | 0 | — |
xAI-Bot xAI's general-purpose retrieval bot. | xAI | llm | 0 | — |
Google-Extended Google's opt-in AI training fetcher (Gemini, Vertex). docs ↗ | llm | 0 | auto-sync | |
Googlebot Google Search — accepts sitemap ping. | search | 0 | auto-sync | |
Bingbot Bing Search / Copilot — accepts sitemap ping. | Microsoft | search | 0 | auto-sync |
Applebot-Extended Apple Intelligence training-data crawler. | Apple | llm | 0 | — |
Meta-ExternalAgent Meta's Llama training crawler. | Meta | llm | 0 | — |
Bytespider ByteDance multimodal crawler (Doubao). | ByteDance | image-dataset | 0 | — |
ImagesiftBot Image-dataset crawler for reverse-image AI. | ImageSift | image-dataset | 0 | — |
Amazonbot Amazon's Alexa / retrieval crawler. | Amazon | general | 0 | — |
cohere-ai Cohere's training & retrieval crawler. | Cohere | llm | 0 | — |
Diffbot Knowledge-graph extraction crawler. | Diffbot | general | 0 | — |
PanguBot Huawei Pangu multimodal crawler. | Huawei | image-dataset | 0 | — |
Timpibot Timpi decentralized image-dataset crawler. | Timpi | image-dataset | 0 | — |
CCBot Common Crawl — feeds most open LLM training corpora. | Common Crawl | general | 0 | — |
Live counts reflect the most recent 1,000 events from ai_access_events. Operators of additional crawlers can request inclusion via the AI policy contact.