The 2026 AI Bot Directory & User Agents
A complete list of User Agents for ChatGPT, Perplexity, Claude, and Gemini. Copy-paste configuration for your robots.txt.
To rank in AI search, you must allow their crawlers to access your content. Many legacy SEO setups accidentally block these bots.
Here is the definitive technical list of User Agents you need to know.
The "Must-Allow" List
To optimize your visibility in the major engines, ensure these are allowed in your robots.txt.
| Engine | User-Agent | Purpose |
|---|---|---|
| ChatGPT | GPTBot | Used to train models and browse the web for live answers. |
| ChatGPT | OAI-SearchBot | Specifically for "SearchGPT" real-time results. Critical for traffic. |
| Perplexity | PerplexityBot | Powers Perplexity's real-time citation engine. |
| Claude | ClaudeBot | Anthropic's crawler for training and retrieval. |
Google-Extended | Controls whether your data works with Gemini/Vertex AI. | |
| Apple | Applebot-Extended | Used for Apple Intelligence features. |
Free AI Visibility Checker
Want to know if you are blocking traffic right now? Use our free tool to scan your robots.txt against these bots instantly.
Recommended robots.txt Configuration
If you want to maximize your Share of Voice in AI, add this to your robots.txt file:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
"Data Magnet" Strategy
Don't just open the door—invite them in. AI bots prioritize pages that are:
- Fast: < 500ms TTFB.
- Clean: JSON-LD schema present (See our GEO Score guide).
- text/markdown: AI parses markdown faster than HTML.
Frequently Asked Questions
Should I block GPTBot?
Only if you have private user data (PII) you don't want in their training set. For marketing/public content, blocking it means voluntary invisibility in the world's biggest AI tool.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot: Crawls for general model training (long-term memory).OAI-SearchBot: Crawls for live search queries (short-term answers). If you block this, you vanish from SearchGPT results.