AI Robots.txt Checker
Check which AI crawlers can access your website. Make sure you're not accidentally blocking AI training bots that could help your visibility.
Known AI Crawlers
Understanding AI Crawlers
AI companies use web crawlers to access and index content from websites. This content may be used to train AI models or to provide real-time information to users. Your robots.txt file controls which crawlers can access your site.
Key AI Crawlers
GPTBot (OpenAI)
Used by OpenAI for training data and ChatGPT browsing.
Google-Extended
Google's AI training crawler, separate from Googlebot.
Anthropic-AI
Anthropic's crawler for Claude training data.
CCBot
Common Crawl's bot, used by many AI training datasets.
Should you block AI crawlers?
It depends on your goals. If you want AI to mention and recommend your brand, allowing AI crawlers to access your content is important. Blocking them may reduce your visibility in AI responses.
Some organizations choose to block AI training crawlers due to copyright or competitive concerns. The choice is yours, but understand the trade-offs.
How to allow AI crawlers
To ensure AI crawlers can access your site, your robots.txt should either:
- • Not exist (default allows all crawlers)
- • Not specifically block AI user agents
- • Explicitly allow AI crawlers if you have a restrictive policy
Allowing crawlers is just the start
Track whether AI actually mentions your brand with Prompt Eden.