Free tool · runs in your browser · no sign-up
AI crawler checker
Free robots.txt checker for AI crawlers. See whether GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and Applebot-Extended are allowed or blocked, with an explanation of each bot and a copyable fix.
How to use it
- Open https://yourdomain.co.uk/robots.txt and copy everything.
- Paste it in the box and press Check.
- Review each crawler: training bots are optional, retrieval and search bots decide whether AI answers can cite you.
- Copy the suggested groups into robots.txt and re-check.
Questions
Does blocking GPTBot stop ChatGPT mentioning my business?
No. GPTBot gathers training data. ChatGPT's live answers use OAI-SearchBot for search results and ChatGPT-User for fetching pages. Block those and ChatGPT cannot cite you; block GPTBot and future models learn less about you.
Is robots.txt enforced?
It is advisory. Reputable crawlers from OpenAI, Anthropic, Google, Microsoft, Perplexity and Common Crawl honour it. It cannot remove information a model has already learned.
Why is Googlebot in the list?
Google AI Overviews and AI Mode are built from the normal Google index, which Googlebot creates. Google-Extended only controls Gemini training, not Search features.
Want this checked automatically, every week?
Seoptist runs these checks and 120 more across your whole site, asks the AI engines your customers' questions, and gives you one prioritised checklist.
No card needed for the free check. Prices exclude VAT.