GEOPROBE / Free tool · robots.txt
Is your robots.txt blocking AI crawlers?
Understand what your robots.txt permits. GeoProbe checks rules for selected AI agents and shows the evidence behind each result. Search access and training permissions are different choices; this check does not test CDN or firewall access.
Scans public web content. Reports are public, shareable and may be indexed.
What this tool checks
- GPTBot and OAI-SearchBot (OpenAI training index and ChatGPT search)
- ChatGPT-User (live browsing inside ChatGPT)
- ClaudeBot and Claude-User (Anthropic)
- PerplexityBot (Perplexity search index)
- Google-Extended (Gemini grounding)
- Wildcard rules and longest-match precedence, the way Google's robots spec defines them
Frequently asked questions
Should I allow AI crawlers?
If you want to be recommended by AI assistants, yes — at least the search-oriented agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User). You can still block training-only crawlers like GPTBot if you object to model training; the two are separate user agents.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data for training OpenAI models. OAI-SearchBot builds the index that ChatGPT search uses to answer questions with citations. Blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
My robots.txt allows everything — why does the check still fail?
Some CDNs and bot-protection services (Cloudflare, Vercel firewall, Akamai) block AI user agents at the network layer regardless of robots.txt. This checker reads robots.txt only; if it passes but AI tools still cannot fetch you, look at your WAF settings.
Run it on your site
About 15 seconds. Free, no signup.