Can AI crawlers see your site?
Plenty of sites block AI crawlers without knowing it. Enter your address and this reads your live robots.txt.
No email, no account. This reads one public file and nothing else.
How do I check whether AI crawlers can read my website?
Open yourdomain.com/robots.txt and look for a Disallow rule covering GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot. A named group replaces the wildcard group entirely, so one rule can block a crawler however permissive the rest of the file is. The checker above reads the live file and reports each one.
The ten-point visibility check·Answer engine optimization·Pricing
Four things a green tick here does not mean.
robots.txt is a request
The major AI crawlers say they honour it. "Allowed" means permission, and says nothing about whether a fetch happened.
Your WAF may block them anyway
Cloudflare, a host firewall or a security plugin can refuse AI crawlers at the edge. Check the bot-management rules too.
Only the site root is tested
A file can allow "/" and still block /blog or /services. Per-page meta robots tags are not read here.
Being readable is not being cited
A crawler still has to find something worth quoting. That is the other nine points of the visibility check.
Questions about AI crawlers and robots.txt
Should I allow GPTBot and the other AI crawlers?
What is the difference between GPTBot and OAI-SearchBot?
Why is my site blocking AI crawlers when I never set that up?
Does blocking AI crawlers affect my Google rankings?
Does askotter set this up for clients?
Open to crawlers, invisible in answers?
That is the usual result. The free audit runs your customers' prompts and shows you what came back.
Self-managed is $99/mo and scales with the compute you use. Our team runs it from $600/mo.