AI Crawler Access Analyzer
See which AI crawlers your robots.txt allows — and whether you have an llms.txt.
Free. No account needed. We analyze the page you enter and show only what we actually measured.
Why AI crawler access matters
To appear in AI answers — ChatGPT, Claude, Perplexity, Google's AI Overviews — those engines' crawlers first have to be allowed to read your site. This tool fetches your robots.txt and checks, for each major AI crawler, whether it's allowed to fetch your site root.
It parses robots.txt per user-agent, so it understands rules that target GPTBot or Google-Extended specifically, not just the catch-all. It also looks for an llms.txt — an emerging standard that points AI agents to your most important content — and reports its structure if present.
Blocking AI crawlers is a legitimate choice, but it should be deliberate. Many sites block them by accident with an overly broad rule, then wonder why they never get cited. This tool makes your actual AI-crawler posture explicit so you can decide on purpose.