The Robots Exclusion Protocol (RFC 9309) is the foundational standard for how web crawlers — including AI bots and LLM data pipelines — are allowed to access a site. This tool fetches https://{domain}/robots.txt and runs 13 checks: file reachability, content type, sitemap, wildcard rule, plus per-bot policy resolution for the eight most active LLM crawlers (GPTBot, ChatGPT-User, Google-Extended, CCBot, anthropic-ai, ClaudeBot, PerplexityBot, cohere-ai).
https://{domain}/robots.txt