Is your site blocking GPTBot, ClaudeBot, or the other AI crawlers?
AI crawlers decide whether your content trains tomorrow's models — and whether AI search cites you at all. Most sites block them by accident: a WAF rule written for scrapers, a CDN bot policy flipped on by default, or a robots.txt line nobody remembers adding. The symptom is always the same — silence — so the block goes unnoticed for months.
Why a simple fetch test lies to you
Fetching your homepage with curl proves one thing: what your server sends to curl. It says nothing about what it sends to GPTBot. Modern edges serve different responses per client: the declared bot user-agent, the TLS fingerprint, the source IP class, even the timing of the request all feed the verdict. A site can answer 200 to your browser, 403 to GPTBot, and a JavaScript challenge to ClaudeBot — simultaneously. Single-identity checks cannot see this. Differential checks can.
What WebMTR actually checks
- Declared bot matrix: your site is fetched as Googlebot, Bingbot, GPTBot, ClaudeBot, CCBot, PerplexityBot, Facebook and LinkedIn crawlers plus Chrome and plain controls — the same URL, many identities, side by side.
- robots.txt vs enforcement: we separate what your robots file asks from what your edge does. A crawler can be allowed on paper and challenged in practice — that mismatch is the most common finding.
- WAF vs robots confusion: when a bot gets a 403 or a challenge page, the report attributes it: WAF signature, CDN bot policy, or origin rule — with the headers that prove it.
- Tor and geo exits: six Tor exits plus three scan origins show whether the block follows the client or the network path.
What you get
A graded report with every posture's verdict (open, challenge, or block), the exact blocking layer, and ready-to-paste fixes: robots.txt and llms.txt snippets, WAF allowlist rules, and Cloudflare rule examples. The sample report shows the full format on a real scan.