Is your WAF blocking Googlebot — or your own visitors?
Web application firewalls fail in two directions. Turned down, scrapers and credential-stuffing crews walk through. Turned up, Googlebot gets challenged, Tor visitors hit walls, and real customers bounce off a CAPTCHA on checkout day. Both failures are silent: nobody files a ticket saying "your WAF challenged a crawler last Tuesday."
Challenge, block, rate-limit: different problems, different fixes
A challenge (JavaScript wall, CAPTCHA, managed challenge) costs crawlers their visit and users their patience. A block (403/406/503 with a WAF signature) is absolute. A rate-limit (429s under burst, clean when quiet) masquerades as flakiness. WebMTR separates all three: a burst pass plus a 60-second-quiet confirmation pass distinguish velocity throttling you inflicted on yourself from a standing gate that needs an allowlist.
What WebMTR actually checks
- Header plus content sniffing: HTTP status and WAF signatures plus a real-Chromium content check, so HTTP-200 block pages — the ones header-only scanners miss — are caught.
- Bot-differential matrix: twelve-plus crawler identities against the same URL reveal whether the gate targets automation markers, IP class, or everyone equally.
- Akamai and Cloudflare signatures: edge-specific tells (AkamaiGHost behaviors, Cloudflare challenge markers, per-colo variance) are attributed to the layer that raised them.
- Quiet confirmation: suspected burst-echo verdicts are re-checked after a quiet window before being called stable gates.
What you get
Blocked scans still complete — network, DNS, TLS and MTR data stay valid — with a clear banner naming the blocking layer, the triggering evidence, and the allowlist or rule fix: WAF rule edits, egress-IP allowlisting, and Cloudflare rule examples. False positives are the enemy here, so every verdict carries its proof. See the format in the sample report.