Discovery
CRITICALNo global wildcard root disallow
robots.txt has no wildcard `User-agent: *` group with `Disallow: /` that locks every crawler out. A site-wide Disallow of `/` blocks every agent crawler at once — catastrophic across every surface.
What this check looks for
RFC 9309 section 2.2.2 defines `Disallow: /` in a `User-agent: *` group as a site-wide opt-out for every crawler that doesn't have its own more specific group. We scan the raw robots.txt line by line, track which group we're inside (stacked User-agent lines form one combined group), and flag the file when the wildcard group has a root Disallow with no offsetting root Allow. Almost always a staging-deploy artifact that escaped to production.
Which AI surfaces it affects
- ChatGPT (ACP)100
- Google AI Mode (UCP)100
- Perplexity100
- Microsoft Copilot100
- Meta AI100
Weighted against the live specs — ACP 2026-04-17, UCP 2026-04-08.
How to fix it
Remove the wildcard `Disallow: /` from robots.txt
Shopify
A few minutes- Online Store → Themes → Edit code → templates/robots.txt.liquid.
- Confirm the file does NOT contain `User-agent: *` followed by `Disallow: /`.
- If your store is in development/preview mode, that can also surface as a wildcard block — disable preview before audit.
BigCommerce
A few minutes- Storefront → SEO → robots.txt.
- Confirm the published file does NOT contain a wildcard `Disallow: /`.
- Also check Settings → Channels — a channel in down-for-maintenance mode can publish a blocking robots.txt.
WooCommerce
A few minutes- Settings → Reading → confirm 'Discourage search engines from indexing this site' is OFF — that toggle generates a wildcard Disallow.
- Yoast SEO → Tools → File editor → robots.txt; verify no wildcard `Disallow: /` is present.
Custom / headless
A few minutes- Open robots.txt and remove or replace any `Disallow: /` line in the `User-agent: *` group.
- If you must keep some paths blocked, enumerate them — never use a root `/` as the Disallow path in a wildcard group.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart
The spec it's pinned to
RFC 9309 section 2.2.2 — Disallow rules
RFC 9309 section 2.2.2 defines Disallow semantics. A `User-agent: *` group with `Disallow: /` and no offsetting `Allow: /` blocks every well-behaved crawler from the entire site — the cardinal robots.txt sin, usually a staging artifact promoted to production.
Does your store pass this check?
Run the full audit — 82 checks across five AI shopping surfaces. Most tools only check whether you get mentioned; we check whether an agent can buy from you.