Discovery
LOW/robots.txt is under 500 KiB (RFC 9309 §2.5 parser cap)
The /robots.txt body is under 512,000 bytes (500 KiB). An oversized robots.txt is truncated by Google and Bing — trailing rules and Sitemap directives are silently lost.
What this check looks for
RFC 9309 §2.5 establishes 500 KiB as the minimum size every compliant crawler will parse. Files above that size risk being truncated by Google and Bing mid-parse, which silently drops trailing rules and Sitemap directives. We check the raw byte count of the file as fetched.
Which AI surfaces it affects
- ChatGPT (ACP)40
- Google AI Mode (UCP)40
- Perplexity40
- Microsoft Copilot40
- Meta AI40
Weighted against the live specs — ACP 2026-04-17, UCP 2026-04-08.
How to fix it
Trim /robots.txt below 500 KiB
Shopify
A few minutes- Shopify's default robots.txt is tiny. If you've customised `robots.txt.liquid` with many hand-listed patterns, replace them with wildcards.
BigCommerce
A few minutes- BigCommerce's robots.txt is generated from the SEO settings UI; the file rarely exceeds 1 KiB. If failing, you've added many manual rules — replace them with wildcards.
WooCommerce
A few minutes- Yoast / All in One SEO let you edit robots.txt directly. Remove hand-listed URL patterns and use `*` wildcards instead.
Custom / headless
Developer- Refactor the file to use wildcard patterns; verify with `wc -c < robots.txt` that you're under 512,000 bytes (500 KiB).
User-agent: *
Disallow: /tag/*
Disallow: /search?*
Allow: /products/*
Sitemap: https://yourdomain.com/sitemap.xmlThe spec it's pinned to
RFC 9309 §2.5 — Limits
RFC 9309 §2.5 sets a minimum parser cap of 500 kibibytes. Files over the cap are truncated by major crawlers, so trailing groups become unreliable.
Does your store pass this check?
Run the full audit — 82 checks across five AI shopping surfaces. Most tools only check whether you get mentioned; we check whether an agent can buy from you.