Discovery
LOW/robots.txt is served as text/plain
The /robots.txt response carries `Content-Type: text/plain`. A wrong content type makes strict crawlers ignore the file entirely and fall back to permissive defaults.
What this check looks for
RFC 9309 §2.3 treats robots.txt as a plain-text resource. Strict crawlers refuse to parse robots directives when the body is served with a non-`text/plain` content type — `text/html` or `application/octet-stream` are both common causes of all-allow defaults silently taking over. We accept any media type whose lower-cased value starts with `text/plain`, with or without a charset parameter.
Which AI surfaces it affects
- ChatGPT (ACP)40
- Google AI Mode (UCP)40
- Perplexity40
- Microsoft Copilot40
- Meta AI40
Weighted against the live specs — ACP 2026-04-17, UCP 2026-04-08.
How to fix it
Send Content-Type: text/plain on /robots.txt
Shopify
Developer- Shopify serves /robots.txt as text/plain by default. If this fails on Shopify, your store is behind a custom proxy that's rewriting the content type — fix the proxy.
BigCommerce
Developer- BigCommerce serves /robots.txt as text/plain by default. Inspect any CDN/edge worker rewriting headers.
WooCommerce
A few minutes- WordPress emits /robots.txt as text/plain by default. If failing, a server-config snippet or security plugin is overriding the type — disable it.
Custom / headless
Developer- Set the content type explicitly when serving /robots.txt.
Content-Type: text/plain; charset=utf-8The spec it's pinned to
RFC 9309 §2.3 — File format (content type)
RFC 9309 treats robots.txt as a plain-text resource. Strict crawlers refuse to parse non-text/plain bodies even when the syntax is valid.
Does your store pass this check?
Run the full audit — 82 checks across five AI shopping surfaces. Most tools only check whether you get mentioned; we check whether an agent can buy from you.