Discovery
LOW/robots.txt is served as UTF-8
The /robots.txt body decodes cleanly as UTF-8 per RFC 9309 §2.3. Non-UTF-8 robots files are silently dropped by Google's parser; the merchant loses all per-UA control.
What this check looks for
RFC 9309 §2.3 mandates UTF-8 encoding for `/robots.txt`. We inspect the raw byte stream captured by the fetcher and confirm UTF-8 decodability. Any file that decodes as another encoding (or fails to decode at all) is reported with the actual Content-Type.
Which AI surfaces it affects
- ChatGPT (ACP)40
- Google AI Mode (UCP)40
- Perplexity40
- Microsoft Copilot40
- Meta AI40
Weighted against the live specs — ACP 2026-04-17, UCP 2026-04-08.
How to fix it
Serve /robots.txt as UTF-8
Shopify
Developer- Shopify serves robots.txt as UTF-8 by default. If this fails on Shopify, you're auditing through a proxy that re-encodes the response.
BigCommerce
Developer- BigCommerce serves robots.txt as UTF-8 by default. Inspect any CDN/edge worker rewriting the response.
WooCommerce
A few minutes- Yoast / Rank Math / All in One SEO all emit UTF-8 robots.txt. Confirm WordPress is set to UTF-8 (Settings → Reading) and that no plugin is overriding the response encoding.
Custom / headless
Developer- Encode the robots.txt as UTF-8 and serve it with `Content-Type: text/plain; charset=utf-8`.
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThe spec it's pinned to
RFC 9309 §2.3 — File format (encoding)
RFC 9309 §2.3 requires `/robots.txt` to be encoded in UTF-8. Non-UTF-8 files are dropped by Google's parser and trip default parsers in every standard library.
Does your store pass this check?
Run the full audit — 82 checks across five AI shopping surfaces. Most tools only check whether you get mentioned; we check whether an agent can buy from you.