Discovery
CRITICALGooglebot allowed on product paths
robots.txt permits Googlebot at a representative product-detail path. Google's shopping and AI Overview answers cite product pages Googlebot was permitted to crawl.
What this check looks for
We pick a representative product path (from the PDP sample first, or from a platform-specific shaped path, or from the sitemap) and ask the parsed robots.txt whether Googlebot is allowed there. Because product paths are the surface Google's merchant-listing pipeline ingests from, a Disallow at the product path is functionally equivalent to opting out of Google shopping even if the homepage is allowed.
Which AI surfaces it affects
- Google AI Mode (UCP)100
- Microsoft Copilot60
- Perplexity50
- ChatGPT (ACP)30
- Meta AI30
Weighted against the live specs — ACP 2026-04-17, UCP 2026-04-08.
How to fix it
Allow Googlebot on product paths
Shopify
A few minutes- Online Store → Themes → Edit code → templates/robots.txt.liquid.
- Search for any `disallow: /products` or wildcard `disallow: /` rules. Remove them or add `allow: /products/` above any restrictive line.
BigCommerce
A few minutes- Storefront → SEO → robots.txt.
- Make sure no `Disallow` line under Googlebot or the wildcard group covers your product URL pattern.
WooCommerce
A few minutes- Yoast SEO → Tools → File editor → robots.txt (or All in One SEO → Robots.txt Editor).
- Confirm no `Disallow: /product/` (or wildcard) rule blocks Googlebot.
Custom / headless
A few minutes- Open robots.txt and identify every group that matches Googlebot (an explicit Googlebot group AND the wildcard `*` group).
- Ensure each one resolves to Allow for your product path prefix.
User-agent: Googlebot
Allow: /products/
Allow: /product/
Allow: /p/
User-agent: *
Disallow: /admin/
Allow: /
The spec it's pinned to
Google merchant listing — crawlability
Google's merchant-listing documentation requires Googlebot to be able to crawl product pages so they can be surfaced in Google's shopping and merchant-listing experiences. A robots.txt Disallow on product paths breaks this end-to-end.
Does your store pass this check?
Run the full audit — 82 checks across five AI shopping surfaces. Most tools only check whether you get mentioned; we check whether an agent can buy from you.