There are three files AI crawlers look for at the root of a domain: robots.txt, llms.txt, and agents.md. They do different jobs, and only one of them is load-bearing today. This page is the honest version of what each one is worth.
robots.txt: the one that actually gates you
robots.txt decides which crawlers can read your store at all. This is the file that matters most, because an AI shopping engine that is blocked here cannot see your products no matter how good your data is.
The crawlers that drive AI shopping answers are search crawlers, not training crawlers. Allow OAI-SearchBot (ChatGPT shopping), PerplexityBot, ClaudeBot, Googlebot, and Amazonbot. You can still block training-only bots like GPTBot or Google-Extended if you want to opt out of model training, with no effect on being recommended. AgentReady audits this and flags any search crawler you have blocked.
llms.txt: a tidy index for assistants
llms.txt is a plain-markdown file at your domain root that points assistants to your key pages and policies. Think of it as a curated table of contents written for machines. AgentReady generates a per-store llms.txt and keeps it current as your catalog changes.
Be clear-eyed about what it is worth: no major engine ranks you on llms.txt today. It is cheap, future-proof hygiene that makes your store easier for an assistant to parse, not a switch that makes ChatGPT cite you. We would rather tell you that than oversell it.
agents.md: the emerging convention
agents.md is a newer markdown guide that describes your store to AI agents (who you are, what you sell, your key pages and policies). Shopify added native support for serving one in 2026. Like llms.txt, it is an emerging convention rather than a ranking signal yet, so it scores as low-weight discovery hygiene.
AgentReady generates and maintains an agents.md for you from a few inputs you control (why buy from you, sizing notes, featured products), so it stays accurate without you hand-editing a file.
What AgentReady does here
- Audits
robots.txtand tells you which AI crawlers can and cannot reach your store. - Generates and keeps a per-store
llms.txtindex current. - Generates and maintains your
agents.mdfrom your own inputs. - Confirms your XML sitemap exists so crawlers can discover every page.
What stays the moat
The files above are hygiene. The thing that actually decides whether an assistant recommends you is the quality and freshness of your product data: clean structured data, identifiers, accurate prices and availability. The discovery files make sure a crawler can find that data. The data itself is what wins the recommendation. See structured data for that side.