--- title: "AI crawlers, robots.txt and duplicate product schema on Shopify" url: "https://aeoultimate.com/docs/shop/crawlers-schema/" description: "See which AI crawlers read your Shopify store, prepare robots.txt rules and content signals for GPTBot, ClaudeBot and others, and switch off your theme’s duplicate product schema." published: "2026-10-24T22:51:57+00:00" modified: "2026-10-24T23:59:57+00:00" language: "en-US" site: "AEO Ultimate" --- Source: https://aeoultimate.com/docs/shop/crawlers-schema/ AEO Ultimate for Shopify # AI crawlers and schema ownership: what the bots read Two Pro pages cover what AI crawlers read from your Shopify store: who is allowed to read it, and whose product structured data they find there. Neither changes your theme: where a theme file has to change, the app prepares the exact edit, you make it, and the app checks the result. AI crawler activity and schema ownership in the Shopify admin. ## How do I know which AI crawlers visit my store? Open **Pro checks → AI crawler activity**. It logs every fetch of the app’s own AI-crawler files: llms.txt, llms-full.txt and the knowledge bundle. A user agent is a claim anyone can type, so each visit is checked against the IP ranges the crawler’s company publishes: **Verified IP** means the address is inside them, **Claimed only** means it is not. Some operators, such as Common Crawl, publish no ranges, and the page says so. It covers the files this app serves, not your whole storefront, because Shopify does not give apps your theme’s server logs. [Image: AEO Ultimate for Shopify AI crawler activity page: what this measures, verified IP versus claimed only, and Who may read your store with notes that you make the robots.txt edit and that blocking does not stop AI agents buying] AI crawler activity: verified visits, and who may read your store. ## How do I control what AI crawlers may read? Under **Who may read your store**, each crawler the app recognises, from GPTBot and ClaudeBot to PerplexityBot, Applebot and Meta’s agent, has its own choices: block everything, or just product, collection, blog, page or policy pages. Everything is allowed by default, which is almost always what a store wants. [Image: AEO Ultimate for Shopify robots.txt choices per AI crawler, such as Bingbot, CCBot, Bytespider, Amazonbot, Applebot and Meta-ExternalAgent, each with Block everything, product, collection, blog, pages and policy pages options] Choose what each AI crawler may not read. An optional **Content-Signal** line (contentsignals.org) says what a crawler may do with what it reads. The recommended setting allows search and AI answers, which is how a store gets cited, and says no to AI training, a separate question. Like the rest of robots.txt, it is a request that well-behaved crawlers honour. [Image: AEO Ultimate for Shopify content signal options for search, AI answers and AI training, the Prepare my robots.txt button, and an empty Crawlers seen log on a password-protected development store] Content signals, and the crawler log (empty here: the development store is password-protected). ## Does the app edit my robots.txt? No. Your theme’s robots.txt.liquid is yours, and Shopify apps do not edit theme code. **Prepare my robots.txt** builds the exact file from what your theme has now, keeping every rule Shopify generates and every rule you added, and shows you where to paste it. After you save it, **Check my live robots.txt** reads your public /robots.txt to confirm what crawlers actually see. [Image: AEO Ultimate for Shopify prepared robots.txt.liquid: steps to open the template, paste the whole file and check the live robots.txt, and the file itself with the app block] The prepared file and where to paste it. You make the edit; the app checks it. ## Does blocking an AI crawler stop AI agents buying from me? No. Crawling and AI purchasing are separate. Blocking a crawler changes how that company reads your pages; agentic selling lives in Shopify under Settings → Sales channels and is on by default for eligible stores. Blocking may still cost you that company’s answers about your content. ## Why does duplicate product schema matter for AI? Open **Pro checks → Schema ownership**. Most themes, including Dawn, already publish Product structured data, and this app adds its own richer version. Its storefront block removes the theme’s copy in the browser, so Google, which runs JavaScript, sees one product. ChatGPT, Perplexity and most AI crawlers read the page as the server sent it, where both copies are still present. The only fix that reaches them is switching the theme’s copy off in the theme itself. [Image: AEO Ultimate for Shopify Schema ownership page: why a theme and the app both publishing Product data matters, why the browser fix is not enough for AI crawlers, and the product picker] Schema ownership: who publishes your product’s structured data. Pick a product and the app reads its page and lists every block of structured data and who publishes it. A store behind a password cannot be read from the app’s server, so paste the page source instead and it is checked with the same rules. A pasted check says what the markup says, not whether crawlers can reach the page. [Image: AEO Ultimate for Shopify Check pasted page source: steps to view a product page source and paste it for the same checks when the store is password-protected] Behind a password? Paste the page source instead. ## How do I switch off my theme’s product schema? **Check my theme** reads your live theme and finds where it prints its own product JSON-LD. For each block it shows the exact source and links straight to that file in your theme code editor, where two lines, {% comment %} above and {% endcomment %} below, switch it off without deleting anything. You make the edit; then press Check my theme again to confirm it worked. [Image: AEO Ultimate for Shopify Switch off the theme’s own product schema: Check my theme found hand-written Product JSON-LD in sections/featured-product.liquid and shows its exact source] Check my theme finds the theme’s own product schema and shows the exact source. ## What should I set up next? Next is [the RAG Engine](https://aeoultimate.com/docs/shop/rag-engine/). Every walkthrough is listed in the [Shopify video tutorials](https://aeoultimate.com/docs/shop/tutorials/).