Docs / for WordPress / How Many AI Crawlers Can AEO Ultimate Manage? All 31, Explained

AEO Ultimate for WordPress

How many AI crawlers can AEO Ultimate manage? All 31, explained

AEO Ultimate manages 31 AI crawlers from 15 companies, including OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic’s ClaudeBot, Google, Perplexity, Meta and Apple. In WordPress, it shows which ones visit your site, lets you block any one of them or give it its own rate limit, and tells all of them what they may use your content for. Everything below is the real aeoultimate.com site.

AI bot controls in AEO Ultimate, on aeoultimate.com.
Read the transcript

0:00 How many AI bots can AEO Ultimate control?
How many AI bots can AEO Ultimate control? Thirty-one AI crawlers, from fifteen companies, including OpenAI, Anthropic, Google, Perplexity, Meta and Apple. You can see which ones visit, block any one of them, give it its own rate limit, and tell all of them what they may use your content for.

0:21 See which AI bots visit: AI Crawler Watch
First, see who comes. AI Crawler Watch, in Command Center, logs every visit from AI crawlers: which bot, which company, which page, and when. Discovery file hits show whether they fetch your robots dot text and llms dot text.

0:40 Training vs answers: content signals
Next, what they may do. Content signals separate three uses: training AI models, indexing for AI search, and AI input, which means reading your page to answer a question and citing you. To be cited, allow search and input. Training is your call.

0:59 robots.txt and your SEO plugin
These signals go out with every page and in robots dot text. You can also write allow and disallow rules for each AI crawler. Leave that off if Yoast or Rank Math already manages AI bots, so the rules don't clash.

1:14 Block one bot: per-bot controls
Per-bot controls list 31 AI crawlers by company, with what each one does. Set any of them to Block, and AEO Ultimate adds a disallow rule to robots dot text and turns the bot's requests away, even if it ignores robots dot text.

1:30 Training bots vs answer bots
Block carefully. Each company runs several bots. OpenAI's GPTBot collects training data, while OAI-SearchBot and ChatGPT-User fetch pages for ChatGPT's answers. Block GPTBot if you don't want training, but leave the search and user bots alone, or you can drop out of the answers you want to be in.

1:51 Slow a bot down: rate limits
If a bot is welcome but too busy, give it its own rate limit: a maximum number of requests per hour. Only bots you set a limit for are slowed on your pages, so Google and the AI search crawlers are never throttled by default.

2:08 Check your firewall isn't blocking AI
Last, check nothing else is blocking them. A firewall, security plugin or CDN can turn AI crawlers away even when robots dot text allows them. The Crawlability Audit requests your home page as GPTBot, ClaudeBot, PerplexityBot and Apple's crawler, and shows what each one got.

2:30 Wrap-up
So: watch who visits, allow the uses that get you cited, block only the bots you mean to, and test that the rest can get in. Thanks for watching.

Which AI crawlers does AEO Ultimate manage?

AEO Ultimate manages all 31 of these AI crawlers, from 15 companies. OpenAI: GPTBot, ChatGPT-User and OAI-SearchBot; Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web and anthropic-ai; Google: Google-Extended, Googlebot-Extended, Googlebot, GoogleOther, Google-Agent, Google-InspectionTool and Google-NotebookLM; Meta: Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot; Apple: Applebot and Applebot-Extended; Perplexity: PerplexityBot and Perplexity-User; You.com: YouBot; ByteDance: Bytespider; Common Crawl: CCBot; Amazon: AmazonBot; Diffbot: Diffbot; Cohere: cohere-ai; Huawei: PetalBot; Webz.io: Omgilibot; iAsk.ai: iaskspider. Each one is listed under TWT AEO → AI Ready → Security & Rate Limiting → Per-Bot Controls, where you can allow it, block it or give it its own rate limit. The list below says what each bot does.

  • OpenAI: GPTBot (Training crawler), ChatGPT-User (Browsing agent), OAI-SearchBot (Search indexer)
  • Anthropic: ClaudeBot (Claude.ai search), Claude-User (Live fetch for a Claude user), Claude-SearchBot (Claude search indexer), Claude-Web (Claude web browsing), anthropic-ai (Training crawler)
  • Google: Google-Extended (Gemini AI training), Googlebot-Extended (AI search indexer), Googlebot (Search Core — standard index & Discover), GoogleOther (AI content feeder — Gemini training & R&D), Google-Agent (Gemini live browser agent (Project Mariner)), Google-InspectionTool (Search Console URL Inspection tool), Google-NotebookLM (NotebookLM research assistant)
  • Meta: Meta-ExternalAgent (Meta AI agent (2024+)), Meta-ExternalFetcher (Meta AI fetcher (2024+)), FacebookBot (Meta AI training)
  • Apple: Applebot (Siri / Apple Intelligence), Applebot-Extended (Apple AI training)
  • Perplexity: PerplexityBot (AI search indexer), Perplexity-User (Live fetch for a Perplexity user)
  • You.com: YouBot (AI search indexer)
  • ByteDance: Bytespider (TikTok AI / training)
  • Common Crawl: CCBot (LLM training datasets)
  • Amazon: AmazonBot (Alexa AI / training)
  • Diffbot: Diffbot (AI knowledge graph)
  • Cohere: cohere-ai (AI training crawler)
  • Huawei: PetalBot (AI indexer)
  • Webz.io: Omgilibot (Training dataset collector)
  • iAsk.ai: iaskspider (AI search)

How do I see which AI bots visit my WordPress site?

Open TWT AEO → Command Center → AI Crawler Watch. It logs every visit from AI crawlers: which bot, which company, which page, and when, with a bot breakdown, your most crawled pages and a live feed of the last 48 hours. AI Discovery File Hits shows whether crawlers fetch your robots.txt, llms.txt and llms-full.txt. Over the last 30 days on aeoultimate.com, Googlebot visited 67 times, PetalBot 5, ClaudeBot 4, Amazonbot and OAI-SearchBot 3 each, and GPTBot once.

How do I let AI cite my content but not train on it?

Use content signals, under AI Ready → Content-Signal Headers & robots.txt. They separate three uses: AI training (building a model from your content), AI search indexing, and AI input, which is an assistant reading your page to answer a question and citing it. To be cited, allow search indexing and AI input. Training is your call: choose “Do not train” to opt out. The signals go out as an HTTP header on every page and in robots.txt.

You can also write per-bot Allow and Disallow rules for known AI crawlers into robots.txt. Leave that off if Yoast SEO, Rank Math or another SEO plugin already manages AI bots, so the rules don’t clash.

How do I block a single AI bot in WordPress?

In Per-Bot Controls, set that bot to Block and save. AEO Ultimate adds a Disallow: / rule for it to robots.txt, and returns 403 Forbidden to its requests, so it is turned away even if it ignores robots.txt. Every other bot is unaffected.

Should I block GPTBot?

Only if you don’t want your content used for training, and then block GPTBot alone. Each AI company runs several bots with different jobs. OpenAI’s GPTBot collects training data, while OAI-SearchBot and ChatGPT-User fetch pages for ChatGPT’s answers. Anthropic splits the same way: anthropic-ai is listed as its training crawler, and Claude-User fetches pages live for a Claude user. Blocking the search and user bots can take you out of the answers you want to be cited in.

Can I rate-limit an AI crawler instead of blocking it?

Yes. If a bot is welcome but too busy, switch Rate Limiting on and give that bot its own number in the Rate limit / hr column. Once it reaches that many requests per hour from one address, it is asked to come back in an hour (HTTP 429). Leave the box blank for no limit on your pages: only bots you set a limit for are slowed, so Googlebot and the AI search crawlers are never throttled by default. The separate global cap under Rate Limiting covers the plugin’s agent endpoints, not your pages.

How do I check a firewall isn’t blocking AI crawlers?

Run Command Center → Crawlability Audit. A firewall, security plugin or CDN can turn AI crawlers away even when robots.txt allows them. The audit requests your home page as GPTBot, ClaudeBot, PerplexityBot and Applebot-Extended and shows the response each one got. On aeoultimate.com, all four get through.

AEO audit and setup · one time

Keep Yoast. We’ll set up the AI side next to it.

Your SEO plugin keeps doing its job. For a one-time $250, we audit your WordPress site for AI search, set up AEO Ultimate alongside Yoast, Rank Math, All in One SEO or SEOPress, and run a week of AI checks on ChatGPT, Google and Perplexity, so you see whether they name you, and who they name instead.

  • Nothing gets uninstalled and no URLs change.
  • Lead tracking on your contact forms, so you see which leads came from AI tools.
  • Done within two business days of getting access, then a walkthrough for you or your SEO team.
  • No monthly fee. The plugin stays free.
Get the AEO audit, $250Or install the plugin free

The audit is sold by Tampa Web Technologies, the company that makes AEO Ultimate, so the button takes you to checkout on tampawebtech.com.

Related: the AI Ready guide, Command Center, SEO vs answer engine optimization, or all tutorials.

Still stuck?

Message me directly — free tier included. Most replies land within the hour during Tampa working hours. Get support →

How does AEO Ultimate stack with your SEO plugin?

Enter your website to see what your site and SEO plugin already cover, and what AEO Ultimate adds on top.

Free, no sign-up. We read only what your site shows the public — about 10 pages and files. We keep anonymous statistics (which checks passed, which SEO plugin a site runs), never your site's address.