---
title: "How Many AI Crawlers Can AEO Ultimate Manage? All 31, Explained"
url: "https://aeoultimate.com/docs/wordpress/ai-bot-controls/"
description: "AEO Ultimate manages 31 AI crawlers from 15 companies: see which visit, block any one, give it a rate limit, and choose what AI may use your content for."
author: "David Chamberlain"
published: "2026-10-28T01:01:25+00:00"
modified: "2026-11-10T01:00:56+00:00"
language: "en-US"
site: "AEO Ultimate"
---

Source: https://aeoultimate.com/docs/wordpress/ai-bot-controls/

AEO Ultimate for WordPress

# How many AI crawlers can AEO Ultimate manage? All 31, explained

AEO Ultimate manages 31 AI crawlers from 15 companies, including OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic’s ClaudeBot, Google, Perplexity, Meta and Apple. In WordPress, it shows which ones visit your site, lets you block any one of them or give it its own rate limit, and tells all of them what they may use your content for. Everything below is the real aeoultimate.com site.

AI bot controls in AEO Ultimate, on aeoultimate.com.

Read the transcript

**0:00 How many AI bots can AEO Ultimate control?**
How many AI bots can AEO Ultimate control? Thirty-one AI crawlers, from fifteen companies, including OpenAI, Anthropic, Google, Perplexity, Meta and Apple. You can see which ones visit, block any one of them, give it its own rate limit, and tell all of them what they may use your content for.

**0:21 See which AI bots visit: AI Crawler Watch**
First, see who comes. AI Crawler Watch, in Command Center, logs every visit from AI crawlers: which bot, which company, which page, and when. Discovery file hits show whether they fetch your robots dot text and llms dot text.

**0:40 Training vs answers: content signals**
Next, what they may do. Content signals separate three uses: training AI models, indexing for AI search, and AI input, which means reading your page to answer a question and citing you. To be cited, allow search and input. Training is your call.

**0:59 robots.txt and your SEO plugin**
These signals go out with every page and in robots dot text. You can also write allow and disallow rules for each AI crawler. Leave that off if Yoast or Rank Math already manages AI bots, so the rules don't clash.

**1:14 Block one bot: per-bot controls**
Per-bot controls list 31 AI crawlers by company, with what each one does. Set any of them to Block, and AEO Ultimate adds a disallow rule to robots dot text and turns the bot's requests away, even if it ignores robots dot text.

**1:30 Training bots vs answer bots**
Block carefully. Each company runs several bots. OpenAI's GPTBot collects training data, while OAI-SearchBot and ChatGPT-User fetch pages for ChatGPT's answers. Block GPTBot if you don't want training, but leave the search and user bots alone, or you can drop out of the answers you want to be in.

**1:51 Slow a bot down: rate limits**
If a bot is welcome but too busy, give it its own rate limit: a maximum number of requests per hour. Only bots you set a limit for are slowed on your pages, so Google and the AI search crawlers are never throttled by default.

**2:08 Check your firewall isn't blocking AI**
Last, check nothing else is blocking them. A firewall, security plugin or CDN can turn AI crawlers away even when robots dot text allows them. The Crawlability Audit requests your home page as GPTBot, ClaudeBot, PerplexityBot and Apple's crawler, and shows what each one got.

**2:30 Wrap-up**
So: watch who visits, allow the uses that get you cited, block only the bots you mean to, and test that the rest can get in. Thanks for watching.

## Which AI crawlers does AEO Ultimate manage?

AEO Ultimate manages all 31 of these AI crawlers, from 15 companies. OpenAI: GPTBot, ChatGPT-User and OAI-SearchBot; Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web and anthropic-ai; Google: Google-Extended, Googlebot-Extended, Googlebot, GoogleOther, Google-Agent, Google-InspectionTool and Google-NotebookLM; Meta: Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot; Apple: Applebot and Applebot-Extended; Perplexity: PerplexityBot and Perplexity-User; You.com: YouBot; ByteDance: Bytespider; Common Crawl: CCBot; Amazon: AmazonBot; Diffbot: Diffbot; Cohere: cohere-ai; Huawei: PetalBot; Webz.io: Omgilibot; iAsk.ai: iaskspider. Each one is listed under **TWT AEO → AI Ready → Security & Rate Limiting → Per-Bot Controls**, where you can allow it, block it or give it its own rate limit. The list below says what each bot does.

- **OpenAI:** `GPTBot` (Training crawler), `ChatGPT-User` (Browsing agent), `OAI-SearchBot` (Search indexer)
- **Anthropic:** `ClaudeBot` (Claude.ai search), `Claude-User` (Live fetch for a Claude user), `Claude-SearchBot` (Claude search indexer), `Claude-Web` (Claude web browsing), `anthropic-ai` (Training crawler)
- **Google:** `Google-Extended` (Gemini AI training), `Googlebot-Extended` (AI search indexer), `Googlebot` (Search Core — standard index & Discover), `GoogleOther` (AI content feeder — Gemini training & R&D), `Google-Agent` (Gemini live browser agent (Project Mariner)), `Google-InspectionTool` (Search Console URL Inspection tool), `Google-NotebookLM` (NotebookLM research assistant)
- **Meta:** `Meta-ExternalAgent` (Meta AI agent (2024+)), `Meta-ExternalFetcher` (Meta AI fetcher (2024+)), `FacebookBot` (Meta AI training)
- **Apple:** `Applebot` (Siri / Apple Intelligence), `Applebot-Extended` (Apple AI training)
- **Perplexity:** `PerplexityBot` (AI search indexer), `Perplexity-User` (Live fetch for a Perplexity user)
- **You.com:** `YouBot` (AI search indexer)
- **ByteDance:** `Bytespider` (TikTok AI / training)
- **Common Crawl:** `CCBot` (LLM training datasets)
- **Amazon:** `AmazonBot` (Alexa AI / training)
- **Diffbot:** `Diffbot` (AI knowledge graph)
- **Cohere:** `cohere-ai` (AI training crawler)
- **Huawei:** `PetalBot` (AI indexer)
- **Webz.io:** `Omgilibot` (Training dataset collector)
- **iAsk.ai:** `iaskspider` (AI search)

## How do I see which AI bots visit my WordPress site?

Open **TWT AEO → Command Center → AI Crawler Watch**. It logs every visit from AI crawlers: which bot, which company, which page, and when, with a bot breakdown, your most crawled pages and a live feed of the last 48 hours. **AI Discovery File Hits** shows whether crawlers fetch your robots.txt, llms.txt and llms-full.txt. Over the last 30 days on aeoultimate.com, Googlebot visited 67 times, PetalBot 5, ClaudeBot 4, Amazonbot and OAI-SearchBot 3 each, and GPTBot once.

[Image: AEO Ultimate AI Crawler Watch for aeoultimate.com, last 30 days: Googlebot 67 visits, PetalBot 5, ClaudeBot 4, Amazonbot 3, OAI-SearchBot 3 and GPTBot 1, with the most crawled pages and a live feed of the last 48 hours]

AI Crawler Watch on aeoultimate.com: which bots came, from which company, and which pages they read.

## How do I let AI cite my content but not train on it?

Use content signals, under **AI Ready → Content-Signal Headers & robots.txt**. They separate three uses: **AI training** (building a model from your content), **AI search indexing**, and **AI input**, which is an assistant reading your page to answer a question and citing it. To be cited, allow search indexing and AI input. Training is your call: choose “Do not train” to opt out. The signals go out as an HTTP header on every page and in robots.txt.

You can also write per-bot Allow and Disallow rules for known AI crawlers into robots.txt. Leave that off if Yoast SEO, Rank Math or another SEO plugin already manages AI bots, so the rules don’t clash.

[Image: AI Ready content signals: AI Training, AI Search Indexing and AI Input / RAG, each with allow, do not and no preference options, the Content-Signal output directive, and the option to write per-bot Allow and Disallow rules to robots.txt]

Three separate permissions. To be cited, allow search indexing and AI input; training is your call.

## How do I block a single AI bot in WordPress?

In **Per-Bot Controls**, set that bot to **Block** and save. AEO Ultimate adds a `Disallow: /` rule for it to robots.txt, and returns 403 Forbidden to its requests, so it is turned away even if it ignores robots.txt. Every other bot is unaffected.

[Image: Per-Bot Controls table listing each AI crawler with its company, what it does, an Allow or Block status and a rate limit per hour: GPTBot, ChatGPT-User and OAI-SearchBot from OpenAI, ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web and anthropic-ai from Anthropic, then Google-Extended and Googlebot-Extended]

Every AI crawler by company, with what it does, Allow or Block, and an optional hourly limit.

## Should I block GPTBot?

Only if you don’t want your content used for training, and then block GPTBot alone. Each AI company runs several bots with different jobs. OpenAI’s **GPTBot** collects training data, while **OAI-SearchBot** and **ChatGPT-User** fetch pages for ChatGPT’s answers. Anthropic splits the same way: **anthropic-ai** is listed as its training crawler, and **Claude-User** fetches pages live for a Claude user. Blocking the search and user bots can take you out of the answers you want to be cited in.

## Can I rate-limit an AI crawler instead of blocking it?

Yes. If a bot is welcome but too busy, switch **Rate Limiting** on and give that bot its own number in the **Rate limit / hr** column. Once it reaches that many requests per hour from one address, it is asked to come back in an hour (HTTP 429). Leave the box blank for no limit on your pages: only bots you set a limit for are slowed, so Googlebot and the AI search crawlers are never throttled by default. The separate global cap under **Rate Limiting** covers the plugin’s agent endpoints, not your pages.

[Image: Rate Limiting card explaining that ordinary pages are never limited by the agent endpoint cap, and the per-bot rate limit column with blank boxes meaning no page limit]

Blank means no page limit. Only a bot you give its own limit is slowed on your pages.

## How do I check a firewall isn’t blocking AI crawlers?

Run **Command Center → Crawlability Audit**. A firewall, security plugin or CDN can turn AI crawlers away even when robots.txt allows them. The audit requests your home page as GPTBot, ClaudeBot, PerplexityBot and Applebot-Extended and shows the response each one got. On aeoultimate.com, all four get through.

[Image: Command Center Crawlability Audit for aeoultimate.com: GPTBot, ClaudeBot, PerplexityBot and Applebot-Extended each got HTTP 200, Accessible, no action needed]

The Crawlability Audit requests your home page as four AI crawlers. Here, every one gets through.

AEO audit and setup · one time

Keep Yoast. We’ll set up the AI side next to it.

Your SEO plugin keeps doing its job. For a one-time $250, we audit your WordPress site for AI search, set up AEO Ultimate alongside Yoast, Rank Math, All in One SEO or SEOPress, and run a week of AI checks on ChatGPT, Google and Perplexity, so you see whether they name you, and who they name instead.

- Nothing gets uninstalled and no URLs change.
- Lead tracking on your contact forms, so you see which leads came from AI tools.
- Done within two business days of getting access, then a walkthrough for you or your SEO team.
- No monthly fee. The plugin stays free.

[Get the AEO audit, $250](https://tampawebtech.com/downloads/aeo-audit/)[Or install the plugin free](https://wordpress.org/plugins/twt-aeo-ultimate/)

The audit is sold by Tampa Web Technologies, the company that makes AEO Ultimate, so the button takes you to checkout on tampawebtech.com.

Related: the [AI Ready guide](https://aeoultimate.com/docs/wordpress/ai-ready/), [Command Center](https://aeoultimate.com/docs/wordpress/command-center/), [SEO vs answer engine optimization](https://aeoultimate.com/docs/wordpress/seo-vs-aeo/), or [all tutorials](https://aeoultimate.com/docs/wordpress/tutorials/).