Cortexa AI Glossary · AI at work and in business
What Is an AI Crawler?
From Cortexa Learn, by Cortexa Consulting. Last checked .
Listen · From Cortexa Learn
Tip: once it’s playing, tap any line to jump the audio there.
Your website has visitors that never read a word for themselves. Some of them work for chatbots.
Visitors that aren't people
If you run a website and open its visitor logs, you'll find plenty of traffic that isn't people. Some of those visitors have names like GPTBot or ClaudeBot. They're programs sent by artificial intelligence (AI) companies to read web pages. A program that fetches web pages on its own is called a crawler, and one that does it for an AI company is an AI crawler.
Three different jobs
AI crawlers do three different jobs, and the difference matters.
- Training crawlers collect pages that may be used to train future models.
- Search crawlers build the index a chatbot searches when it answers with links.
- User fetchers visit a page because a person just asked the chatbot about it.
OpenAI and Anthropic each run all three, under separate names, so a site owner can treat each one differently.12
The robots file
Site owners talk to crawlers through a small text file called robots.txt, which sits at the top of a website. It lists which crawlers may read which pages. Following it is voluntary. The web company Cloudflare puts it plainly: the file expresses your preferences, but it doesn't stop a crawler at a technical level. A well-behaved crawler follows it, and Anthropic says its crawlers do.13
Block, allow, or choose
Each choice has a cost. Block the training crawlers, and you signal that your future pages should stay out of training data. Block the search crawlers or the user fetchers, and you can drop out of chatbot answers that would have linked to you. Because the names are separate, a site can welcome a company's search crawler and still turn its training crawler away. Defaults are shifting, too. Since September 2026, new sites on Cloudflare start out blocking training crawlers and user fetchers on pages that show ads, while search crawlers stay allowed. The owner can change that.14
A name is only a label
A crawler's name is a line of text the visitor writes about itself, so anyone can borrow it. A 2026 field study of one new website found traffic calling itself GPTBot and ClaudeBot that wasn't crawling at all. It was hunting for passwords and private keys. Every one of those requests was refused. Anthropic publishes the internet addresses its crawlers come from, so a site can check that a visitor is who it claims to be.1
Read, then quoted
For a business, AI crawlers are the first step toward showing up in AI answers. A page that's never read can't be quoted. Being read is no promise, though. In that same field study, the site's logs showed an AI company fetching its pages in the same minutes it answered a question, and then answering without citing them.
Your call
There's no single right setting. A news site protecting its archive and a small shop that wants to be recommended may choose opposite answers. If you run a site, open its robots.txt file and read what it allows today. Then decide: who do you want reading your pages?
Works cited
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” (2026) (checked )
- Cloudflare, “Bot reference” (AI Crawl Control documentation) (checked )
- Cloudflare, “robots.txt setting” (Bots documentation) (checked )
- Cloudflare, “Block AI Bots” (Bots documentation) (checked )