SEO & GEO
AI crawlers and llms.txt
Which AI crawlers Startica checks your robots.txt for, what blocking each kind means, how robots.txt is read, and what llms.txt is.
AI companies read websites with crawlers, for different reasons: to train models, to build the search index their assistants cite, and to fetch a page when someone asks about it. Your site’s robots.txt can allow or block each one. This page explains the crawlers Startica checks for, what blocking each kind means, how Startica reads robots.txt, and what llms.txt is. Both files show on the SEO & GEO Overview, in the AI search access card and the AI crawlers table.
The crawlers Startica checks
Each crawler is known by the name (user-agent token) it answers to in robots.txt. Startica checks these, by what they’re for:
| Crawler | Operator | For |
|---|---|---|
GPTBot |
OpenAI | Model training |
OAI-SearchBot |
OpenAI | AI search index (citations in answers) |
ChatGPT-User |
OpenAI | Fetching a page someone asked about |
ClaudeBot |
Anthropic | Model training |
Claude-SearchBot |
Anthropic | AI search index (citations in answers) |
Claude-User |
Anthropic | Fetching a page someone asked about |
PerplexityBot |
Perplexity | AI search index (citations in answers) |
Perplexity-User |
Perplexity | Fetching a page someone asked about |
Google-Extended |
Model training | |
Applebot-Extended |
Apple | Model training |
Google-Extended and Applebot-Extended aren’t separate crawlers. They’re names you can use in robots.txt to say whether pages the company’s normal crawler fetched may be used for AI training.
The list was last checked against the operators’ published lists on 2026-10-10; the AI crawlers card shows that date. Operators add and rename crawlers, so the list is updated with Startica. It isn’t a list of every AI crawler.
Training versus search
The three kinds of crawler do different things, so blocking them has different effects:
| If you block | What happens | How Startica reports it |
|---|---|---|
| Training crawlers | Your pages aren’t used to train those models. Assistants can still cite your site in answers, as long as their search and user crawlers are allowed. | AI training crawlers are blocked, under Good to know. It doesn’t count against the score: it’s your choice. |
| Search crawlers | The assistant’s search index leaves your site out, so its answers can’t cite you. | AI search or answer crawlers are blocked, high severity. |
| User crawlers | The assistant can’t open your page when someone asks about it or pastes its link. | AI search or answer crawlers are blocked, high severity. |
A common reason for blocked search crawlers is a broad rule meant for training crawlers, such as blocking every AI crawler by name, or a User-agent: * group with Disallow: /. If you want to stay out of AI training but still appear in AI answers, block only the training crawlers.
Note
Allowing a crawler makes it possible for an assistant to read and cite your site. It doesn’t mean the assistant will. Whether and how a site appears in AI answers is up to each AI provider.
How Startica reads robots.txt
When a scan finishes, Startica fetches /robots.txt from the site and works out, for each crawler, whether it may read the site’s start address. It follows the rules in RFC 9309, the standard search engines use:
- Groups. A crawler follows the group that names it (
User-agent: GPTBot, upper or lower case). If no group names it, it follows theUser-agent: *group. SeveralUser-agentlines in a row share one group. - Longest rule wins. Of the
AllowandDisallowrules that match the address, the longest one applies. If anAllowand aDisalloware equally long,Allowwins. - Wildcards.
*matches any characters, and$at the end of a rule means the address must end there. - No file, no limits. If
robots.txtdoesn’t answer with HTTP 200, or no group applies, every crawler is allowed.
This check is separate from the site setting Follow the site’s robots.txt, which decides which pages Startica’s own scanner opens. See Skipped and failed pages.
robots.txt is a request, not a lock. Well-behaved crawlers follow it, and some operators say that fetches made on a user’s request don’t. Startica can tell you what your file asks for, not what each crawler does. Check each operator’s documentation for how its crawlers behave.
Example: allow AI search, block AI training
This robots.txt for fieldnote.example lets search engines and AI search and user crawlers read the whole site, and asks the training crawlers to stay out:
# Search engines and every crawler not named below
User-agent: *
Allow: /
# AI training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# AI search, and pages people ask about: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Sitemap: https://fieldnote.example/sitemap.xmlThe search and user crawlers would be allowed by the * group anyway. Naming them makes the intent clear, and keeps them allowed if you later tighten the * group. With this file, the AI crawlers table shows the four training crawlers as Blocked and the rest as Allowed, AI search access says Open to AI search, and Good to know lists the blocked training crawlers.
llms.txt
llms.txt is an emerging, informal convention: a plain Markdown file at /llms.txt that briefs AI systems on what the site is and where its key pages are. It isn’t a standard, and support varies by AI provider. Some may read it and others may ignore it, so treat it as a cheap extra, not a way to get into AI answers.
Startica checks two things when a scan finishes:
| Result | Shown as | Severity |
|---|---|---|
/llms.txt answers with HTTP 200 and starts with a Markdown # title |
llms.txt: found | None |
It answers, but doesn’t start with a # title |
llms.txt: found, but not in the usual format, and the problem llms.txt doesn’t follow the expected format | low |
| It doesn’t answer with HTTP 200 | llms.txt: none, and the problem No llms.txt | low |
Startica doesn’t check the rest of the file: whether the summary is accurate or the links work.
To get a first version, use Draft llms.txt on the No llms.txt opportunity in Opportunities: the AI writes one from the pages the scan found, and Startica checks its format and links before you copy it.
Example llms.txt
A short file with a title, a one-paragraph summary, and links to the most important pages:
# Fieldnote
> Fieldnote is a note-taking app for field researchers: offline notebooks, photos and GPS tags, synced when you're back online.
## Product
- [Features](https://fieldnote.example/features): offline notebooks, photo and GPS tagging, and sync for field teams
- [Pricing](https://fieldnote.example/pricing): free for one notebook; Pro for unlimited notebooks and team sharing
## Help
- [Getting started](https://fieldnote.example/help/getting-started): set up a notebook and sync it
- [Working offline](https://fieldnote.example/help/offline): what works without a signal, and how sync catches upKeep it short and factual, and update it when your key pages change.
What Startica fetches
At the end of each scan, Startica fetches two files from the site, besides the pages it checks: /robots.txt and /llms.txt, with the user agent StarticaAccessibilityScan/0.1 (+https://mystartica.com; scans sites their owners add). Nothing is fetched for reserved names that never resolve, such as addresses ending in .example, .test or .invalid. See Privacy and security.