Claude SEO: How Anthropic’s Claude Finds and Cites Websites
Anthropic publishes how Claude’s robots, search and citations work, and says nothing about ranking. Here’s what’s documented, what vendors only claim, and what to check on your own site first.
Claude SEO is making sure Anthropic’s three robots can read your pages, then writing passages short enough to be quoted whole. Anthropic’s pages cover access and citation, not ranking factors.

If you run an MSP or IT company and you’ve asked Claude about your own company and gotten a vague answer, or none, the first instinct is to go hunting for the trick.
There must be a setting. A file. A phrase Claude likes.
Search taught everyone there’s a lever somewhere, and a hundred vendor posts now claim to know where it is. Word counts. Schema lifts. Secret crawler names.
But then you read what Anthropic actually publishes, and the lever hunt falls apart, since what’s documented is narrower and more useful, covering who comes to your door, when Claude decides to look something up, and how much of your page survives the quote. This isn’t a content problem. It’s an access-and-excerpt problem.
The Short Version
Claude SEO comes down to three questions. Can Claude reach the page, will it go looking for what the page says, and will the passage still make sense when it’s cut down to a quote? Anthropic documents the first and parts of the other two, and says nothing on ranking, which is why this guide keeps the documented separate from the claimed. The cross-assistant view is in our guide to LLM optimization, the ChatGPT version is ChatGPT SEO (or our ChatGPT SEO services if you want it run for you), and the service behind all of it sits on our page for generative engine optimization services for tech companies. To test one page against the three Anthropic robots and the rest, run our AEO checklist for MSP and IT company pages.
What Is Claude SEO, and What Does Anthropic Actually Document?
Claude SEO is the work of making a site readable, findable and quotable by Anthropic’s Claude, which searches the web, fetches pages and cites what it used. It’s not a separate discipline from your other search work. It’s one assistant’s rules for the door, the lookup and the quote.
Sort the claims first. Searching for this topic turns up vendor posts, nearly all of them, and they blend three different things together. What Anthropic states, what Anthropic states more narrowly than the vendors do, and what nobody has published at all.
Documented, Narrower, or Not Documented?
| The common claim | Where it stands | What the Anthropic pages say |
|---|---|---|
| Claude has one crawler | Narrower | Three robots with separate jobs: ClaudeBot, Claude-User and Claude-SearchBot |
| Blocking ClaudeBot removes you from Claude’s answers | Not what the page says | ClaudeBot is the training robot. Search indexing and user-directed fetching belong to the other two |
| Claude cites its sources | Documented | The help center says every response includes citations, and the API’s web search tool has citations always on |
| Claude reads any page you give it | Narrower | The web fetch tool does not support JavaScript-rendered sites and handles only text, HTML and PDF |
| Claude’s text results come from one named search engine | Not documented | No page used here names it. The help center says only that image results are powered by Bing |
| Claude favors a page length, format or markup | Not documented | None of the pages gives one |
| Claude follows an llms.txt file | Not documented | The crawler and web search pages never mention one |
The table draws on five Anthropic pages, read on November 17, 2026. Anthropic updates them, so check the date on each before you act on a detail. Every row marked “not documented” is a gap in what Anthropic has said, not proof the thing doesn’t matter.
How Claude Reaches Your Pages: Three Robots and a Browser
Anthropic’s crawler page says it uses different robots to gather data for model development, to search the web, and to retrieve content at a user’s direction. Three jobs. Three separate switches.
Robots, Jobs and Switches
| Robot | Its job, per Anthropic | What Anthropic says happens when you disable it |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Signals that your future materials should be excluded from training datasets |
| Claude-SearchBot | Navigates the web to improve search result quality | Stops indexing your content for search, which may reduce your visibility and accuracy in results |
| Claude-User | Fetches a site when a person’s question to Claude calls for it | Stops retrieval in response to a user query, which may reduce visibility for user-directed search |
| Claude-Agent (a hosted browser) | A browser Anthropic hosts so Claude can complete tasks a user asks for | Controls whether users can ask Claude services to complete tasks on your site |
Our read for a tech company is short. ClaudeBot is a training decision, and the rule for it is in should you block AI crawlers. Keep Claude-SearchBot and Claude-User open if being found is the point, since Anthropic says disabling either may reduce visibility. Claude-Agent is its own call.
Three details sit in that Anthropic page and get lost in the vendor posts, starting with the fact that the robots honor robots.txt, support the non-standard Crawl-delay line, and don’t try to get around CAPTCHAs. Anthropic also warns that blocking by IP address may not give you a dependable opt-out, because it can keep the robot from reading your robots.txt in the first place.
So the control is one file. A group per robot, a User-agent line naming it, and then Allow: / or Disallow: /. Anthropic says the file has to be set on every subdomain you want to opt out from, so check each one.
The browser is a different door
The hosted browser carries the token Claude-Agent inside an otherwise normal browser user-agent string. Anthropic’s page on hosted browsers says to match on the token and not the full string, because the rest changes. Anyone can fake the header, it adds. So check the visit’s IP against Anthropic’s published list before you trust it.
The October 30, 2026 training signal
Anthropic’s crawler page says that as of October 30, 2026 it began recognizing a machine-readable signal for “no AI training”, following a draft from the IETF’s aipref working group. Where your robots.txt names ClaudeBot, Anthropic says it follows that rule even if the general setting differs. It also says standards here are still evolving, so the way it reads these settings may change. That’s about training, not search.
None of this makes you visible. It only stops you from being invisible by accident, which is a smaller win and a very common one.
How Claude Decides to Look Something Up
Having the door open doesn’t mean Claude walks through it. Claude chooses.
Search is a judgment call
Per the API’s web search tool documentation, Claude searches when a request depends on information that is current, changing or outside its training data, and answers directly for stable knowledge, math, creative work and anything already in the conversation. The help center adds that in the newer Claude experience there’s no web search toggle. Claude searches the web when it helps.
What does that mean for a tech company? Questions a buyer asks about a market, a vendor list or a current price lean toward a lookup. A question Claude can answer from memory doesn’t send it anywhere near your site. No lookup, no visit.
Fetching needs a URL
Fetching is the other path. Anthropic’s web fetch documentation says Claude fetches when the request points at a specific page, such as a URL in the conversation, or when a user names a specific resource and search is on so Claude can locate it first. It can only fetch URLs that already appeared in the conversation, including ones that came back from an earlier search.
A buyer pasting your pricing page into Claude is a fetch. A buyer asking who offers managed IT in Denver is a search. You want to survive both.
JavaScript is a wall
This one surprises people. The web fetch tool, in Anthropic’s words, currently does not support websites dynamically rendered with JavaScript, and it handles only text, HTML and PDF content. Its error list also includes a URL blocked by robots.txt.
It’s documented for that API tool. Anthropic’s pages don’t say how claude.ai itself treats script-heavy pages, so treat it as a risk, not a verdict. If your service copy loads after a script runs, view the page source and see whether the answer is in it. If it isn’t, fix that before anything else on this list.
An administrator can narrow the field
In the API, web search and web fetch take allowed and blocked domain lists, and the docs say an organization administrator can restrict which domains the tools use from the Claude Console. You can’t see or change someone else’s list. It’s a fair reason to be the plainly named, easily verified source, since a narrowed field favors names the buyer already trusts. That last part is our inference, not Anthropic’s claim.
What Claude Quotes, and How to Write for a 150-Character Window
Claude’s answers carry citations. In the API, each web search citation includes the URL, the title and cited text of up to 150 characters, and each search result carries a page_age field Anthropic describes as when the site was last updated. In claude.ai, the help center says responses include direct citations, source links and relevant quotes when appropriate.

150 characters is about one short sentence. That’s the quote that travels.
Writing for that is like recording a voicemail greeting. Nobody hears the second paragraph. If your service page opens with three sentences of scene-setting and puts the answer in the fourth, the quote will be the scene-setting.
Here’s what we’d do. This is C4’s working view, not an Anthropic rule, and no page we read promises it moves a citation.
- Open each page that should win a question with one standalone sentence that answers it, with the topic named in the sentence itself.
- Name the real things. The company, the service, the city, the platform. A sentence that says “our solution” can’t be quoted usefully.
- Show a true date. Search results carry a page age, and a page that says it was updated this month had better have been.
- Skip the filler. If a sentence still reads fine with the second half cut off, you’re close.
This is the same rule behind how we write every answer page, which we explain in AEO versus GEO. Claude gives it a number.
What Nobody Has Published, and What Vendors Sell Instead
Not one of the five Anthropic pages gives a ranking factor. No weights, no ideal length, no preferred schema type, no list of signals. Plenty of posts offer “the five signals Claude weights” anyway. They sound confident. They’re measurements of someone’s own sample, dressed as rules.
Treat those the way you’d treat an unsourced salary figure. Interesting, possibly right, and not something to build a budget on. Disconnected tactics don’t compound.
One thing is worth doing even though Anthropic is silent on it, because it holds across assistants, so make sure other pages confirm who you are and what you do, in the same words. We cover that in brand mentions and AI search. And since the help center ties image results to Bing, the Microsoft side is worth a read too, in our guide to Bing and Copilot SEO for MSPs.
A Claude SEO Check for a Tech Company, in Order
The order matters, because each step is pointless if the one before it fails. Skip one and you’re guessing. A closed door beats a clever page every time.
- Read your robots.txt once per robot. Look for a blanket rule, a copied block list, or a plugin that added a Claude name when nobody was looking. Decide ClaudeBot on its own terms, and keep Claude-SearchBot and Claude-User open if being found is the point.
- View the source of your service pages, not the inspector. If the answer isn’t in the raw HTML, the fetch tool’s JavaScript limit means it may not be there for Claude either.
- Rewrite the first sentence of each page that should win a question so it stands alone.
- Check that every date on the page is true, and that old pages say so.
- Look for the tokens in your server logs, covered in the next section.
- Run the prompt test once a month, and write down what comes back.
Step one takes ten minutes. It’s also the step with the biggest downside if it’s wrong, which is why it’s first.
How to Test Whether Claude Can Find You
You can’t see Claude’s index. So you test from the outside and keep notes.
Start with the logs. Search your server or CDN logs for the tokens ClaudeBot, Claude-SearchBot, Claude-User and Claude-Agent in the user-agent field. Seeing them means the door works. Not seeing them proves little, because Claude only fetches when a question calls for it.
Then run prompts. The help center says that including “Search the web” or “Use web search” in your prompt makes sure Claude uses it, so the test doesn’t depend on whether it would have chosen to. Ask ten questions a buyer would ask, name no vendors, and record which sources Claude cites and whether your company shows up. Answers shift between runs, so one result tells you nothing and a month of results starts to.
Last, look in your analytics for visits from claude.ai. Anthropic doesn’t document how referrals appear, so treat any count as a floor and never as the total. The wider setup is in our guide to measuring AI search visibility, and if you’d rather have it done for you, our AI visibility audit runs the same checks across assistants.
Where Claude SEO Stops
Claude is one assistant. Your buyers also use ChatGPT, Gemini, Perplexity, Copilot and plain Google, and the work that earns a quote in one usually earns it in the others.
So don’t build a Claude project on its own. Fix the door, fix the first sentences, tell the truth about dates, and test monthly, then spread the same work across the others using LLM optimization as the map. Unglamorous? Yes. It’s also the same advice we’d give if Claude didn’t exist.
Want help with the checklist? Book a growth strategy call.
What Owners Ask About Claude
Which search engine feeds Claude’s answers?
Anthropic’s pages used for this guide don’t name one for text results.
They do say Claude searches the web when it helps and that image results are powered by Bing. Vendor posts name specific engines with total confidence, and none of them cites an Anthropic page that says so. Plan around the documented parts instead.
Does blocking ClaudeBot hide us from Claude’s answers?
Not on Anthropic’s own description, which ties ClaudeBot to model training and gives search indexing and user-directed retrieval to Claude-SearchBot and Claude-User.
Disable those two, though, and Anthropic says your visibility may be reduced. The robot-by-robot decision is laid out in our crawler guide.
Do we need an llms.txt file for Claude?
Nothing on the Anthropic pages used here mentions one.
That’s a statement about what’s documented, not a verdict. If you already keep one, it costs little, and our walkthrough on building an llms.txt file covers it. Just don’t count on it as the reason you get cited.
How long is the quote Claude pulls from a page?
Up to 150 characters per citation in the API’s web search tool.
For claude.ai, the help center says responses include direct citations, source links and relevant quotes when appropriate, and gives no length. So write the first sentence of each section as if it might be the only sentence anyone reads.
Will a JavaScript-heavy site still get read?
Test it first.
Anthropic’s web fetch tool says it does not support JavaScript-rendered sites. It’s documented for that tool and not for claude.ai as a whole, so check your page source, and if the answer only appears after scripts run, move it into the HTML.
Is Claude worth a project of its own?
Probably not. The part that’s Claude-specific fits in an afternoon.
That’s the robots.txt review, the source-view check and a monthly prompt test. The rest is the work that helps in every assistant. Our ChatGPT SEO guide shows what carries over.
Find out whether Claude can reach and quote your pages
Talk to us about your robots.txt, your page source and the first sentence of your key pages, so Claude can read what you publish and quote it cleanly.
Book a Growth Strategy Call