Free Growth Assessment
SEO & AEO

Claude SEO: How Anthropic’s Claude Finds and Cites Websites

Anthropic publishes how Claude’s robots, search and citations work, and says nothing about ranking. Here’s what’s documented, what vendors only claim, and what to check on your own site first.

By Holly Mack Last updated: November 17, 2026 11 min read

Claude SEO is making sure Anthropic’s three robots can read your pages, then writing passages short enough to be quoted whole. Anthropic’s pages cover access and citation, not ranking factors.

A concrete corridor with three steel doors, the nearest fully open with blue light spilling out, the middle ajar, the last shut
Each of Anthropic's robots is a separate door, and you set each one on its own.

If you run an MSP or IT company and you’ve asked Claude about your own company and gotten a vague answer, or none, the first instinct is to go hunting for the trick.

There must be a setting. A file. A phrase Claude likes.

Search taught everyone there’s a lever somewhere, and a hundred vendor posts now claim to know where it is. Word counts. Schema lifts. Secret crawler names.

But then you read what Anthropic actually publishes, and the lever hunt falls apart, since what’s documented is narrower and more useful, covering who comes to your door, when Claude decides to look something up, and how much of your page survives the quote. This isn’t a content problem. It’s an access-and-excerpt problem.

Key Takeaways

The Short Version

Claude SEO comes down to three questions. Can Claude reach the page, will it go looking for what the page says, and will the passage still make sense when it’s cut down to a quote? Anthropic documents the first and parts of the other two, and says nothing on ranking, which is why this guide keeps the documented separate from the claimed. The cross-assistant view is in our guide to LLM optimization, the ChatGPT version is ChatGPT SEO (or our ChatGPT SEO services if you want it run for you), and the service behind all of it sits on our page for generative engine optimization services for tech companies. To test one page against the three Anthropic robots and the rest, run our AEO checklist for MSP and IT company pages.

What Is Claude SEO, and What Does Anthropic Actually Document?

Claude SEO is the work of making a site readable, findable and quotable by Anthropic’s Claude, which searches the web, fetches pages and cites what it used. It’s not a separate discipline from your other search work. It’s one assistant’s rules for the door, the lookup and the quote.

Sort the claims first. Searching for this topic turns up vendor posts, nearly all of them, and they blend three different things together. What Anthropic states, what Anthropic states more narrowly than the vendors do, and what nobody has published at all.

Documented, Narrower, or Not Documented?

The common claimWhere it standsWhat the Anthropic pages say
Claude has one crawlerNarrowerThree robots with separate jobs: ClaudeBot, Claude-User and Claude-SearchBot
Blocking ClaudeBot removes you from Claude’s answersNot what the page saysClaudeBot is the training robot. Search indexing and user-directed fetching belong to the other two
Claude cites its sourcesDocumentedThe help center says every response includes citations, and the API’s web search tool has citations always on
Claude reads any page you give itNarrowerThe web fetch tool does not support JavaScript-rendered sites and handles only text, HTML and PDF
Claude’s text results come from one named search engineNot documentedNo page used here names it. The help center says only that image results are powered by Bing
Claude favors a page length, format or markupNot documentedNone of the pages gives one
Claude follows an llms.txt fileNot documentedThe crawler and web search pages never mention one

The table draws on five Anthropic pages, read on November 17, 2026. Anthropic updates them, so check the date on each before you act on a detail. Every row marked “not documented” is a gap in what Anthropic has said, not proof the thing doesn’t matter.

How Claude Reaches Your Pages: Three Robots and a Browser

Anthropic’s crawler page says it uses different robots to gather data for model development, to search the web, and to retrieve content at a user’s direction. Three jobs. Three separate switches.

Robots, Jobs and Switches

RobotIts job, per AnthropicWhat Anthropic says happens when you disable it
ClaudeBotCollects web content that could contribute to model trainingSignals that your future materials should be excluded from training datasets
Claude-SearchBotNavigates the web to improve search result qualityStops indexing your content for search, which may reduce your visibility and accuracy in results
Claude-UserFetches a site when a person’s question to Claude calls for itStops retrieval in response to a user query, which may reduce visibility for user-directed search
Claude-Agent (a hosted browser)A browser Anthropic hosts so Claude can complete tasks a user asks forControls whether users can ask Claude services to complete tasks on your site

Our read for a tech company is short. ClaudeBot is a training decision, and the rule for it is in should you block AI crawlers. Keep Claude-SearchBot and Claude-User open if being found is the point, since Anthropic says disabling either may reduce visibility. Claude-Agent is its own call.

Three details sit in that Anthropic page and get lost in the vendor posts, starting with the fact that the robots honor robots.txt, support the non-standard Crawl-delay line, and don’t try to get around CAPTCHAs. Anthropic also warns that blocking by IP address may not give you a dependable opt-out, because it can keep the robot from reading your robots.txt in the first place.

So the control is one file. A group per robot, a User-agent line naming it, and then Allow: / or Disallow: /. Anthropic says the file has to be set on every subdomain you want to opt out from, so check each one.

The browser is a different door

The hosted browser carries the token Claude-Agent inside an otherwise normal browser user-agent string. Anthropic’s page on hosted browsers says to match on the token and not the full string, because the rest changes. Anyone can fake the header, it adds. So check the visit’s IP against Anthropic’s published list before you trust it.

The October 30, 2026 training signal

Anthropic’s crawler page says that as of October 30, 2026 it began recognizing a machine-readable signal for “no AI training”, following a draft from the IETF’s aipref working group. Where your robots.txt names ClaudeBot, Anthropic says it follows that rule even if the general setting differs. It also says standards here are still evolving, so the way it reads these settings may change. That’s about training, not search.

None of this makes you visible. It only stops you from being invisible by accident, which is a smaller win and a very common one.

How Claude Decides to Look Something Up

Having the door open doesn’t mean Claude walks through it. Claude chooses.

Search is a judgment call

Per the API’s web search tool documentation, Claude searches when a request depends on information that is current, changing or outside its training data, and answers directly for stable knowledge, math, creative work and anything already in the conversation. The help center adds that in the newer Claude experience there’s no web search toggle. Claude searches the web when it helps.

What does that mean for a tech company? Questions a buyer asks about a market, a vendor list or a current price lean toward a lookup. A question Claude can answer from memory doesn’t send it anywhere near your site. No lookup, no visit.

Fetching needs a URL

Fetching is the other path. Anthropic’s web fetch documentation says Claude fetches when the request points at a specific page, such as a URL in the conversation, or when a user names a specific resource and search is on so Claude can locate it first. It can only fetch URLs that already appeared in the conversation, including ones that came back from an earlier search.

A buyer pasting your pricing page into Claude is a fetch. A buyer asking who offers managed IT in Denver is a search. You want to survive both.

JavaScript is a wall

This one surprises people. The web fetch tool, in Anthropic’s words, currently does not support websites dynamically rendered with JavaScript, and it handles only text, HTML and PDF content. Its error list also includes a URL blocked by robots.txt.

It’s documented for that API tool. Anthropic’s pages don’t say how claude.ai itself treats script-heavy pages, so treat it as a risk, not a verdict. If your service copy loads after a script runs, view the page source and see whether the answer is in it. If it isn’t, fix that before anything else on this list.

An administrator can narrow the field

In the API, web search and web fetch take allowed and blocked domain lists, and the docs say an organization administrator can restrict which domains the tools use from the Claude Console. You can’t see or change someone else’s list. It’s a fair reason to be the plainly named, easily verified source, since a narrowed field favors names the buyer already trusts. That last part is our inference, not Anthropic’s claim.

What Claude Quotes, and How to Write for a 150-Character Window

Claude’s answers carry citations. In the API, each web search citation includes the URL, the title and cited text of up to 150 characters, and each search result carries a page_age field Anthropic describes as when the site was last updated. In claude.ai, the help center says responses include direct citations, source links and relevant quotes when appropriate.

Two hands setting a short blank paper strip onto a larger blank sheet beside a steel ruler on a dark steel table
The quote that travels is a short strip of the page, not the whole sheet.

150 characters is about one short sentence. That’s the quote that travels.

Writing for that is like recording a voicemail greeting. Nobody hears the second paragraph. If your service page opens with three sentences of scene-setting and puts the answer in the fourth, the quote will be the scene-setting.

Here’s what we’d do. This is C4’s working view, not an Anthropic rule, and no page we read promises it moves a citation.

  • Open each page that should win a question with one standalone sentence that answers it, with the topic named in the sentence itself.
  • Name the real things. The company, the service, the city, the platform. A sentence that says “our solution” can’t be quoted usefully.
  • Show a true date. Search results carry a page age, and a page that says it was updated this month had better have been.
  • Skip the filler. If a sentence still reads fine with the second half cut off, you’re close.

This is the same rule behind how we write every answer page, which we explain in AEO versus GEO. Claude gives it a number.

What Nobody Has Published, and What Vendors Sell Instead

Not one of the five Anthropic pages gives a ranking factor. No weights, no ideal length, no preferred schema type, no list of signals. Plenty of posts offer “the five signals Claude weights” anyway. They sound confident. They’re measurements of someone’s own sample, dressed as rules.

Treat those the way you’d treat an unsourced salary figure. Interesting, possibly right, and not something to build a budget on. Disconnected tactics don’t compound.

One thing is worth doing even though Anthropic is silent on it, because it holds across assistants, so make sure other pages confirm who you are and what you do, in the same words. We cover that in brand mentions and AI search. And since the help center ties image results to Bing, the Microsoft side is worth a read too, in our guide to Bing and Copilot SEO for MSPs.

A Claude SEO Check for a Tech Company, in Order

The order matters, because each step is pointless if the one before it fails. Skip one and you’re guessing. A closed door beats a clever page every time.

  1. Read your robots.txt once per robot. Look for a blanket rule, a copied block list, or a plugin that added a Claude name when nobody was looking. Decide ClaudeBot on its own terms, and keep Claude-SearchBot and Claude-User open if being found is the point.
  2. View the source of your service pages, not the inspector. If the answer isn’t in the raw HTML, the fetch tool’s JavaScript limit means it may not be there for Claude either.
  3. Rewrite the first sentence of each page that should win a question so it stands alone.
  4. Check that every date on the page is true, and that old pages say so.
  5. Look for the tokens in your server logs, covered in the next section.
  6. Run the prompt test once a month, and write down what comes back.

Step one takes ten minutes. It’s also the step with the biggest downside if it’s wrong, which is why it’s first.

How to Test Whether Claude Can Find You

You can’t see Claude’s index. So you test from the outside and keep notes.

Start with the logs. Search your server or CDN logs for the tokens ClaudeBot, Claude-SearchBot, Claude-User and Claude-Agent in the user-agent field. Seeing them means the door works. Not seeing them proves little, because Claude only fetches when a question calls for it.

Then run prompts. The help center says that including “Search the web” or “Use web search” in your prompt makes sure Claude uses it, so the test doesn’t depend on whether it would have chosen to. Ask ten questions a buyer would ask, name no vendors, and record which sources Claude cites and whether your company shows up. Answers shift between runs, so one result tells you nothing and a month of results starts to.

Last, look in your analytics for visits from claude.ai. Anthropic doesn’t document how referrals appear, so treat any count as a floor and never as the total. The wider setup is in our guide to measuring AI search visibility, and if you’d rather have it done for you, our AI visibility audit runs the same checks across assistants.

Where Claude SEO Stops

Claude is one assistant. Your buyers also use ChatGPT, Gemini, Perplexity, Copilot and plain Google, and the work that earns a quote in one usually earns it in the others.

So don’t build a Claude project on its own. Fix the door, fix the first sentences, tell the truth about dates, and test monthly, then spread the same work across the others using LLM optimization as the map. Unglamorous? Yes. It’s also the same advice we’d give if Claude didn’t exist.

Want help with the checklist? Book a growth strategy call.

What Owners Ask About Claude

Which search engine feeds Claude’s answers?

Anthropic’s pages used for this guide don’t name one for text results.

They do say Claude searches the web when it helps and that image results are powered by Bing. Vendor posts name specific engines with total confidence, and none of them cites an Anthropic page that says so. Plan around the documented parts instead.

Does blocking ClaudeBot hide us from Claude’s answers?

Not on Anthropic’s own description, which ties ClaudeBot to model training and gives search indexing and user-directed retrieval to Claude-SearchBot and Claude-User.

Disable those two, though, and Anthropic says your visibility may be reduced. The robot-by-robot decision is laid out in our crawler guide.

Do we need an llms.txt file for Claude?

Nothing on the Anthropic pages used here mentions one.

That’s a statement about what’s documented, not a verdict. If you already keep one, it costs little, and our walkthrough on building an llms.txt file covers it. Just don’t count on it as the reason you get cited.

How long is the quote Claude pulls from a page?

Up to 150 characters per citation in the API’s web search tool.

For claude.ai, the help center says responses include direct citations, source links and relevant quotes when appropriate, and gives no length. So write the first sentence of each section as if it might be the only sentence anyone reads.

Will a JavaScript-heavy site still get read?

Test it first.

Anthropic’s web fetch tool says it does not support JavaScript-rendered sites. It’s documented for that tool and not for claude.ai as a whole, so check your page source, and if the answer only appears after scripts run, move it into the HTML.

Is Claude worth a project of its own?

Probably not. The part that’s Claude-specific fits in an afternoon.

That’s the robots.txt review, the source-view check and a monthly prompt test. The rest is the work that helps in every assistant. Our ChatGPT SEO guide shows what carries over.

READY TO ACT?

Find out whether Claude can reach and quote your pages

Talk to us about your robots.txt, your page source and the first sentence of your key pages, so Claude can read what you publish and quote it cleanly.

Book a Growth Strategy Call