How to get your site cited by Claude

Quick summary

• Claude has web search on every plan, and when it uses information from the web it provides direct citations to the sources.
• It doesn't always search. According to Anthropic's documentation, it searches when a request depends on information that's current, changing or outside its training data: prices, news, details about specific companies or products. If it answers from memory, there's no citation.
• Anthropic runs three crawlers: ClaudeBot (training), Claude-SearchBot (search index) and Claude-User (user-initiated fetches). Blocking the last two may reduce your visibility in answers.
• Unlike some others, Anthropic says all of its bots honor robots.txt, including the one acting on a user's behalf.

How Claude's search works

Anthropic launched web search for Claude in 2025 and, according to its announcement, made it available globally on all plans on 27 May 2025. The idea is the same as any assistant with search: instead of a list of results, Claude processes the sources and answers conversationally, citing them so you can fact-check.

The technical documentation for the web search tool describes the process in three steps:

1. Claude decides whether to search based on the prompt.
2. The searches run and Claude gets the results. This can repeat several times within a single response.
3. Claude answers with cited sources. In the API, citations are always on for web search and include the URL, the title and the cited text.

When it searches, and when it doesn't

This is the part most people miss. Per the same documentation, Claude searches when a request depends on:

• Recent events, news or announcements.
• Current prices, rates, scores or statistics.
• Information about specific organizations, people or products that might have changed.
• Explicit requests to look something up.

It answers without searching for stable knowledge: established facts, concepts, creative writing, or analysis of content already in the conversation.

For a business, that means buyer questions ("what invoicing software would you recommend for freelancers in the UK?", "how much does X cost?") are exactly the ones most likely to trigger a search. Generic questions ("what is e-invoicing?") may be answered from memory, linking to nobody.

Anthropic's three crawlers

Anthropic describes them in its help center, along with what happens if you block each one:

Bot	What it's for	If you block it
ClaudeBot	Collecting content that could contribute to model training	Your future content is excluded from training datasets
Claude-SearchBot	Navigating the web to improve search result quality	Your content isn't indexed for search; may reduce visibility
Claude-User	Accessing sites when a user asks Claude a question	Claude can't retrieve your content for that answer; may reduce visibility

Three useful details from the same page:

• All of them honor robots.txt. Anthropic says its bots respect standard robots.txt directives. That's a difference from OpenAI and Perplexity, which warn that their user-initiated agents may not apply robots.txt.
• It supports Crawl-delay, a non-standard extension, if load is a concern.
• Blocking by IP isn't reliable: according to Anthropic, it stops its bots from reading your robots.txt and may not persist. To opt out, use robots.txt, on every subdomain.

The decision you need to make

You may not want your content used for training while still wanting to appear in answers. With Anthropic that's possible: block ClaudeBot, allow Claude-SearchBot and Claude-User.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Watch for robots.txt files inherited from old templates that block anthropic-ai or Claude-Web, or blanket User-agent: * rules with Disallow: /. Check what each bot sees with the AI crawler checker.

Which pages stand the best chance

Anthropic doesn't publish how it chooses among results. What you can reason from how it works:

Pages that answer search-triggering questions

If Claude searches mainly for prices, news and details about specific companies or products, pages that contain that information, current and easy to find are the natural candidates. A clear pricing page, a product page with specs, a dated comparison, a case study with numbers.

Text that can be quoted

The API returns the cited text along with the URL. A paragraph that holds the full answer (what, how much, for whom, since when) is easier to cite than an idea scattered across five design blocks.

Consistent facts everywhere

If your site says one price, your directory listing another and a third-party comparison a third, the model has to pick. Keeping the basics consistent (name, what you do, pricing, markets) lowers the risk of being described wrongly.

Presence on the sources already cited

As with the other engines, third-party sources weigh heavily for "best X" questions. Look at which URLs Claude cites when you're missing and ask whether you could legitimately be there. More in why AI cites your competitors instead of you.

A five-minute checklist

• robots.txt allows Claude-SearchBot and Claude-User (and doesn't catch them with a blanket Disallow: /).
• Your pricing, product and comparison pages are current and dated, because those are the questions that trigger a search.
• The key facts about your company (what you do, for whom, where, how much) are stated in one place, in plain text, not spread across images or tabs.
• Those facts match what your directory listings and profiles say.
• You know which third-party pages Claude cites for your buyer questions, and whether you're on them.

How to measure it

Ask your buyer questions in Claude with search on, in new conversations, and repeat. Record whether you're mentioned, whether you're cited and which URLs it uses. Visits from claude.ai carry that domain as the referrer and can be filtered by source in Analytics.

Citable measures Claude on its top plan. It does so through the model's API with the web search tool, not the consumer app, and once a week rather than three times, because every Claude answer runs real web searches and costs more per sample. We explain it in the methodology; what each plan includes is on the pricing page.

FAQ

Does Claude always cite its sources?

When it uses information from the web, yes: Anthropic says it provides direct citations. But it doesn't search for every question. If it answers from its own knowledge, for example a conceptual question, there are no sources to cite.

If I block ClaudeBot, do I disappear from Claude?

Not necessarily. ClaudeBot is the training crawler. For search, Claude-SearchBot and Claude-User are what count. Anthropic says blocking those two may reduce your visibility in answers.

Does Claude-User respect robots.txt?

According to Anthropic, yes: all its bots honor robots.txt directives, and Claude-User lets site owners control which sites can be accessed through user-initiated requests.

Can I pay to be recommended by Claude?

Anthropic offers no documented way to pay to appear in Claude's answers. What you can do is allow its search crawlers and publish clear, current, verifiable information.

Conclusion

For Claude to cite you, three things need to happen: the question triggers a search, its search crawlers can read you, and your page holds the answer in quotable text. The first depends on the question; the second and third are on you. Check your robots.txt, decide what you want to do about training, and measure with repeated runs.