# Agentry is built for AI agents and LLM-driven retrieval. Maximum # crawler compatibility: every well-behaved bot — including training # crawlers and retrieval/grounding agents — is allowed to access # everything under /. # # IMPORTANT — Cloudflare zone setting: # Cloudflare auto-prepends a "Block AI bots" block (with # `Content-Signal: ai-train=no` and `Disallow: /` for the major AI # crawlers) when its AI bot management is enabled. That setting # overrides this file at the edge and is what triggers Lighthouse's # "Unknown directive" robots.txt error on `Content-Signal:`. # To make this file authoritative again: # Cloudflare dashboard → agentry.sh → Security → Bots # → AI Crawl Control / Block AI bots → OFF # (also turn off "Content Signals Policy" if listed separately). User-agent: * Allow: / # Markdown alternates for agent-discoverable content # (also advertised via in HTML ) Allow: /install.md Allow: /agentry.md # Belt-and-braces: explicit Allow for every major LLM training + # retrieval crawler. Redundant with `User-agent: *` above, but if a # crawler matches a more specific User-agent line it ignores the # wildcard block — listing them prevents any future managed-content # disallow from accidentally winning the longest-match. User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / User-agent: GoogleOther Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: Cohere-AI Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: / User-agent: FacebookBot Allow: / User-agent: Diffbot Allow: / User-agent: YouBot Allow: / User-agent: Timpibot Allow: / User-agent: omgili Allow: / User-agent: ImagesiftBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Kagibot Allow: / User-agent: MistralAI-User Allow: / Sitemap: https://agentry.sh/sitemap.xml