How Obaron respects bot controls

Obaron audits public web pages the same way an AI search agent would. We honor the same controls a site uses to manage AI and search bots — including robots.txt, AI-specific directives, and rate-limiting headers. If you don't want us to crawl your site, you can block us in one line.

What we honor

  • robots.txt rules. We obey Disallow rules for both our user-agent (ObaronBot) and the wildcard User-agent: *. If a path is disallowed, we don't fetch it.
  • Crawl-delay headers. If your robots.txt sets a Crawl-delay, we respect it. With or without one, we wait 1.2–1.8 seconds between requests by default.
  • AI-bot directives. If you've blocked the major AI bots (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, and others), we treat that as a signal. Our Score reflects what AI agents would actually see, so blocking them appears in your report — but we don't ignore your stated intent. Blocking ObaronBot directly stops our crawl outright.
  • Rate-limit responses. If your server returns 429 Too Many Requests or 503 Service Unavailable, we back off automatically and respect any Retry-After header you send.
  • Standard HTTP semantics. noindex, nofollow, and canonical links are all factored into the report. Pages that explicitly opt out of indexing are noted as opted-out, not penalized for being uncrawlable.

What we don't do

  • We don't authenticate, log in, or attempt to access private content. We crawl only what's reachable from a public URL.
  • We don't ignore robots.txt. There is no opt-out override, even for the site owner who paid for the audit.
  • We don't run heavy parallel crawls. One request at a time per host, with the delay above.
  • We don't share, resell, or redistribute crawled data. It's used solely to generate your report.

How to block ObaronBot

To prevent ObaronBot from crawling your site, add this to your robots.txt: 

User-agent: ObaronBot
Disallow: /

Our user-agent string is ObaronBot/1.0 (+https://obaron.ai/bot). The about-the-bot page covers the full specification, including how to identify our requests at the server level.

One thing to know: if you've purchased an audit and then block ObaronBot, we can't complete the scan. We'll detect this and issue a full refund — no follow-up needed on your end.

Why this matters

The reason Obaron's audit is useful is that we crawl the same way the AI agents whose crawl behavior determines your AI Readiness Score crawl — and we honor the same controls. That means the Score you see is the Score those bots would see. If your robots.txt is blocking AI agents from your important content, your report will tell you. If it isn't, you'll see what they'd see when they ask about your product.

Questions? Email hi@obaron.ai.