AI Readiness Methodology

AI Readiness = how well AI systems can understand, retrieve, cite, and act on your content.

The 8-category rubric

The AI Readiness Score is a 0–100 deterministic measurement summed from eight categories, each with a weight that reflects its contribution to whether AI systems can understand, retrieve, cite, and act on the content. Categories appear below in priority order — most load-bearing first.

AI Crawlability & Access — 16 points

Whether AI crawlers can reach the site at all.

What this category covers

robots.txt rules for the major AI crawlers (GPTBot, ClaudeBot, CCBot, OAI-SearchBot, Googlebot), HTTP status code on the homepage and sampled inner pages, TLS health on apex and www variants, and redirect behavior. The category includes the meta-robots directive on each page, since a noindex tag is functionally equivalent to a robots.txt block at the page level.

Why it matters

This is the catastrophic-failure gate. A site blocking AI crawlers scores 0 on every other category by definition — pages that cannot be fetched cannot be parsed, structured, or cited. The universal all-AI-bots-blocked hard cap (below) reflects this: a blocked site cannot exceed 40 regardless of all other category strength.

Hard caps that apply

All AI bots blocked → score capped at 40. When robots.txt blocks every monitored AI bot from the site root, the Score cannot exceed 40 regardless of other category strength.

Documentation Patterns — 18 points

Docs-native signals AI engines look for when retrieving documentation as documentation.

What this category covers

Schema.org markup specific to documentation — TechArticle, APIReference, and HowTo types — applied at the page level. Code-block markup using semantic <pre><code> patterns or class-based syntax-highlighter conventions (class="language-*"). And the agent-metadata trio: llms.txt, agents.md, and .well-known/mcp.json — three emerging standards that signal a site has been authored with AI agents and Model Context Protocol clients in mind. Each file is validated for structural integrity and earns graduated credit — partial credit drives remediation rather than binary pass/fail. Obaron identifies the docs platform powering the site (Mintlify, Docusaurus, VitePress, GitBook, ReadMe) as contextual metadata — it does not affect the Score.

Why it matters

This is the docs-vertical differentiator. AI engines retrieving documentation lean on TechArticle and APIReference schema to distinguish reference material from blog posts, and read code-block markup to extract examples cleanly. The concentration of weight on docs-native signals is what makes a Docs Readiness Audit a docs audit rather than a generic AEO scan.

Hard caps that apply

Agent-metadata trio absent → score capped at 75. When llms.txt, agents.md, and the .well-known/mcp.json file are all absent on a site claiming docs-vertical positioning, the Score cannot exceed 75. The trio is low-effort to add, which is why the cap sits relatively high — it's a soft signal, not catastrophic. The cap evaluates HTTP-layer presence, not content quality: a present-but-malformed file still counts as present (it earns partial credit on the content-quality side instead of triggering the cap).

Docs schema missing on >50% of doc pages → score capped at 65. Applies to the paid Docs Readiness Audit only, since the trigger requires multi-page evidence the free Lightning Scan doesn't collect. When TechArticle, APIReference, or HowTo schema is missing on more than half of the sampled documentation pages, the Score caps at 65 — the audit is grading a site that hosts documentation rather than a site authored as documentation.

Auth-walled docs → score capped at 50. Paid-only. When more than half of sampled doc pages return HTTP 401 or 403 to ObaronBot, the Score caps at 50 — the rest of the rubric measures readiness against unreachable content.

Platform identification (metadata, not scoring)

Obaron also identifies which docs platform powers the site — Mintlify, Docusaurus, VitePress, GitBook, ReadMe, or none-detected. Platform identification is metadata: it does not affect the Score. A site running stock Mintlify and a site running custom-built docs are scored on the same rubric — penalizing custom-built docs for not using a recognized platform would systematically miscount excellent bespoke docs as worse than recognized-platform docs (Stripe Docs, for instance, is bespoke and excellent). The detection threads the platform name into report copy ("your Mintlify docs") and into the AI Readiness Score breakdown for richer context. Future Fix Kit remediation will use the detected platform to tailor fix advice; the present Audit reports the detection as informational. Detection runs at three confidence levels: high (unique signature present), medium (multiple corroborating signals), or low (filtered from customer-facing copy but preserved in scan data for downstream use).

Structured Data — 13 points

Schema.org markup beyond docs-specific types — the structured layer AI engines parse for direct answers.

What this category covers

JSON-LD presence, required-field coverage on high-value schema types (Article, FAQPage, Organization, BlogPosting, NewsArticle), schema breadth across the sampled URL set, and validity of the JSON-LD structures themselves. Documentation-specific schema types (TechArticle, APIReference, HowTo) are scored separately under Documentation Patterns; this category covers the remainder of the structured-data layer that applies regardless of vertical.

Why it matters

Structured data is how answer engines understand what each page is about without reading its prose. FAQPage schema with a clean Question/Answer hierarchy gives the engine an extractable structure that does not require inference; a page with no schema forces inference. The audit measures presence first, then required-field quality, then schema breadth — each layer compounds.

AEO Readiness — 12 points

Patterns that let AI engines extract direct answers — question-style headings, concise answers, list and table structure.

What this category covers

Headings phrased as questions (How do I…?, What is…?) versus headings phrased as labels (Authentication, Endpoints). The brevity and placement of the answers that follow question-style headings. The presence of lists and tables as scannable structures. The sizing of meta descriptions to land in the 50–160 character range AI engines preview cleanly. And the <html lang="…"> attribute that tells AI engines what language to process the content as.

Why it matters

AEO — Answer Engine Optimization — is the practice of writing content that AI engines can extract direct answers from, not just retrieve and summarize. The single highest-leverage AEO signal in the rubric is question-style headings: a page with What is OAuth? as an H2 followed by a tight 30-word definition is roughly an order of magnitude easier for an AI engine to use than a page with Authentication as an H2 followed by a 500-word essay. The category measures whether the content has been authored with AI extraction in mind — not just SEO retrieval, which is a different and older concern.

Content Structure — 11 points

The semantic-HTML5 backbone AI engines walk to chunk content for retrieval.

What this category covers

H1 presence and uniqueness per page, heading hierarchy discipline (H1 → H2 → H3 without level skips), content depth measured in word count, paragraph length, image alt-text coverage, and the use of semantic HTML5 elements over generic <div> soup. The category overlaps slightly with traditional content quality, but the criteria are AEO-shaped: AI engines reward structure that supports clean chunking, not just structure that humans find readable.

Why it matters

AI engines chunk pages into retrievable sections by walking the heading hierarchy and semantic-HTML5 backbone to decide where one logical section ends and another begins. A page with no H1 and a flat sequence of <div>s gets chunked badly regardless of how good its prose is. Clean structure is a prerequisite for retrieval, not an aesthetic preference.

SSR & AI Rendering — 11 points

Whether schema and critical content are present in raw HTML — not lazily JavaScript-injected.

What this category covers

Schema.org structures present in raw HTML versus only in the rendered DOM. H1 and body text present in raw HTML versus injected after JavaScript runs. Meta-description tags present in raw HTML. The audit fetches every page twice — once as raw HTML (what an AI crawler sees with no JavaScript execution) and once as rendered HTML (what a browser sees after the JavaScript settles) — and compares the two.

Why it matters

Most AI crawlers don't execute JavaScript. Single-page applications that hydrate their schema, headings, or body content after a JavaScript bundle loads are effectively invisible to those crawlers — the content exists for human browsers but not for the AI systems being measured. The Lightning Scan cap at 8 of 11 points reflects that detecting SSR/CSR drift across a site requires the multi-page evidence base the paid audit collects.

Site Architecture / Navigation — 10 points

The shape of how content is connected — what AI agents traverse to find related answers.

What this category covers

Sitemap presence (referenced in robots.txt and accessible at the conventional /sitemap.xml path), BreadcrumbList JSON-LD on sampled pages, semantic <nav> elements present in raw HTML, internal-link density per page, and URL hierarchy diversity (whether sampled URLs sit at multiple depths or all live at the root). The category focuses on the navigable connections between pages, not the content of any single page.

Why it matters

AI agents traversing a site for related answers depend on its navigable connections — sitemaps, breadcrumbs, semantic navigation, internal links. A flat site with no sitemap and sparse internal links reads as a collection of unrelated documents. Site Architecture is what makes a documentation site agent-traversable rather than just agent-readable.

Title & Identity — 9 points

The hygiene layer — title quality, brand signals, Organization schema, canonical URLs, and Open Graph basics.

What this category covers

Page title length and descriptiveness, whether the title is generic or unique to the page's content, brand-name signals across the title plus Open Graph plus schema (the audit looks for the brand name in three independent sources), Organization or LocalBusiness schema, canonical URL declaration, meta-description presence, and the Open Graph triple (og:title + og:description + og:image) that determines whether a shared link previews cleanly in AI assistants and chat surfaces.

Why it matters

Title and Identity is the smallest category by weight (9 points) because most sites pass it. The rare misses are quick to fix with outsized impact: a page with a generic Home title is invisible to AI engines trying to identify what it's about; a site with no Organization schema cannot be cleanly attributed to its operator.

Hard caps

Some failure modes are catastrophic enough that the rest of the rubric becomes irrelevant. The hard caps capture those cases: each cap names a specific trigger condition and the maximum Score the rubric will emit when that condition fires. Caps apply after the per-category points sum; the lowest cap wins when multiple fire on a single scan.

Universal caps

Apply across every vertical, in both Lightning Scan and full Audit modes.

Trigger Score capped at Modes
All monitored AI bots (GPTBot, ClaudeBot, CCBot, OAI-SearchBot, Googlebot) blocked in robots.txt 40 Lightning + Audit

Docs-tilt caps

Apply when the audit is scoring against the docs-vertical canonical (the current default). Future vertical tilts (Catalog, Brand) will publish their own per-tilt cap tables.

Trigger Score capped at Modes
Agent-metadata trio (llms.txt + agents.md + .well-known/mcp.json) all absent 75 Lightning + Audit
TechArticle / APIReference / HowTo schema missing on >50% of sampled doc pages 65 Audit only (multi-page evidence)
Auth-walled docs — >50% of doc pages return 401/403 to ObaronBot 50 Audit only (multi-page evidence)

Determinism

Property

The same pages scanned with the same rubric produce the same Score, by design. The Score is deterministic; it does not vary between scans of unchanged content.

Mechanism

No LLM is involved in scoring. Every check is a rule-based, deterministic evaluation against the page's structure, schema, and HTTP behavior. The audit pipeline does use an LLM to write the executive summary and remediation recommendations in the customer's report — those are commentary on the Score, not the Score itself. The Score is computed by the deterministic rubric, end-to-end.

UX promise

A change in the Score between two scans of the same site indicates one of two things: the pages changed, or the rubric versioned. There is no model variance and no sampling noise in the Score. When the Score moves, the cause is recoverable from the diff between the two scans plus the methodology version history.

What AI Readiness is NOT

AI Readiness shares territory with adjacent measurement categories. The boundaries:

vs. SEO

Different goal. SEO optimizes for human search-engine ranking — the question is whether Google or Bing ranks the page above competitors when a human types a query. AI Readiness optimizes for AI-system comprehension, retrieval, citation, and action — the question is whether Claude, GPT, Perplexity, or an autonomous agent can fetch the page, parse its structure, cite it accurately, and take action on what's there. Some signals overlap (sitemaps and structured data help both audiences), but the optimization targets diverge sharply once the work gets specific. SEO rewards keyword-rich titles and link-equity flows; AI Readiness rewards question-style headings and clean schema. A page can score well on SEO and badly on AI Readiness, and vice versa.

vs. web accessibility (WCAG, a11y)

Different audience. Web accessibility is for humans with disabilities — screen-reader users, keyboard-only navigation, sufficient color contrast, captions on video. AI Readiness is for machines. Some technical patterns overlap (semantic HTML helps both), but the audiences are different and so are the failure modes. A site can be fully WCAG-compliant and still score badly on AI Readiness if its content is gated behind JavaScript that AI crawlers don't execute. The two measurements are complementary; neither subsumes the other.

vs. Adobe's Brand Visibility

Different scope. Adobe's offering tracks brand mentions in AI engine outputs — an outcome layer measured by sampling AI responses and counting citation frequency over time. AI Readiness measures the content infrastructure that determines whether AI engines can find and cite the brand at all — an input layer measured by inspecting the site's structure, schema, and accessibility to AI crawlers. The two are complementary: AI Readiness is upstream of Brand Visibility outcomes. A brand cannot improve Brand Visibility without first being readable by the AI systems generating the visible outputs, which is what AI Readiness measures.

vs. management-consulting "AI Readiness"

Different layer entirely. The Big Four consultancies publish "AI Readiness Assessments" that measure organizational transformation — change-management readiness for AI tool deployment, internal AI strategy maturity, training and governance frameworks, integration of AI into business processes. Those assessments ship as decks, scorecards, and quarterly engagements at five- or six-figure prices. The methodology on this page is a different product entirely. It measures whether AI systems can read the site's content — the input infrastructure — not whether the company is ready to adopt AI internally. The Big Four flavor of AI Readiness is org transformation; this product's AI Readiness is content infrastructure. Same words, different products, different layer.

Per-vertical tilts

The rubric is structured as a shared core plus per-vertical tilts. The eight categories are stable across verticals; the weights and the additional vertical-specific checks vary per vertical. The current production rubric documents the docs vertical (v4.3) — the canonical Docs Readiness Audit. Two future tilts are named without committing their weights:

  • Catalog Readiness Audit — for ecommerce sites. Methodology page publishes when the catalog vertical launches.
  • Brand Readiness Audit — for marketing and brand sites. Methodology page publishes when the brand vertical launches.

This page documents the docs-vertical canonical (v4.3). When Catalog Readiness Audit and Brand Readiness Audit publish their methodology pages, those pages will document their own per-tilt weights and any vertical-specific checks. The shared core (the eight categories, the universal hard caps, the determinism property) stays stable across tilts; the per-vertical layer is additive.

Lightning vs. paid: bounded scoring

The free Lightning Scan covers single-page evidence — one fetch of the homepage plus single-fetch checks for robots.txt, llms.txt, agents.md, .well-known/mcp.json, and sitemap.xml. The paid Docs Readiness Audit covers multi-page evidence — roughly 30 pages smart-sampled across the site, fetched twice each (raw HTML and rendered HTML), with checks that compare across pages and detect template-level patterns the homepage cannot reveal.

Lightning's maximum possible Score is therefore 90, not 100 — by design. The 10-point gap represents the categories whose checks structurally require multi-page evidence that Lightning doesn't collect:

  • Documentation Patterns — Lightning maxes at 14 of 18 points. Lightning checks the agent-metadata trio (single-fetch) but cannot evaluate "TechArticle present on >50% of doc pages" without sampling the doc pages.
  • SSR & AI Rendering — Lightning maxes at 8 of 11 points. Detecting SSR/CSR drift requires comparing raw versus rendered HTML across multiple pages.
  • Site Architecture / Navigation — Lightning maxes at 7 of 10 points. URL hierarchy diversity, internal-link density distribution, and breadcrumb consistency all require multi-page sampling.

The bounded property is visible on every Lightning Scan result: the score-circle visualization on the homepage shows the earned Score, the empty track up to 90, and a hatched segment representing the 10 points the full Audit measures beyond Lightning. Lightning is honest about what it can and cannot see; the upgrade path to the full Audit is the literal hatched segment.

Lightning page-type calibration

Lightning Scan grades a single URL — typically the homepage. That URL might be a docs index hub, a marketing landing page, an e-commerce category root, or a long-form article. The rubric's article-shaped checks (TechArticle schema, code blocks, FAQ-style question headings) fit the article case; they misfit a docs index that routes to the articles themselves. Page-type calibration adjusts a small set of Lightning checks based on the detected page type, so that hub pages aren't graded against article-shaped expectations they cannot satisfy.

Calibration applies only at Lightning mode. The paid Docs Readiness Audit samples ~30 pages and scores each against the unmodified rubric — the page-type problem doesn't arise when the audit can see the article pages directly.

Detection

Page type is detected from a small set of signals: the URL path shape (root vs. multi-segment), the docs-platform fingerprint produced by platform detection, the inferred vertical, and the DOM shape of the homepage (presence of an H1-led hero, primary nav, a CTA-class anchor, an article schema or element). Detection emits a label plus a confidence tier:

  • High — multiple corroborating signals agree (e.g., docs platform detected at high confidence and the URL is at the docs root). Modifiers apply.
  • Medium — fewer signals or weaker corroboration (e.g., a hero-shaped marketing home at the root). Modifiers apply.
  • Low — ambiguous evidence. Modifiers do not apply; the rubric runs unmodified.

Page types

Page type Typical signals
docs_index Docs platform detected (Mintlify, Docusaurus, VitePress, GitBook, ReadMe) and the URL is the docs root or /docs
marketing_home Root URL with hero shape: a single H1, a body-copy paragraph nearby, a CTA anchor or button, primary nav
ecommerce_home Root URL with shop-vertical inference
article Multi-segment path or an Article / TechArticle schema or <article> element on the page
unknown Default. No modifiers apply; the rubric runs as-is.

Modifiers — docs_index

The first calibration target is docs index hubs. The modifier table below is published verbatim from the rubric module. Each row names a check that gets adjusted, the action taken, and the reason — so a reviewer can verify the adjustment is methodology-shaped, not score-massaging.

Check Action Reason
docs.tech_schema Swap → docs.collection_schema Docs index pages announce themselves with WebSite / SiteNavigationElement / CollectionPage schema rather than article-level TechArticle / APIReference / HowTo.
docs.code_blocks Swap → docs.reference_link_density Index pages route to articles that contain code; the index itself shouldn't be penalized for absent code blocks.
aeo.question_headings Reweight × 0.5 Question-style nav cards appear on docs indexes but the FAQ-style answer-density expectation does not fit a hub page — half-weight keeps the signal alive without dominating the score.

Effect on the bounded Lightning ceiling

Calibration preserves the bounded-Lightning property. The current docs_index modifier set uses swap and reweight actions only — both keep the Lightning maximum at 90. If a future modifier uses the skip action, the per-scan ceiling could fall below 90 (the rubric carries a per-scan lightning_max_points_sum field for exactly that case). The score-circle visualization reads that per-scan ceiling, so the hatched-segment story stays accurate when it does.

Disclosure on the result page

When modifiers apply, the Lightning result page lists the affected checks with a "(page-type calibrated — see methodology)" suffix that links back to this section. A buyer can see exactly which checks were swapped and why, and the modifier table above is the authoritative reference.

Source of truth for the modifier table and the apply logic: the canonical rubric module.