RAG Portfolio Governance. A low-cost control plane for monitoring and re-evaluating platform-specific RAG systems.

Status: reference design. The release unit, the golden-set CI gate and the four runtime layers it builds on come from production work on a shared enterprise retrieval platform (see Making enterprise RAG a platform, not a project). The portfolio control plane that extends them — tiered sampling, the judge ledger, error-budget policy as code — is a design; nothing here claims it is deployed as described. Every number is either derived from a stated formula or labelled as an assumption. Scope: an independent reference design written from public platform documentation and general practice. It describes no employer’s systems, data, vendors, metrics or internal documents, and every example is generic.

0. The thesis in one paragraph

Building a RAG system on a managed platform is now a short build. Knowing, a year later, whether it still answers correctly is where the money goes — because the obvious ways to find out all scale with the wrong thing. Judging every answer scales with traffic. Labelling by hand scales with people. Re-running every evaluation on every change scales with change. A portfolio of platform-specific systems multiplies all three. This design makes the cost of knowing flat: free structural signals on 100% of traffic, a judged sample whose size is set by statistics rather than by volume, human review fixed in hours per week, re-evaluation triggered by what changed and scoped to what it could have broken, and governance decisions made by a policy file against error budgets so that people meet only when a budget burns. One trace contract, one control plane, as many platforms as you have. Monitoring a RAG system well should cost about the same whether it answers two thousand questions a week or two million — and this is the design that makes that true.

1. The problem, stated plainly

“Platform-specific” here means a RAG system built with a platform’s native retrieval stack — Azure AI Search with Foundry, Amazon Bedrock Knowledge Bases, Vertex AI’s RAG Engine, Databricks Vector Search with its agent tooling, or a SaaS copilot’s built-in grounding — rather than hand-assembled from components. The platform does the chunking, embedding, indexing and retrieval. That is exactly why the build is cheap, and exactly why the monitoring is not:

1. Quality is not observable from logs. Latency, tokens and error rates are in the telemetry. Whether the answer was right is not. Getting that signal costs either a model call or a person, per answer — and the per-answer cost is what every naive design multiplies by traffic.
2. The world moves under a system that has not changed. The corpus churns. The questions drift. The model provider rolls a version. A prompt that was fine in March is answering a different question set over a different document set with a different model by September. A system that “has not been touched” is not a system that has not changed.
3. Every platform measures differently, or not at all. One platform offers sampled continuous evaluation of production traffic; another offers a batch evaluation job over a dataset you upload; a third exposes answers and citations but not the chunks it retrieved or their scores. Say, twenty systems on four platforms: that is twenty dashboards nobody can compare and one spreadsheet somebody maintains by hand.
4. The cost lands after the budget closed. The build was a project with a budget and an end date. Monitoring is an operating cost nobody put in the business case. Governance that is not cheap enough to live inside an operating budget is governance that quietly stops.
5. Re-evaluation is treated as an event. “We’ll do a full eval next quarter” means the system is unmeasured for a quarter and then measured in a panic. Both halves are expensive.

The naive responses each fail in a specific way:

Everything that follows is what is left after removing each of those.

2. Seven principles the design is built to enforce

1. Measure everything for free; judge a sample; escalate by disagreement. Structural signals that cost nothing run on every answer. A model judges a fixed-size sample. A stronger judge sees only the cases the cheap signals could not settle. A person sees only what the strong judge could not settle. Cost falls by an order of magnitude at each step; coverage falls with it, deliberately.
2. The sample size is set by the statistics, not by the traffic. Estimating a rate to ±5 points at 95% confidence takes a few hundred judged answers whether the population is two thousand or two million. The judged slice is a fixed weekly number per system. The cost of knowing is flat.
3. Retrieval is judged before generation, and without a model. Recall and ranking against known-good chunks are arithmetic. Most bad answers are retrieval failures wearing a generation costume, so the cheapest check is also the most diagnostic — and it is the one that runs on every corpus change.
4. Re-evaluate on change, scoped to the blast radius — never on the calendar. The release unit says what changed; a map from documents to golden items says what it could have broken; only that runs. A system nobody touched and whose corpus did not move is not re-evaluated; it is monitored.
5. Governance is a policy file, not a meeting. Service-level objectives, error budgets and the trigger matrix live in a versioned file evaluated by a scheduled job and in CI. The actions are mechanical: notify, ticket, block the release, freeze changes, escalate, propose retirement. The standing meetings are thirty minutes a month per high-risk system, driven by a generated evidence pack, and one portfolio session a quarter.
6. The judge is an instrument: calibrated, versioned, never trusted by default. Every judge is scored against human labels before it scores anything else; its agreement is tracked; its version is part of the measurement unit, and scores are never compared across measurement units. A judge that drifted is a broken ruler, and a broken ruler is worse than no ruler because it produces confident numbers.
7. One trace contract, one control plane, however many platforms. Each platform’s native telemetry and evaluators are adapters that fill a contract you own. The scorecard, the golden-set format, the rubrics and the policy are the same for every system, which is what makes every system comparable with every other and what lets one part-time governance owner run the portfolio.

A consequence worth stating as its own rule: every system earns its place. Usage and value are on the scorecard, and retirement is a first-class outcome of the quarterly review. The cheapest RAG system to govern is the one you switched off because nobody used it.

2.1 Where this departs from common practice

Five decisions separate this design from the usual LLMOps checklist, and each one is a cost decision before it is a quality decision:

1. Two judged slices with different jobs. Many programmes run one sample and use it both to report quality and to find failures. The random slice measures and is never used to pick fixes; the targeted slice finds and is never reported as quality. Mixing them makes the reported number drift toward whatever the team is chasing.
2. Retrieval regression without a model in the loop. Recall and ranking against known-good chunks are arithmetic, so the most frequent change — the corpus — is gated at the cost of a query, and the gate names the failing layer before anything is generated.
3. A measurement unit alongside the release unit. The judge, its prompt and its rubrics are versioned exactly like the thing they measure. Dashboards commonly compare scores across a judge change without saying so; this one refuses.
4. Budgets fixed in hours and in sample size, not in traffic. The human-review budget is hours per week; the judged sample is a number set by the precision wanted. Neither grows when the system succeeds, which is the only way a monitoring programme survives success.
5. Governance actions as a file. Error budgets and the trigger matrix are evaluated by a job, and every rung of the escalation ladder but the last happens without a meeting — the last lands in the quarterly review. Governance headcount is the cost most designs hide; here it is one part-time role for a portfolio.

3. Architecture at a glance

Six components, built in the order listed:

The registry is the spine: nothing is monitored that is not registered, and registration is where the owner, the risk tier and the budgets are declared — once, by the platform, not per team.

4. The registry — one record per system

Registration is the moment governance attaches. It is deliberately short, because a long registration form is one nobody fills in.

# registry/hr-policy-assistant.yaml  (illustrative values)
        system_id: hr-policy-assistant
        owner: people-ops-platform            # a team alias, never a person
        platform: azure-ai-search+foundry
        risk_tier: A                          # A customer-facing/regulated/writes · B internal decision support · C convenience
        surface: internal-chat                # where answers are shown; drives feedback capture
        release_unit:                         # what you ship — change any line and it is a release
          prompt_template: hr-answer@v14
          model: <deployment>@<pinned-version>
          retrieval: hybrid+rerank, k=8, entitlement_filter=in-query
          chunking: 512tok/64overlap@v3
          index: hr-policies@2026-10-28
          guardrails: std-internal@v2
        measurement_unit:                     # what you measure with — change it and you re-baseline, never compare across
          judge_small: judge-s@v3
          judge_escalation: judge-x@v2
          rubrics: v2
          calibration_set: calib/hr-policy-assistant.jsonl
        golden_set: golden/hr-policy-assistant.jsonl   # hold-out flag inside the items
        budgets:
          t1_random_per_week: 400
          t1_targeted_per_week: 400
          t2_escalation_share_max: 0.15
          t3_hours_per_week: 2
          judge_spend_cap_per_week_usd: 40    # the sampler shrinks the targeted slice first to stay under it
          cost_per_answer_max_usd: 0.05       # inference-cost ceiling, checked at release gates
        slos:                                 # set from a four-week baseline, then frozen for the quarter
          grounded_random: 0.95               # judged on the random slice only
          citation_coverage: 0.98
          golden_recall_at_k: 0.90
          false_refusal_rate_max: 0.05
          missed_refusal_rate_max: 0.02
          stale_citation_rate_max: 0.02
          observability_coverage: 0.99
          weekly_active_users_floor: 25       # the value floor for the quarterly earn-your-place rule
        corpus_freshness_days: 30             # a cited chunk older than this counts as stale
        retention:
          answer_text_days: 90                # Tier A keeps text for replay; Tier C may keep hashes only
        

Three fields do most of the work:

• risk_tier sets everything downstream — sample sizes, which judge, human hours, SLO strictness, retention, review cadence. Tier A is anything customer-facing, regulated, money-, health- or legal-adjacent, or that produces proposals that write to a system of record. Tier B is internal decision support. Tier C is internal convenience. A typical portfolio is mostly C with a handful of A; the budgets below reflect that.
• release_unit is the correction most teams still need: the thing you release is the whole configuration — prompt template, pinned model version, retrieval strategy and parameters, chunking and index version, guardrail thresholds — not the code. A provider rolling a model version underneath you is a release you did not perform; pinning turns a silent change into a dated one you schedule.
• measurement_unit is the half of that idea most teams miss. The judge, its prompt and its rubrics are also a versioned configuration. Scores produced under different measurement units are not comparable, and the registry makes that impossible to forget.

5. The trace contract — one record per answer

Everything in this design is derived from one record. If a platform cannot fill a field, the field is null, and the share of nulls is itself a scorecard metric (observability_coverage), so a platform’s blind spot is visible rather than silent.

{
          "trace_id": "…", "system_id": "hr-policy-assistant", "ts": "2026-11-01T14:02:11Z",
          "release_unit_hash": "7f3a…",            // from the registry at answer time
          "request": {
            "role_scope": "hr-generalist/EU",      // entitlement context, never the identity
            "user_ref": "hmac:9c1e…",              // keyed hash for re-asks and usage counts; never the identity
            "query_text": "…",                     // Tier A/B: text (retention-governed); Tier C: hash only
            "query_embedding_ref": "…",            // for novelty and drift, not for retrieval
            "session_id": "…"                      // for re-ask and abandonment signals
          },
          "retrieval": {
            "index_version": "hr-policies@2026-10-28",
            "k": 8, "filter_applied": true,
            "candidates": [{"chunk_id": "pol-leave-2026#4.2", "doc_id": "pol-leave-2026", "score": 0.81,
                            "source_modified": "2026-09-30"}, …]
          },
          "generation": {
            "model": "<deployment>@<version>", "prompt_template": "hr-answer@v14",
            "tokens_in": 2140, "tokens_out": 212, "latency_ms": 1830, "cache_hit": false
          },
          "answer": {
            "text": "…",                            // retention-governed, same rule as query_text
            "cited_chunk_ids": ["pol-leave-2026#4.2"],
            "claim_sentences": 4, "cited_sentences": 4,   // emitted by the orchestrator, or null
            "refusal": false, "guardrail_verdicts": {"input": "pass", "output": "pass", "groundedness": 0.93}
          },
          "feedback": {                             // appended as it arrives
            "explicit": null, "reask_within_10m": false, "citation_clicked": true,
            "escalated_to_human": false
          },
          "judgements": [                           // appended by T1/T2/T3; absent on most traces
            {"tier": "T1", "slice": "random", "judge": "judge-s@v3", "rubrics": "v2",
             "scores": {"grounded": 1, "citations_correct": 1, "context_relevant": 1, "complete": 0, "refusal_appropriate": 1},
             "rationale": "Omits the pro-rata rule stated in the cited section."}
          ]
        }
        

Design notes that save pain later:

• Store chunk IDs and scores, not chunk text. Text is re-resolvable from the index snapshot; IDs and scores are what every T0 signal needs and they are small.
• Candidates and citations are different lists. Citations must be a subset of candidates; a citation outside the candidate list is a structural failure worth its own alert, and it is free to detect.
• claim_sentences / cited_sentences make citation coverage a structural metric instead of a judged one — but only if the orchestrator emits them. Where it does not, coverage falls to T1.
• Map to OpenTelemetry where the attributes exist, but own the contract. The GenAI semantic conventions (gen_ai.*) were still at Development status in mid-2026, having moved into their own repository at v1.42.0 without a tagged release; the gen_ai.evaluation.result event is early. Emit OTel spans by all means — pin the instrumentation version and record the schema it targets — but treat the contract above as the stable thing and the OTel mapping as an adapter.
• A keyed hash of the user, never the user. user_ref exists so that re-asks, weekly active users and questions per user can be computed. The key lives with the governance owner and rotates; without it the trace store cannot be joined back to a person.
• Privacy is a tier decision, made once. Tier C keeps query and answer as hashes plus scores; that is enough for every T0 signal and for drift. Tier A keeps text for replay, under the retention window in the registry and behind access control. Data residency that forbids storing answers degrades T3, not T0.

6. Four tiers of signal — and why the cost is flat

6.1 T0 — free signals on 100% of traffic

Everything in this tier is a SQL view over the trace store, plus one feed that is not a trace — the ingestion log that says which documents were added, changed or deleted. No model is called. These run continuously and they are the alarms that decide where the judged budget goes.

Two of these deserve a sentence. Re-ask within ten minutes is a free proxy for a bad answer that few teams compute: a user who asks the same thing again, rephrased, did not get what they needed, and said so without clicking anything. Novelty rate is the free early warning that the golden set no longer represents production — it rises before any quality metric falls.

6.2 T1 — a small judge on a fixed weekly sample

A small, cheap model judges a fixed number of traces per system per week against short, binary rubrics. Two slices, with different jobs:

• The random slice measures. Uniformly sampled from the week’s traffic. Its scores are the system’s reported quality and the only ones the SLOs are evaluated on. It is never used to pick what to fix, so it stays unbiased.
• The targeted slice finds. Filled most-informative-first from T0 anomalies: low retrieval margin, novel queries, re-asks, thumbs-down, refusals, answers citing documents changed in the last week, narrow-permission roles, citations outside the candidate list. Its failure rate is not reported as quality — it is designed to be high. Its job is to surface failures cheaply and feed them to T3 and the golden set.

Rubrics are binary and reference-anchored — a third value, cannot tell, is permitted and routes the trace to T2 — because small judges are reliable on “is each claim supported by the cited chunk — yes or no” and unreliable on “rate this answer 1–10”:

The judge sees the question, the candidates, the citations and the answer. It never sees the user’s identity. Its prompt is versioned in the repository, and every verdict carries a one-line rationale, which is what makes T3 fast.

6.3 T2 — an escalation judge for what T1 could not settle

A stronger model — ideally from a different family than the generator, so its blind spots differ — sees a trace only when:

• cheap verdicts disagree: the judge against the runtime groundedness score, against explicit feedback, against a structural signal (a citation outside the candidate list), or against itself (grounded = 1 with citations_correct = 0) — or the judge answered cannot tell;
• the system is Tier A and the answer touches money, health, legal or eligibility terms;
• the query is novel (far from every golden item) and the T1 verdict was negative.

The design target is that T2 sees no more than 10–15% of the T1 volume; the registry caps it (t2_escalation_share_max), and the cap is enforced by sampling the escalation queue rather than by hoping. T2 also judges the golden set at release time, so that the gate uses the stronger instrument.

6.4 T3 — human review, fixed in hours

People review a queue, filled most-informative-first, for a fixed number of hours per week set in the registry — two for a Tier A system, one for Tier B, none standing for Tier C (quarterly only). The budget is hours, not cases, which is what makes it an operating cost someone will approve. The queue contains, in priority order:

1. T2 verdicts with low confidence or with a rationale that contradicts the runtime verdict;
2. every Tier A explicit complaint whose T1/T2 verdict was “fine” (the judge might be wrong — that is a calibration case);
3. a fixed audit of twenty randomly chosen judge verdicts, for calibration (§10) — Tier A and B; Tier C is audited at the quarterly session;
4. golden-set candidates and staleness checks (§7).

Every label a human produces goes to two places automatically: the calibration set, and — if it is a failure with a confirmed cause — the golden set.

6.5 The sampling arithmetic — why the cost does not grow with traffic

To estimate a proportion to within ±e at confidence z, the sample needed is

n = z² · p(1 − p) / e²

and at the worst case p = 0.5 with z = 1.96 (95% confidence):

This number does not depend on the population size until the population is small enough for the finite-population correction to matter — at which point the sample shrinks, never grows. A system answering 2,000 questions a week and one answering 2,000,000 need almost the same random slice for the same precision — the correction trims the smaller one a little. That is the whole economic argument: the cost of knowing is bounded by the precision you want, not by the traffic you carry.

Detecting a change needs more than estimating a level. For a two-sided test at α = 0.05 with 80% power, comparing one week’s random slice against the previous week’s:

So the default Tier A budget of 400 random judged answers per week reports the level to about ±5 points and sits just under the 435 that gives full 80% power to catch a 5-point fall from a 95% baseline in a single week; a slower slide is caught by a cumulative-sum chart over the weekly slices, which detects small sustained drift without a larger sample — it just takes a few weeks longer. Tier B halves the budget; Tier C runs 100 a week or judges everything if traffic is below that, whichever is cheaper.

7. Golden sets that maintain themselves

A golden set is the artefact that converts “we think it’s fine” into evidence, and the one that decays fastest if left alone. The design makes maintenance a by-product of operation.

7.1 The item

{
          "id": "g-0142",
          "question": "How much annual leave does a part-time employee in the EU accrue?",
          "asked_as": {"role": "hr-generalist", "region": "EU"},
          "expected": {"doc_ids": ["pol-leave-2026"], "anchors": ["4.2 Part-time accrual"]},
          "answer_must": ["pro rata", "28 days full-time equivalent"],
          "answer_must_not": ["carries over indefinitely"],
          "should_refuse": false,
          "category": "changed-recently",
          "created_from": "trace:…",
          "last_verified": "2026-10-20",
          "hold_out": false
        }
        

Expected chunks are expressed as document ID plus anchor text, not raw chunk IDs, because chunk IDs change on every re-index and anchors survive it. The resolver maps anchors to current chunk IDs at run time; an anchor that no longer resolves is a staleness flag, free of charge.

7.2 Category quotas

A set made only of questions the system answers well measures nothing. Each system’s set carries minimum quotas per category, and the registry check fails if a quota is unmet:

7.3 Seeding before the first ship

Build the set before launch, not after the first complaint — retrofitting one under time pressure means reconstructing what “good” meant from memory. Two low-cost sources:

• An hour with the subject-matter owner, producing the questions they are tired of answering and the ones they fear being answered wrongly.
• Synthetic generation from the corpus: a cheap model writes a question for a passage, and the passage is the expected chunk by construction. The owner accepts or rejects in bulk. This gives breadth; the owner’s hour gives the cases that matter.

A few hundred items covers most systems. Breadth across categories matters more than count.

7.4 Growing from production at near-zero marginal cost

• Every T3 label on a failed trace with a confirmed cause becomes an item, in the category of its cause.
• Every targeted-slice failure that led to a fix becomes an item, so the fix is a permanent test.
• Every month, the largest novelty cluster that has no golden coverage gets five items from T3’s hour.

7.5 Staleness, blast radius and the hold-out

• Staleness is detected for free. An item whose source document was modified or deleted after last_verified, or whose anchor no longer resolves, is flagged. Flagged items join the T3 queue, where re-verifying one takes minutes. Unverified items older than a quarter are flagged regardless.
• Blast radius is a map from document to items. expected.doc_ids is indexed in reverse, so a corpus change produces the exact list of items it could affect — plus the no-answer items, because a new document may have made an answer exist. That list, not the whole set, is what re-runs (§8).
• A hold-out subset is never used for tuning. One item in five is flagged hold_out; it runs only at release gates and is never shown to anyone tuning a prompt or a retrieval parameter. Without it, the set is quietly optimised against and stops measuring anything.

8. Re-evaluation: triggered by change, scoped by blast radius

The calendar is not a trigger. The release unit, the corpus and the measurement unit are. What changed decides what runs, and the gate is proportional to what could have broken.

Three properties of this matrix do the cost work. Corpus changes — usually the most frequent event — trigger only arithmetic over a short list. Model and prompt changes — the expensive evaluations — are rare and scheduled. And the “nothing changed” row has no evaluation at all; it has monitoring, which is free.

Retrieval gates carry no tolerance, because recall against known chunks is deterministic arithmetic: a regression is a regression, and the published rule stands — a change that regresses retrieval on the golden set does not ship. Judged generation scores carry a small tolerance (two points in the policy file below) because judging is noisy, and the hold-out subset keeps that tolerance from being gamed.

9. SLOs, error budgets and policy as code

9.1 Objectives, set from a baseline

An objective declared before measurement is a guess. The design runs each system for four weeks under T0 and T1 before any SLO is set, then sets objectives slightly below the observed baseline for the tier, and freezes them for the quarter. Illustrative targets, to be replaced by your baselines:

9.2 Error budgets, borrowed from SRE

A 95% objective on grounded_random is a 5% budget of judged-ungrounded answers over a 28-day window; every tier has one, at its own objective. The budget is the governance currency: while it holds, the owner ships freely; as it burns, the ladder below engages mechanically. Because the signal arrives in weekly slices, burn is computed per slice. At 400 judged answers a week and a 95% objective the budget is 80 ungrounded verdicts, so one week at 90% (40) notifies the owner, a second week at 90% (80) opens a ticket and blocks releases, and a freeze needs the slide to continue into a third.

9.3 The policy file

# policy/portfolio.yaml  (illustrative) — thresholds are read from each system's registry entry, never repeated here
        policies:
          # ---- budgets (continuous) ----
          - id: grounded-budget
            applies_to: {risk_tier: [A, B, C]}
            signal: judged.grounded.random_slice        # never the targeted slice
            window_days: 28
            objective: registry.slos.grounded_random
            ladder:
              - burn: 0.50  → notify: owner
              - burn: 1.00  → open_ticket: {severity: high}; block_releases: true
              - burn: 1.50  → freeze: {except: fixes}; escalate: governance_owner
        
          - id: judge-spend-floor-and-cap
            applies_to: {risk_tier: [A, B, C]}
            rule: |
              if weekly_traffic < registry.budgets.t1_random_per_week: judge_everything          # the floor
              if judge_spend_week > registry.budgets.judge_spend_cap_per_week_usd:               # the cap
                shrink: [t1_targeted, t1_random]
                flag_scorecard_if: precision_worse_than(5pts)
        
          # ---- release gates (on change) ----
          - id: release-gate-retrieval
            trigger: release_unit.changed(chunking | index | retrieval)
            run: golden.retrieval(scope: all)
            pass_if: no_item_regressed and recall_at_k >= registry.slos.golden_recall_at_k
            then: golden.generation(scope: sample(0.2), judge: escalation, tolerance: 0.02)
            fail: block_release
        
          - id: release-gate-generation
            trigger: release_unit.changed(prompt_template | model)
            run: golden.generation(scope: all, judge: escalation)
            pass_if: no_rubric_regresses(tolerance: 0.02)
                     and cost_per_answer <= registry.budgets.cost_per_answer_max_usd
                     and latency_p95 <= baseline.latency_p95 * 1.2
            after_pass: {if: model, t1_random_multiplier: 2, weeks: 1}   # the doubled week closes a model change
            fail: block_release
        
          - id: release-gate-guardrails
            trigger: release_unit.changed(guardrails)
            run: golden.generation(scope: categories[no-answer, ambiguous, injection-in-document], judge: escalation)
            pass_if: new_false_refusals == 0 and new_missed_refusals == 0
            fail: block_release
        
          - id: corpus-delta
            trigger: corpus.changed
            run: golden.retrieval(scope: blast_radius + category:no-answer)
            pass_if: no_item_regressed
            fail: {hold_index_promotion: true, notify: owner}
        
          - id: measurement-unit-change
            trigger: measurement_unit.changed
            run: judge.kappa(on: calibration_set + last_4_random_slices)
            pass_if: kappa > 0.60 per rubric
            then: {scorecard.rebaseline: true, scorecard.compare_across_units: false}
            fail: {revert_measurement_unit: true, notify: governance_owner}
        
          # ---- drift and the instrument ----
          - id: query-drift
            trigger: t0.novelty_rate > threshold for 2 consecutive weeks  or  t0.psi > 0.25   # 0.25: the conventional "significant shift" line in model-risk practice
            action: {open_ticket: {severity: medium}, t1_targeted_weight: novel_clusters, t3_queue: golden_candidates(5)}
        
          - id: judge-drift
            applies_to: {risk_tier: [A, B]}             # Tier C: at the quarterly session
            trigger: schedule.monthly
            run: judge.kappa(labels: last_4_weeks)      # about 80 labels: the twenty-a-week T3 audit
            pass_if: kappa > 0.60 per rubric
            fail: {recalibrate: true, scorecard_flag: "judge-unreliable"}
        
          # ---- the quarter ----
          - id: earn-your-place
            trigger: schedule.quarterly
            condition: usage.weekly_active_users < registry.slos.weekly_active_users_floor for 8 weeks
            action: propose_retirement
        

The file is evaluated by a scheduled job against the trace store and by CI on every change to a registry entry. Nobody has to remember a rule, which is the point: a control someone has to remember is not a control.

9.4 The escalation ladder

Notify the owner → open a ticket → block releases in CI → freeze changes except fixes → escalate to the governance owner → propose retirement. Each rung but the last is a file change and an automated action, not a meeting; a retirement proposal lands in the quarterly review. The monthly thirty minutes exist to look at the evidence pack and decide the things a file cannot: whether a cause is understood, whether a fix is a fix, and whether a system still earns its place.

10. Judge governance — the judge is an instrument

LLM-as-judge programmes commonly fail here, quietly: the judge is adopted because it produced plausible numbers, and nobody checks whether the numbers mean anything.

• Calibrate before trusting. Each Tier A or B system builds a calibration set of 100–200 human-labelled traces from its T3 queue — in its first month if the hours allow, otherwise as they accrue — stratified across rubric outcomes and deliberately including failures; Tier C systems share a portfolio-level set labelled at the quarterly session. A judge is accepted per rubric when its agreement with the human labels, measured as Cohen’s κ, is above 0.60 — the floor of the “substantial” band (0.61–0.80) on the Landis–Koch scale. Below that, the rubric prompt is revised or the rubric moves to T2.
• Audit continuously, cheaply. For Tier A and B systems, twenty of each week’s T1 verdicts are re-labelled by a human inside the T3 hours; κ is recomputed monthly on the last four weeks’ labels, and a fall below threshold flags the scorecard and triggers recalibration.
• Version the measurement unit; never compare across it. The judge model, its prompt and the rubric set are hashed into the registry. Changing any of them re-baselines the scorecard. Comparing last quarter’s groundedness under judge v2 with this quarter’s under judge v3 is comparing two rulers, and the policy file refuses to do it.
• Prefer binary rubrics with the evidence attached. “Is this sentence supported by this chunk” is a question a small model answers well; “rate the answer’s quality” is not. Reference-anchored, yes/no, one rationale line.
• Cheap disagreement before expensive judgement. A verdict that contradicts the runtime groundedness score, the user’s explicit feedback, a structural signal or the judge’s own other rubrics is the cheapest possible detector of a hard case — and hard cases are what T2 is for.
• Different family for the escalation judge. A judge that shares the generator’s training lineage shares its blind spots. Where the platform allows it, T2 uses a different model family.
• Anti-Goodhart, structurally. The hold-out golden subset is never tuned against. The random slice is never used to select fixes. The targeted slice’s failure rate is never reported as quality. Judge rationales are stored so that T3 reviews the reasoning, not just the verdict.

And the limit that must be written where everyone sees it: grounded is not correct. A judge scoring groundedness verifies that the answer follows from the retrieved context. If the context is wrong — an outdated policy that was never withdrawn, a plausible document that contradicts the authoritative one — the answer is grounded and wrong, and no judge without ground truth will say so. Correctness is measured only on golden items, which is why the golden set exists, and why corpus curation is a governance concern rather than a content-team chore.

11. Cadence and roles

Roles, kept to three because governance headcount is the cost most designs hide:

• System owner — accountable for the SLOs; does the T3 hours; approves golden items; answers for the evidence pack. One per system, already exists.
• Governance owner — runs the control plane, the policy file, the judge ledger and the quarterly review. One person, part-time, for a portfolio. This is a role, not a team.
• Security and compliance — consumes evidence packs; approves changes to the policy file; is not in the weekly loop at all.

The evidence pack is the artefact that makes the monthly review thirty minutes and the audit conversation short. One generated page per system: objectives against actuals with budget burn; the release log for the month with each gate’s result; golden-set health (size, category quotas, stale items, hold-out untouched); judge κ per rubric; top three failure categories from the targeted slice; usage and cost per answer; open tickets. It is produced from the trace store by a job, which means it is never out of date and nobody wrote it.

12. The cost model

Per system, per week, the model spend is

C = n_T1 · r · c_small + n_T1 · s · r · c_esc + (release runs: |golden| · (c_gen + r · c_esc) per generation-affecting release)

where n_T1 is the T1 sample (random + targeted), r the number of rubrics, c_small and c_esc the cost of one judge call at each tier, s the escalation share, and the release term applies only in weeks with a prompt, model or guardrail change. Retrieval evaluation has no model term at all. Human cost is the registry’s T3 hours.

An illustrative instantiation — every unit cost here is an assumption to replace with your price list, and the shape of the result is the claim, not the figures:

The comparison that matters: judging every answer of a Tier A system at 2,000,000 answers a week with the same five rubrics, even at the small-judge price, would be on the order of $10,000 a week — for precision beyond ±5 points that nobody needs. In this illustration the sampled design, escalation included, is cheaper by a factor in the hundreds (about $22 against about $10,000); whatever your unit costs, the factor grows as the system succeeds.

What scales with traffic in this design: trace storage and the SQL that runs over it. What does not: everything with a model or a person in it. That is the property that lets governance survive an operating budget.

Two guard-rails on the budget itself, both in the policy file: a floor — if weekly traffic is below the sample size, judge everything, which is cheaper than the sample; and a cap — judge spend per system per week may not exceed the registry’s judge_spend_cap_per_week_usd, and the sampler shrinks the targeted slice first, then the random slice, to stay under it, flagging the scorecard when precision falls below ±5 points.

13. Platform mapping — native tooling as adapters

The rule: the platform’s evaluators are adapters, not the governance. The contract, the rubrics, the sampling and the policy are yours, so that twenty systems are comparable and no vendor’s price list sets your sampling rate. What each platform contributes, at the time of writing (November 2026):

The honest consequence of the last row: a platform that will not show you what it retrieved cannot be governed to Tier A standards, and the registry should say so by capping its tier.

14. Adoption: thirty days to a running plane, six weeks to frozen SLOs

Start with one or two systems — one Tier A, one Tier C — and onboard the rest at a system a week once the plane exists.

By the end of the first quarter: every registered system has a baseline, an evidence pack, a tier assigned by its exposure and a place it has earned; the portfolio review has retired or merged whatever nobody used.

15. Honest limits

• Grounded is not correct. Sampling and judging verify that answers follow from the context. If the corpus is wrong, the system is confidently, measurably, grounded — and wrong. Correctness is measured only on golden items with known answers. Corpus curation is the control, and this design makes it visible, not automatic.
• A rate is not an incident. Statistical sampling tells you that about 4% of answers are ungrounded; it does not tell you which user got one. A Tier A system where a single bad answer carries liability needs the per-answer runtime layers — content safety screening on input and output, citations as a hard requirement, groundedness scoring with refusal below threshold, human approval before any write — and this plane does not replace them. It measures whether they are working.
• Explicit feedback is sparse and biased. It is a targeting signal for the targeted slice, never a quality metric.
• Judges share blind spots with generators. Different model families help; they do not eliminate the problem. The twenty-label weekly audit is the detector, and it detects slowly.
• Drift detectors flag shifts, not causes. A rising novelty rate says the golden set is behind; it does not say what the new questions are. Somebody looks — that is what the five monthly items are for.
• Closed platforms cap what can be governed. The coverage metric makes the gap visible; it does not close it. Where the contract cannot be filled, the tier is capped, and that is a product decision made explicit.
• Where answers cannot be stored, the contract runs on hashes and scores. T0 and drift are unaffected; T3 loses the ability to replay, and calibration has to be done on a consented sample.
• The figures in §12 are assumptions. The formulas are the claim; the dollar amounts are placeholders chosen to show the shape. Replace them with your price list before anyone quotes them.
• The hard part is not technical. A policy file that blocks a release only works in an organisation that lets a file block a release. The design gives the governance owner a mechanism; it does not give them the authority.

16. Compliance mapping, summarised

The evidence pack is designed to be the artefact an auditor asks for, produced as a side-effect rather than as a documentation project. Mapped loosely, and not as legal advice:

• NIST AI RMF — the Measure function (metrics, monitoring, evaluation of trustworthiness characteristics over time) and Manage (responding to and documenting identified risks) are what the scorecard, the policy ladder and the ticket trail produce.
• ISO/IEC 42001 — the monitoring, measurement, analysis and evaluation requirements of an AI management system, with records: the registry is the inventory, the evidence packs are the records, the quarterly review is the management review.
• EU AI Act — following the AI Omnibus (Regulation (EU) 2026/1744, in force 27 July 2026), the Annex III high-risk obligations now apply from 2 December 2027 and the Annex I product-embedded obligations from 2 August 2028; the Article 50 transparency obligations apply from 2 August 2026. An internal RAG assistant is unlikely to be a high-risk system on its own, but one used to inform employment, education or credit decisions can be — and the logging, monitoring and record-keeping a high-risk designation demands are the kind of evidence this design produces as a by-product. It is cheaper to have the records than to retrofit them; whether they are sufficient is a question for counsel.

Closing

The expensive half of RAG was never the build. It was the question “is it still right?”, asked every week, for every system, forever. The answer to that question does not have to cost more as the system succeeds — and a governance programme that costs more as the system succeeds is one the organisation will eventually stop paying for.

If a provider retired your current model next month, how many of your RAG systems could prove the replacement was safe with a file — and how many would need somebody’s afternoon?

Related public write-ups: Making enterprise RAG a platform, not a project · Schema-to-Screen — an agentic, RAG-grounded reference architecture · TOGAF, applied for real.

Sources consulted for the platform and regulatory statements in §5, §13 and §16 (November 2026): Microsoft Foundry observability documentation; Amazon Bedrock RAG evaluation metrics documentation; the OpenTelemetry semantic-conventions-genai repository (no tagged release; schema URL marked TODO) and an independent summary of its status; the Vertex AI Gen AI evaluation service overview; the Databricks MLflow 3 production-monitoring documentation; legal commentary on Regulation (EU) 2026/1744. Platform capabilities change quickly; re-verify before relying on a row of §13.