Making enterprise RAG a platform, not a project. What broke when every team built their own.

A note on scope. What follows is the generalised reference architecture. The shapes, the failure modes and the platform decisions are real and mine; the employer’s deployment topology, domain data, index schemas and prompts stay behind their firewall. Nothing here is specific enough to reconstruct their system, and that is deliberate.

The problem: a dozen RAG stacks, none of them the same

Every team that wants an AI feature reaches for the same six pieces: a chunker, an embedding model, an index, a retriever, a prompt, and a model call. The first team builds it. The second team copies it. The third team copies the copy, from a branch, six months later, with a different embedding model because that one was cheaper that quarter.

Within a year you don’t have one RAG system. You have a dozen — each with its own chunking strategy, its own idea of what “fresh” means, its own entitlement handling (usually none), its own cost profile, and no shared way to tell whether any of them is answering correctly. The expensive part isn’t the duplicated effort. It’s the divergence.

Divergence is what makes ordinary governance questions unanswerable. Which of our AI features can see this document? has a dozen answers, held in a dozen codebases, none of them written down. Every new feature repeats the same security review from scratch because nothing is shared enough to inherit an approval. And no one can improve retrieval quality once, for everyone — the same tuning change becomes a dozen tickets, of which four get done.

So the goal was never “build a RAG app.” It was: make retrieval a piece of platform infrastructure, the way authentication or logging is. Teams register a workflow, point it at a scoped index, and inherit hybrid retrieval, semantic caching, entitlement filtering, groundedness scoring, tracing and a token budget. What follows is what it took to get there, in the order the problems actually surfaced.

What broke first: retrieval, not generation

The instinct when a RAG answer is wrong is to blame the model, and then to reach for a bigger one. Almost none of the early production incidents were generation failures. They were retrieval failures wearing a generation costume.

Pure vector search fails hardest on exactly the queries an enterprise cares most about: identifiers. A user asks about contract A-4471-B, or a specific VIN, or a policy number. Embeddings are built to collapse surface form into meaning — which is precisely the wrong behaviour when the surface form is the meaning. The nearest neighbours of an identifier are other identifiers that look similar and mean something completely different. The retriever returns confident, well-scored, wrong documents, and the model dutifully summarises them.

Nearly every “the model hallucinated” bug I chased turned out to be “the model was never shown the answer.”

The fix is hybrid retrieval, and it needs to be there from the first day rather than bolted on after a quality complaint. Azure AI Search runs BM25 keyword scoring and vector similarity over the same index, fuses the two candidate sets with reciprocal rank fusion, and then applies semantic ranking as a second-stage re-rank over the fused top-N. Keyword recall catches the exact token. Vector recall catches the paraphrase that shares no words with the document. The re-ranker decides which of the surviving candidates actually answers the question rather than merely resembling it.

The second retrieval failure was chunking, and it was quieter. Fixed-size windows cut tables in half, strand a heading from the body it introduces, and split a clause across two chunks so that neither one is retrievable by the thing it says. Structure-aware chunking — split on the document’s own structure first, then pack to a token budget rather than the other way round — fixed more answer-quality problems than any prompt change did.

The part of chunking that matters more than the split, though, is what you attach to each chunk. Every chunk carries its source system, document identifier, section path, effective date and entitlement tags as retrievable fields. Everything downstream depends on those existing: filtering, citation, freshness checks, security trimming, evaluation. And they can only be added at index time — deciding you need one later means a full reindex of the corpus. Choosing that metadata carefully is the highest-leverage hour in the whole build.

Keeping the index honest

The first version reindexed on a schedule, because that is what every tutorial does. A nightly rebuild produces a window — sometimes many hours wide — in which the platform answers confidently from data it already knows is stale. In a fleet and mobility business, an asset’s status changing is exactly the kind of fact somebody asks about within minutes of it changing. “Correct as of last night” is not a property you can put in front of an operations team.

The fix has two halves, and the second one is the one people skip.

The first half is event-driven incremental indexing. Change events from the source systems land on Event Grid; Azure Functions consume them and drive the Azure AI Search indexers — pull the delta, crack the document, re-embed only what actually changed, upsert. The jobs are idempotent and replayable from the event log, which is what makes an embedding-model change survivable: you roll the version back and re-run the affected partitions instead of rebuilding the world.

The second half is deciding what doesn’t belong in the index at all. Facts that must never be stale — balances, live statuses, current positions, anything transactional — are not retrieved from the index. The orchestrator queries SQL Managed Instance directly for those, at answer time, and the index carries the documents and the narrative context around them. Which facts are “index facts” and which are “live facts” is an architectural decision made once per domain, not something the retriever should be guessing at per query.

Entitlements belong in the index, and in the query

The obvious first design retrieves the top K chunks and then filters out whatever the caller isn’t allowed to see. It has two problems, and one of them is serious.

The mild problem: you asked for ten and got three, because seven were filtered after the fact. Recall silently collapses for exactly the users with the narrowest permissions, and nobody notices until one of them complains that the assistant “doesn’t know anything.”

The serious problem: post-filtering puts the security boundary in application code, which means every new workflow has to remember to call it. On a platform that expects to host twenty-plus workflows written by different teams, “remember to call the filter” is not a control. It is a future incident with a date on it.

So entitlement tags are indexed fields, and the filter is composed into the query itself, from the caller’s identity, before the search ever executes. Retrieval cannot return a chunk the caller isn’t entitled to, so no downstream component can leak one by forgetting a step.

POST /indexes/{index}/docs/search
{
  "search": "<user question>",
  "vectorQueries": [
    { "kind": "vector", "fields": "contentVector", "k": 50 }
  ],
  "queryType": "semantic",
  "filter": "domain eq 'contracts'
             and effectiveDate le {asOf}
             and entitlements/any(t: search.in(t, '{callerGroups}'))",
  "top": 12
}

The claims that populate callerGroups come from the token APIM already validated at the edge. A workflow never asserts its own entitlements and never passes a user identity it constructed itself — it forwards the one the platform authenticated. That is the whole reason the edge and the retrieval layer have to be designed together rather than in sequence.

The semantic cache, and when it lies

Semantic caching is the cheapest large win available in a RAG platform, and the easiest place to introduce a correctness bug that nobody catches for months. The naive version embeds the incoming question, looks for a previous question within a similarity threshold, and returns the cached answer. It is fast, it is obviously a good idea, and it is wrong.

Two questions can embed almost identically and still require completely different answers. What’s the mileage limit on this lease? is the same sentence for two different leases. The same question asked by two users with different entitlements has two legitimate answers. The same question asked before and after a fact changed has two legitimate answers. Similarity tells you the questions are alike; it tells you nothing about whether the previous answer is still legal for this caller, right now.

A cache that returns the right answer to the wrong question is worse than no cache, because it is fast and it is confident.

So the cache key isn’t the question. Similarity selects the bucket; a tuple decides whether the hit is admissible — the index and domain scope, a fingerprint of the caller’s entitlements, the model deployment and prompt template version, and a freshness generation stamp that the ingestion pipeline bumps whenever the underlying partition changes. A hit has to match on all of it. Bumping the generation stamp on write is what makes the cache self-invalidating instead of TTL-hopeful.

The threshold is per workflow, not global. A workflow answering from stable policy documents can afford to be generous. A workflow answering about one specific asset cannot, and its threshold sits high enough that it effectively only serves exact repeats. Making that a workflow-level setting rather than a platform constant is the difference between a cache teams trust and a cache teams quietly disable.

One more thing worth doing: cache the retrieval result set separately from the generated answer. Retrieved context is far more reusable than completions, and a retrieval-layer hit still avoids the search and embedding round trips even when the generation has to run fresh.

One orchestrator, many workflows

The orchestration layer is custom .NET on Azure Container Apps, with Azure Functions carrying the event-driven and long-running work. The code is the less interesting half. The half that turns a shared library into a platform is the workflow registry.

A team doesn’t stand up a stack. They register a workflow, and the registration is a declaration of everything the platform needs to enforce on their behalf: which index scopes it may read, which model tier it uses, its retrieval strategy, its prompt template version, its token budget, its cache policy, and its guardrail profile.

{
  "workflow":   "contract-summary",
  "owner":      "team-contracts",
  "scopes":     ["contracts", "policy-docs"],
  "retrieval":  { "mode": "hybrid", "k": 40, "rerank": "semantic", "top": 12 },
  "model":      { "tier": "balanced", "fallback": "economy" },
  "prompt":     { "template": "contract-summary@v7" },
  "budget":     { "promptTokens": 12000, "completionTokens": 900 },
  "cache":      { "similarity": 0.94, "ttl": "PT30M" },
  "guardrails": { "contentSafety": true, "groundedness": 0.70,
                  "requireCitations": true, "humanApprovalOnWrite": true }
}

Declarative configuration earns its keep three times over. It is reviewable — a security reviewer reads the scopes and the guardrail profile rather than auditing somebody’s resolver code. It is enforceable at the platform boundary rather than trusted to each team’s diligence. And it is the natural unit of attribution: every call carries its workflow identifier, so cost, latency, cache hit rate and quality all roll up per workflow without anyone instrumenting anything.

Model selection goes through the same indirection. A workflow names a capability tier; the platform resolves it to a specific Azure OpenAI deployment or an Azure AI Foundry model. That one layer of naming is why a model upgrade is a platform change rather than twenty coordinated team changes, and why a capacity problem in one region is a routing decision instead of an outage.

Everything enters through Front Door and APIM — TLS, WAF, JWT validation, rate limiting and per-workflow throttling — under a Zero-Trust posture. The orchestrator is written on the assumption that it only ever sees authenticated, well-formed traffic with a real identity attached, and that assumption is enforced somewhere it can’t be bypassed.

Groundedness, citations, and the human in the loop

The dangerous output is not the one that is obviously wrong. It is the fluent, well-structured, correctly-formatted answer that is entirely unsupported by anything the system retrieved. It reads exactly like the good ones.

Four layers sit between the model and the user, and each catches a different failure:

• Content Safety on both sides. Input and output are screened, because prompt content arriving from a document is as much an untrusted input as prompt content arriving from a user.
• Citations as a hard requirement. Every claim-bearing sentence must be attributable to a retrieved chunk, and the orchestrator returns those citations alongside the answer so the interface can show them. An answer that cannot be cited is not a formatting problem to paper over — it is the system telling you it made something up.
• Groundedness scoring. A second pass scores the answer against the context it was actually given. Below the workflow’s threshold, the platform doesn’t return the answer. It returns “I don’t have enough to answer that,” plus what it did find and where. Refusal is a feature. A platform that cannot say no cannot be trusted with anything that matters.
• Human review before any write. Read workflows can be autonomous. Workflows that write back into a system of record produce proposals, and a person approves them. The approval step is enforced by the platform from the registration, not implemented per workflow.

The measure of an enterprise AI platform isn’t how often it answers. It’s how reliably it declines.

What actually makes it a platform

The band running along the bottom of the diagram is easy to skim past, and it is the part that separates shared infrastructure from a library four teams happen to import.

• Tracing, end to end. Application Insights spans stitch together question, retrieval query, candidate chunks, cache decision, model call and guardrail verdict into one trace. When somebody reports a bad answer, you can replay exactly what the model was shown. Without that, quality work is guesswork dressed up as iteration.
• Token budgets per workflow, enforced. Not advisory limits in a wiki. A runaway prompt in one workflow cannot consume the capacity every other workflow depends on, because the platform refuses the call.
• Cost attribution. Because every call carries its workflow identifier, spend rolls up per workflow and per team. That is what turns “should this one use a larger model?” from an argument into a decision somebody can make with a number in front of them.
• Evaluation gates in CI. Each workflow maintains a golden set — representative questions, the chunks that should be retrieved for them, and the characteristics a good answer has. It runs on every prompt, model or retrieval change. A change that regresses retrieval on the golden set does not ship.

That last one is the payoff for everything above it. Because retrieval is shared, a better chunker or a hybrid-search tuning improvement lands once and every workflow inherits it — and the golden sets are what make that safe to do. In the twenty-separate-stacks world, the same improvement is twenty tickets, and the honest number that get done is closer to four.

Black Friday: why peak load is the honest test of shared infrastructure

My first exposure to consumer goods had nothing to do with AI. Between 2006 and 2009 I led FAST — field force automation and sales tracking — J2ME clients on the cheapest handsets on the market, syncing to ASP.NET services, for domestic and international pharma and FMCG customers. Reps walked their routes with a phone, took orders at the shelf, and recorded what they found there.

The handset was never the hard part. The hard part was that the answer a rep needed while standing in the aisle — is this SKU on deal at this account, what have we actually committed to them, does this display meet the terms we are paying for — lived across a sales system, a contract, a planogram and a PDF nobody was carrying. Twenty years later that is still the shape of the problem. Only the retrieval layer has changed.

One clarification before the architecture, because it is where most retail AI writing goes wrong: a consumer-goods manufacturer does not run Black Friday. The retailer does. The manufacturer’s Black Friday happens in the months before it and the weeks after — negotiating trade terms, planning and funding promotions, committing allocation against a forecast, verifying that the execution the contract paid for actually appeared in the store, then settling the claims and deductions that follow. That sell-in and execution side is where the documents live, where the entitlement problems are, and where a shared retrieval platform earns its place. An architecture aimed at checkout and recommendations is answering a different company’s question.

Now the useful part. Every decision in this piece is defensible at steady state, and a little abstract there too. Under a concentrated peak they stop being abstract, because peak withdraws the slack that was quietly hiding what they were for.

Peak doesn’t introduce new failure modes. It withdraws the headroom that was absorbing the ones you already had.

• The semantic cache stops being an optimisation. At steady state it saves money. In a seventy-two-hour promotional window it decides whether you stay up. Question shape collapses under a peak — a few hundred distinct questions asked thousands of times, because everyone is asking about the same forty SKUs at the same dozen accounts on the same day. That is close to an ideal cache profile, and a platform that files caching under “later optimisation” ends up building it during the event.
• The freshness generation stamp becomes the whole design. This is the one that flips hardest. In a fleet business a fact changes and somebody asks about it within minutes. In a promotion window, prices, allocations and stock positions change hourly, and a cached answer that was correct forty minutes ago is now a wrong price being quoted to a retailer. A TTL cannot express that; it can only be tuned short enough to be useless or long enough to be wrong. Bumping the generation stamp on write — the detail three sections up that looks like housekeeping — is what makes an aggressive cache safe under precisely the conditions that make an aggressive cache necessary.
• Entitlement filters change legal category. In fleet, the filter enforces internal policy. In consumer-goods trade it enforces a contract: what one retailer is paying for is confidential from every other retailer. Someone on one account retrieving another account’s promotional allowance is not an internal embarrassment, it is a breach of a supply agreement with a named counterparty. The control is identical — indexed entitlement tags, filter composed into the query, never post-filtering — but the price of the version that depends on remembering goes up by an order of magnitude.
• Human approval before writes becomes a pricing-compliance control. A wrong read-only answer during a promotion is a bad day. A wrong answer that writes — a committed allocation, a confirmed promotional price, an approved deduction — is a financial event with a paper trail, and in some markets a regulatory one. Proposals plus approval, enforced by the platform from the registration rather than implemented per workflow, is what makes it defensible to put agents anywhere near this.
• Token budgets stop being tidiness and start being availability. At steady state a per-workflow budget prevents an embarrassing invoice. At peak, shared model capacity is the scarce resource, and one runaway workflow consuming it is an outage for every other workflow on the one day when every workflow matters.
• The eval gates are what let you ship during change freeze. This is the argument I would lead with in a retail organisation. Most of them freeze changes for weeks ahead of peak, and the freeze exists because nobody can demonstrate that a change is safe — so the only available control is no change at all. A golden set per workflow, running in CI, converts “we think it’s fine” into evidence. That is the difference between being frozen out of fixing a retrieval bug on the highest-revenue weekend of the year and being able to prove the fix is a fix.

Two things I would add for this vertical that the fleet build never needed. The first is a deliberate degradation ladder. At peak you want the platform to fail in an order you chose in advance — lean harder on cache, drop to a cheaper model tier, queue write proposals for later approval, and finally refuse with a message that tells the caller what to do instead — rather than discovering its own order under load. Declarative registration is the right home for that, because degradation policy is exactly the kind of decision that should be reviewed before the event rather than improvised during it.

The second is a rehearsal. Last year’s question distribution is a gift: replay it against the current index and the current golden sets a month out, at the volume you expect, and you find the retrieval gaps and the cache misses while there is still time to fix them. Every organisation I have watched go into a peak has load-tested the web tier and not the retrieval layer — and the retrieval layer is now the part that answers the questions.

None of that is a different architecture, which is the point of this section. A platform built for a fleet and mobility business survives a consumer-goods promotion peak because the peak stresses properties it already had: freshness that invalidates rather than expires, entitlements that cannot be forgotten, refusal that works, and evidence that a change is safe. If those four only hold at average load, they were never properties. They were conditions.

Outcome, and what I’d keep

Twenty-plus production AI workflows now run on one retrieval framework instead of a dozen divergent stacks. Adding a workflow is a registration, a scoped index and a prompt — not a project with its own security review, its own cost surprise and its own private opinion about chunking. Security reviews the registry. Cost rolls up by owner. Retrieval improvements are platform-wide by default.

Doing it again, four decisions I would make identically:

• Hybrid retrieval from day one. Pure vector search demos beautifully and fails on identifiers, which is most of what enterprise users actually ask about.
• Entitlements as indexed fields, filtered inside the query. Post-filtering is a control you have to remember; a query filter is a control you cannot forget. And retrofitting the fields is a reindex.
• Freshness as a per-fact-class design decision. Some things belong in the index, some things must be read live at answer time, and pretending there is one answer for both is how you ship a confidently stale assistant.
• The registry as the contract. Declarative configuration is the specific thing that turned this from a library teams import into infrastructure teams rely on.

And two I would change. I would start the golden sets before the first workflow ships, not after the first quality complaint — every workflow that launched without one had to be retrofitted with evaluation later, under time pressure, by someone reconstructing what “good” had meant. And I would push back harder, earlier, on RAG as the default answer. High-volume structured aggregation belongs in SQL. Deterministic lookups belong behind an API. RAG earns its considerable complexity when the question is open-ended and the evidence is scattered across documents nobody has time to read — and a platform that makes RAG easy makes it far too easy to reach for when something simpler would have been correct, cheaper and faster.