Enterprise Logs Analyzer Agent. A generic agentic AI design where the agent investigates, people steer and policy acts.

Status: reference design. An independent design written from public platform documentation and general practice. It describes no organisation’s systems, data, vendors or metrics, and every example is generic. Targets are proposals to confirm in a pilot; every number is either derived from a stated formula or labelled as an assumption. Nothing here is deployed as described. Try it: the interactive walkthrough lets you pick a failure, turn the autonomy dial and press the buttons on the brief.

0. The thesis in one paragraph

Operational failures are an evidence problem before they are a reasoning problem. When a scheduled job fails, the expensive part is not reading the log. It is finding the right lines: the ones this run wrote, on this host, between this start and this end, in a file that never mentions the run. Get that join right and most failures are a lookup. Get it wrong and the best model in the catalogue reasons confidently over the wrong evidence. So this design spends its determinism where failures hide (attribution, settling, signatures and storms) and its agents where judgement helps: investigating, explaining and drafting the fix. People steer those agents from the incident thread, and a signed policy, not the model, decides what runs.

Agentic where judgement helps; deterministic where it must be right.

1. The problem, stated plainly

Operational failures share one shape: the evidence sits in systems that do not share keys, and the people who can read it are busy. The worked example here is an estate of batch applications under a commercial workload scheduler. The logs land in two places, text files on a file share and rows in relational log tables. Jobs fail every day. Diagnosis means opening the scheduler, finding the run, finding its log, reading it, and remembering whether this is last week’s failure. That knowledge lives in people’s heads, and nobody can rank causes by the hours they cost.

Six facts shape every decision that follows.

1. Logs do not know their run. Few batch apps print the scheduler’s run ID. A line belongs to a run only because of where and when it was written.
2. Events and logs travel separately. The scheduler reports a failure on one channel and the log lines arrive on another, seconds to minutes apart and in either order.
3. Failures cluster. One shared outage fails dozens of jobs at once, each logging it in its own words.
4. New failures often follow a change. A deploy, a definition edit, a rotated credential or an upstream format change usually sits between the last good run and the first bad one.
5. Some failures look like success. A run that exits cleanly after reading nothing, or hangs without ever failing, raises no alarm.
6. A wrong action is worse than no action. Rerunning a half-committed job can double-post data. Every automated write must be provably allowed, once.

The obvious agentic answers each fail on one of those facts:

Constraints taken as given. Applications, the scheduler and the databases stay in the datacentre. Every request from the datacentre into the cloud enters through one API gateway. Nothing in the cloud holds credentials to the scheduler or the databases. The reference deployment runs on managed Azure services with models from the Foundry catalogue, and every part has a portable equivalent (section 13).

2. Seven principles: what makes it agentic, and what keeps it safe

Each principle is enforced by a test, not by a promise.

1. A deterministic skeleton with agentic steps. A durable workflow owns each incident from open to close. Agents do only the open-ended parts: investigating, planning and talking with people. Workflows follow predefined code paths, agents direct their own steps, and the simplest form that works wins (Building effective agents).
2. Evidence before reasoning. Attribution, settling, signatures and folding finish before any model is called, so an agent reasons over the right lines.
3. Read-only tools for every model-driven agent. Agents reach evidence through MCP servers that can only read. Actions flow through the policy engine and signed orders, never through a tool call.
4. Confidence is computed, never self-reported. A calibrator turns facts the code can check into the probability that a diagnosis is right.
5. People steer in conversation. Every incident has a thread where anyone can ask why, see the evidence, approve, take over or mark the answer wrong. Owners set an autonomy dial per job family and failure class.
6. One generic core, many domain packs. The loop, the guardrails and the interaction know nothing about batch jobs. A domain pack supplies the adapters, join rules, taxonomy and actions for one kind of work.
7. Everything is replayable and measured. Every agent step is traced, every diagnosis can be replayed on the same evidence, and nothing acts until replay and shadow runs show it is right.

3. Architecture at a glance

The edge ships every line, redacted, through the one API gateway into the evidence store, and the correlator turns lines and scheduler events into incidents. A durable workflow then runs four agents: an investigator that explores the evidence with read-only tools, a planner that maps the diagnosis to a playbook, a concierge that talks with people in the incident thread, and a curator that drafts playbooks from human fixes. The verifier checks every answer, the signed policy sets the tier, and the only path back to the scheduler is an order the datacentre fetches and verifies itself.

The interactive walkthrough lets you open every part, follow four failure scenarios step by step, and turn the autonomy dial to see what changes.

4. Agents, tools and memory

Four agents do the open-ended work. Each is bounded by what it may read, how long it may run and what it may return, and none holds a tool that changes a system.

Tools are served over the Model Context Protocol, so the same servers work with any agent framework and with an interactive operations agent bought off the shelf. In the current revision, 2026-07-28, each server is an OAuth 2.1 resource server that accepts only tokens issued for it and passes no token on (MCP authorization).

Memory comes in four kinds, each in the store that suits it.

• Working memory: the incident record and its evidence bundle, kept as long as the incident.
• Episodic memory: confirmed past incidents with their causes and fixes. Only a diagnosis a person confirmed enters it, so a crafted failure cannot teach the agent.
• Semantic memory: job definitions, the dependency and resource graphs, owners and runbooks.
• Procedural memory: playbooks as signed code, and the policy bundle.

5. The incident loop

A durable workflow runs every incident through fixed states: settling, triage, investigating, deciding, acting or proposing, verifying, closed. Each transition is a checkpoint, so a restart resumes where it stopped and no step runs twice. In the reference deployment the workflow and the agents run on Microsoft Agent Framework 1.0, whose graph-based workflows checkpoint and pause for human approval (Agent Framework 1.0).

• Budgets. The workflow owns every budget: tool calls, wall clock and tokens per step. An agent cannot extend its own. Exhausting one ends the step with what it has, marked incomplete, and the incident falls to people rather than to a guess.
• Pauses for people. The workflow waits at three points: a proposal awaiting approval, a person taking over, and an incident nobody owns.
• Traces. Every step records the agent, its tool calls with argument hashes, the line IDs returned, tokens and latency, in the OpenTelemetry conventions for generative AI, so any tracing backend can read them.
• Replays. Every input is recorded, so any diagnosis can be re-run on the same evidence with a new model or prompt. That is how every change is tested (section 12).

6. The batch pack: six hard problems, six answers

Everything above is the generic core. This section is the worked example, the batch domain pack, and each answer sits where its information already lives.

Detection starts it. Seven triggers open an incident: failed, aborted, overran (elapsed time above the P95 of the last 30 runs on the same weekday plus three median absolute deviations), hung (no new line for three times the job’s 99th-percentile gap, and at least 10 minutes), never started (expected start plus an owner-set tolerance, holiday calendar applied), bad success (ERROR lines, a failed output assertion, or a duration under 10% of its median) and at risk (the remaining critical path at P90 durations passes a deadline). Owners declare three things per job family when they onboard: deadlines, output assertions where “success” can lie, and the family’s rerun-safety class.

6.1 Attribution: one rule list, applied centrally

The correlator attributes every line using the run windows the scheduler reports. The rules run in order, and the first that fires sets the strength.

ε is the host’s measured clock offset plus one second. Lines that match nothing count against the family’s attribution coverage, and automation for a family waits until at least 90% of its failed runs have exact or window evidence. Attribution lives in one versioned service rather than on every host, so a rule fix can re-attribute history.

6.2 Completeness: decide on evidence, not on a timer

A failure opens the incident in settling state, and triage waits for two checkable facts. First, the host’s heartbeat must show an empty send queue at a time later than the run’s end plus a 90-second grace for the last writes. Second, two evidence reads 10 seconds apart must return the same lines, which covers any ingestion delay in the store. A hard cap of 5 minutes releases the incident with a logs_incomplete flag rather than waiting forever. This is event-time processing with allowed lateness, the idea behind watermarks in the Dataflow model.

Time is handled where it is known. The edge converts each line to UTC with the host’s own zone rules; within a file, order comes from the byte offset, because the daylight-saving fall-back hour repeats timestamps; continuation lines take the timestamp of their record’s first line; and a host whose clock offset exceeds the guard has its lines downgraded to ambiguous, so it can never drive automation.

6.3 Signatures: name the failure so it can be counted

A signature must stay the same when only the variables change, and change when the failure changes. Stack traces get an exception fingerprint: a hash of the exception type and the top three frames from the job’s own code, with line numbers removed. Messages get a template ID from streaming template mining, with numbers, GUIDs, paths, host names and timestamps masked (Drain3). Error codes become fields. The signature is the fingerprint when a stack trace exists, otherwise the template ID plus the error code, and a coarser family key tracks recurrence across small rewordings. A language model may draft parsing rules for a new log format offline, for review, but never parses lines at runtime.

6.4 Correlation: one incident per cause, not per symptom

The correlator keeps a resource graph of jobs, hosts, databases by connection alias, file shares, service accounts and upstream feeds. Three or more failures on different jobs that share a resource within ten minutes fold into one resource incident with every member run listed. A job blocked because its predecessor failed joins the predecessor’s incident instead of opening its own. Before any action, the policy asks whether the run belongs to an open resource incident; if it does, single-run actions wait and the recovery plan governs. The same signature on three of the last seven days opens a problem record for the owner.

6.5 Change and impact: what changed, and what it will cost

A change feed collects deploy events, the scheduler’s definition audit, configuration commits, schema changes, patch windows, and credential and certificate rotations with their expiry dates. For every incident, the changes made between the last good run and this failure that touch the job, its host, its resources or its upstream jobs are ranked by graph distance, then recency, and the top five go to the investigator, which must say whether one is implicated. Impact comes from walking the dependency graph downstream: only incidents whose predicted finish at P90 durations passes a deadline, or that touch a critical service, page anyone. The same data gives prevention: credentials and certificates expiring within 14 days, and durations trending toward a deadline, reported weekly before they fail.

6.6 Trust: the model advises, code decides

The action gate never reads the model’s own confidence. It computes evidence confidence from facts the code can check: the cited root line exists in the bundle; attribution is exact or window and the logs are complete; the class agrees with the rules classifier or with a human-confirmed diagnosis of the same signature; repeated samples agree; and an approved playbook resolved the signature before. A calibrator fitted on the replay set maps those features to the probability that the diagnosis is right. Self-reported confidence is excluded because models stating their confidence tend to be overconfident, while agreement across responses carries signal (Xiong et al., ICLR 2024). Meta’s incident tooling makes the same trade and withholds low-confidence answers, “sacrificing reach in favor of precision” (Meta Engineering, 2024).

7. Diagnosis: one bounded agent, one strict answer

The automatic path is a durable workflow with one bounded agent in it, not an open-ended agent. The evidence favours the bound. Microsoft’s RCACopilot collects diagnostic data with handlers matched to the alert type, then has a model predict the root-cause category with an explanation, reaching accuracy up to 0.766 on a year of real incidents (RCACopilot, EuroSys 2024). Open-ended agents remain weak at the task: on the OpenRCA benchmark of 335 real failures, the best model with an agent framework solved 11.34% (OpenRCA, ICLR 2025).

1. Rules first. A rules classifier runs on the evidence. If it is decisive and an approved playbook matches the signature, the incident goes to tier 0 with no model call at all.
2. Build the bundle, deterministically and within a token cap. Every line carries a stable line ID: the run header, the captured output and attributed lines (the first ERROR or FATAL line with its stack trace, 20 lines either side, the last 50 lines), the difference from the last good run, host events and metrics, the top five changes, the last 14 runs and the five most similar confirmed incidents. KQL’s reduce groups lines into patterns and diffpatterns compares a failed set with a good one.
3. Investigate, then answer in a strict schema. The investigator starts from the bundle and may call the read-only tools to look further: the full log around a line, an older run, a change’s details, a similar incident. Its last step is one call to the model deployment under a strict JSON schema: {root_line_id, failure_class, cause_statement, implicated_change_id, evidence_line_ids[], recommended_step, owner_question}. The class and step are enums, so nothing free-form reaches the policy. Strict structured outputs come from the model API; the hosted agents service does not support them (structured outputs).
4. Verify. Every cited ID must exist in the bundle, and the brief renders line text from the store, never from the model. A missing ID, a schema failure or a content-filter block makes the class unknown.
5. Calibrate and decide. The calibrator turns the evidence features into a probability, and the policy engine sets the tier.
6. Record. Each diagnosis is stored with the model version, prompt version and a hash of its bundle, so any brief can be reproduced.

If the model is unavailable, over budget or filtered, the brief still goes out with the rules result and the evidence: the system degrades to a better log viewer, never to silence.

The taxonomy is drawn by who acts and whether a retry can help, not by technology.

8. Remediation: tiers, safety classes and evidence

An incident drops to the lowest tier whose every condition holds, and the system improves by moving signatures down the tiers. Every condition is a check in code; the model’s output is one input to those checks, never a check itself.

Rerun-safety classes. Owners declare one class per job family, and the allow-list can never grant more than the class permits: idempotent (a rerun from the start is always safe), restartable (safe from a checkpoint, with a second approver), cleanup_first (a registered cleanup runs first, as one plan with a second approver) and never_auto (postings, payments, anything whose duplicate is a business error: briefs and proposals only). A family not yet classified is never_auto.

Playbooks are code. A playbook is a signed, versioned file in Git. It declares what it matches, its preconditions, its steps, its post-conditions and what happens if they fail. Steps come from a fixed vocabulary: wait_for_file, wait_for_health, hold_downstream, release_downstream, rerun_job, rerun_from_step, rerun_chain, run_registered_cleanup, open_ticket, notify, page. CI validates every change with a schema check, policy tests and a replay against every past incident the playbook would have matched. Tier 0 looks playbooks up by key, never through similarity search.

Promotion needs evidence. An allow-list entry starts in shadow, then proposes, then acts. It earns automatic status after n correct proposals in a row, where n is the smallest run of successes that rules out a success rate below the target p at 95% confidence: n = ⌈ln 0.05 / ln p⌉.

A fixed ten in a row would demonstrate only about 74%. Any wrong action demotes the entry to proposals and opens a review, and a playbook whose rolling success rate falls below its target is demoted automatically.

From order to outcome. The remediation service turns the playbook into ordered steps and signs each as a JWS over every field (order, incident, run, step, arguments, policy version, not-before, expiry, approvers) with a key held in Key Vault. The gateway in the datacentre long-polls the API gateway for orders, so nothing in the cloud opens a connection inward. It checks the signature against its pinned key, the expiry, that the order ID has never run, and the decision under the same signed policy bundle the cloud used. It writes executing locally before calling the scheduler, reconciles after a crash by asking the scheduler for runs started since the order arrived, and closes the incident only when the rerun succeeded and the output assertions pass.

Rate limits. One automatic rerun per run, ever. At most ten independent tier-0 actions per hour per tenant. An approved recovery plan runs outside that cap under its own concurrency limit, five at a time by default, so a forty-job outage recovers in minutes, not hours.

9. People steer the agent

People steer the agent from where they already work: a thread in chat for every incident, a console for depth, and the ITSM tool for the record. A brief succeeds if the on-call engineer can act on it in under a minute. A conversation succeeds if every answer points at evidence and every request for action becomes a proposal the policy can judge.

The brief. Every brief follows one contract in about ten lines: the headline (job, what happened, when, the impact); the root line, rendered from the store by ID, with a link to the full run log; the class and a one-sentence cause; what changed, when a change is implicated; history (“4th time in 30 days; last fixed by”); the recommended step; confidence as a band from the calibrated probability; and owner, ticket and incident links.

9.1 Every brief is a control panel

Each brief is an interactive card, so its buttons work inside the thread and the card updates in place. On Teams, Universal Actions return a fresh card after each action and can show different buttons to different people, so only an eligible approver sees Approve (Universal Actions).

9.2 Ask the agent

Anyone in the thread can ask the concierge in plain words. It answers from the same read-only tools as the investigator, under the asker’s own permissions, and it never acts: a request to do something becomes a proposal that the policy judges like any other.

9.3 The autonomy dial

Owners set how far the agent may go for each job family and failure class. The dial is the tiers seen from the owner’s side, and policy caps it: the family’s safety class, the class’s eligibility and the promotion evidence each set a ceiling the dial cannot pass.

Turning the dial down takes one person and works at once. Turning it up takes the owner and the platform owner, and only to the level the evidence supports. The policy always reads the lower of the dial and the evidence ceiling.

9.4 The console

The console is where people go when the thread is not enough: an incident timeline of every state change, agent step and approval; an evidence viewer with the cited lines highlighted beside the last good run; replay of any diagnosis on the same evidence with another model or prompt; a playbook editor that simulates a change against past incidents before review; a policy what-if that says whether an action would be allowed and, if not, which rule stops it; and the dials, the ledger and the evaluation results over time.

Briefs post to the owning team’s channel and update in the same thread; one ticket per incident lives in the ITSM tool; pages go only to at-risk deadlines and critical services; low-impact incidents and everything resolved automatically go to a daily digest; and a weekly causes report ranks causes by engineer-minutes lost rather than by count, which puts the expensive recurring failure ahead of the noisy cheap one. Cards and the console target WCAG 2.2 AA.

10. Domain packs: one core, many kinds of work

The loop, the guardrails and the interaction know nothing about batch jobs. A domain pack supplies the eight things that do.

Other packs plug into the same core. Each brings its own join key, which is usually easier than the batch case because the logs already carry it.

11. Security: what the agent can never do

Nothing a model writes can change a system. The only path to a change is an order that code has checked against a signed policy, signed with a key held in Key Vault, and fetched and verified by the datacentre itself. Trust changes hands in five places, and each has its own check.

• Datacentre to cloud. Redacted lines and events travel on a connection the datacentre opens; the API gateway checks a per-host certificate, a token and a quota.
• Log text to the model. Text from any application is untrusted. It is redacted, delimited and declared data before the model sees it.
• Model output to decisions. The model returns advice as line IDs and enums. Code verifies and calibrates it, and the signed policy decides.
• Cloud to the datacentre. Orders are signed, expiring and single-use. The gateway fetches them and re-checks each one against the same signed policy.
• People to the agent and to approvals. People talk to the concierge with their own permissions; every approval carries an identity, and a second approver joins where the safety class requires one.

Two OWASP lists apply. The Top 10 for LLM Applications, 2026 edition, covers the model (CSA research note); its top three are prompt injection, sensitive information disclosure and excessive agency. The Top 10 for Agentic Applications, released in December 2025, covers what agents do with tools, memory and each other (OWASP). The structure of the design answers the worst of both: no model-driven agent holds a tool that acts, and nothing a model writes is executed.

The rest is conventional and complete: managed identities only and no secret in code or settings; private endpoints for every platform service; redaction at the edge, a second pass at ingest, and a labelled test set that must keep redaction recall at or above 99%; append-only ledger tables for orders, results, approvals and policy versions, with digests in immutable storage; each active playbook registered as a pre-approved standard change; segregation of duties (a playbook’s author cannot approve it); and kill switches that stop automation, proposals or the whole loop while ingestion continues.

12. Operations and evaluation

The agent is a production service with its own objectives, watched by a job that fails on purpose, and allowed to act only after replay and shadow runs show it is right.

Watching the watcher. Every edge supervisor posts a heartbeat every 30 seconds, and a host silent for two minutes raises an alert and marks its open incidents logs_incomplete. A sandbox canary job fails every hour with a known signature; its brief must arrive within five minutes with the right class, and its playbook must rerun the sandbox job through the gateway, which tests the whole path. A dead man’s switch outside the pipeline fires when the scheduler counts failures but no incidents open. The scheduler’s native alerts stay on, so if everything here fails, operations fall back to today, not to nothing.

Tracing the agents. Every agent step is a span: the workflow run is the root, each agent invocation a child, each tool or model call a grandchild, following the OpenTelemetry conventions for generative AI (semantic-conventions-genai). Spans carry the incident ID, agent, tool, argument hash, returned line IDs, tokens and latency, never raw log text. Dashboards show cost per incident by agent, trajectories that hit a budget, tool errors by server, and the questions people still have to ask the concierge, which say what the brief should already contain.

Evaluation. The last ninety days of failures are a test set waiting to be labelled: the scheduler knows which runs failed, the logs are still on disk, and tickets and reruns record what people did. Every failed run is replayed through attribution, settling, correlation and diagnosis. Two engineers label a stratified sample independently, and labels are trusted only once agreement reaches Cohen’s κ ≥ 0.7. The set is split once: a calibration part, and a held-out 20% that nothing is tuned on. Metrics are root-line exact match, class accuracy and macro-F1, precision of recommended actions, expected calibration error (target ≤ 0.05), coverage, and attribution coverage per family. To measure an accuracy p to within ±E at 95% confidence takes about n = 1.96² · p(1 − p) / E² cases: 35 per class for 90% to ±10 points, 139 overall for ±5 points, 385 in the worst case.

Then two weeks of shadow on live traffic before anything acts, with would-have actions compared against what people actually did. After that, every change to the model, prompt, taxonomy or parsers runs as a challenger beside the live version and is promoted only if it is not worse on root-line match and calibration, judged by a paired test such as McNemar’s on the same cases. Every replayed incident also scores the investigator’s trajectory (the right run window, the calls it used, whether every claim cites a line), and twenty briefs and twenty concierge answers are audited each week, sampled at random rather than by complaint.

13. Portable by design

Every part is chosen for the reference deployment, not welded to it. The contracts, MCP for tools, OpenTelemetry for traces and Open Policy Agent for policy are open standards, so moving a part changes an adapter, not the design.

14. Rollout: gates, not dates

The first automatic action comes after four measured gates, not after a date. A gate that is not met extends its phase, and today’s alerting stays on throughout.

The concierge goes live, read-only, in the shadow phase, so people learn to steer the agent before anything acts. The tenant key is in every schema from day one, so a shared platform for other teams later extends these contracts rather than rewriting them.

15. Honest limits

• Attribution fails where runs overlap. Two concurrent runs of one job writing to one shared file stay ambiguous. The fix is in the app (log the run ID); the design makes the gap countable, not invisible.
• A verified root line is not a verified cause. The check proves the cited line is real, not that the model read it correctly. Class accuracy on the replay set is the only measure of that.
• Calibration is only as good as its labels. A class with too few labelled cases stays at tier 2 until it has enough.
• Maps go stale. Storm folding and change correlation are only as good as the resource graph and the change feeds.
• Promotion evidence ages. A major release of a job resets its entry to proposals.
• Idempotency is a human claim. The design trusts the owner’s safety class. A wrong class is caught by audit, not prevented.
• An agent can wander. A tool-using investigator may chase the wrong thread within its budget. The budget bounds the cost, not the quality; trajectory evaluation is the check.
• Conversation invites over-trust. A fluent answer reads as a right one. Answers cite line IDs and show a calibrated band, and the agent never reports a fix as done until a post-condition passes.
• Dials drift upward under pressure. Raising autonomy takes two people and evidence, and every raise appears in the monthly evidence pack.
• Generic is not free. Each new domain pack needs its own join rules, taxonomy and golden set. The core saves the loop, the guardrails and the interaction, not the domain work.
• The agent does not reduce failures. It shortens them and ranks them. Fewer failures come from owners fixing the top causes.

Closing

The agent investigates, people steer, policy acts. That sentence is the whole architecture: agents where an open question needs exploring and explaining, deterministic code where a join, a gate or an order must be right every time, and people in the loop through the place they already work, with a dial that only they can turn up. Batch failures are the worked example because they are the hardest join; the same core carries pipelines, deployments and service incidents once a pack supplies their keys.

Sources

• Building effective agents, Anthropic
• Microsoft Agent Framework 1.0
• MCP authorization, revision 2026-07-28 and MCP versioning
• OWASP Top 10 for Agentic Applications for 2026 and a summary of its ten entries
• OWASP Top 10 for LLM Applications 2026, CSA research note
• OpenTelemetry semantic conventions for generative AI
• Universal Actions for Adaptive Cards
• Structured outputs in Microsoft Foundry
• RCACopilot, EuroSys 2024
• OpenRCA, ICLR 2025
• Can LLMs Express Their Uncertainty?, ICLR 2024
• Leveraging AI for efficient incident response, Meta Engineering
• The Dataflow Model, VLDB 2015
• Drain3
• Azure SQL Database ledger
• Azure SRE Agent overview
• LangGraph persistence