From Observability to Agentic Operations

Investigation in a distributed estate is mostly clerical. The evidence exists, but it sits in systems that do not talk to each other, and the first twenty minutes of an incident go on assembling it rather than thinking about it — a PromQL query in one tab, a log search in another, a trace viewer in a third, audit records elsewhere. Every pivot is manual, and it falls under time pressure on whoever picked the incident up, support analyst or on-call engineer.

What We built

Kagent runs the agents on Kubernetes and OpenShift, declared as resources alongside the workloads they investigate. It owns the agent loop, the tool wiring and the agent's Kubernetes workload identity, so an agent is deployed, versioned and governed like any other workload.

The MCP servers are the integration layer. Each one exposes only the bounded operations a support investigation needs, against one system, scoped to a tenant, with a read-only credential of its own. Those systems vary by engagement: metrics in Prometheus, logs and often traces in Elastic, traces in Jaeger elsewhere, audit records in a database of their own. The agent asks the same question regardless and absorbing that variation is what the layer is for.

The reason to put an agent there is narrow but real. A dashboard answers a question somebody already knew to ask. An investigation is an evolving question: the second query depends on what the first returned. An agent chooses that next query, runs it against a different system, and returns a hypothesis with the evidence behind it.

What an Investigation Looks Like

Latency rises on a payment API. The agent starts in Prometheus. The regression is real and bounded to one service, but the metric cannot say which requests were slow. It pivots through an exemplar to a trace ID, pulls that trace, and locates the span where the time went. A downstream call, not the service that raised the alert. It then runs an ES|QL query for the downstream service's logs on the same trace ID, and checks audit records for changes in the window.

What comes back is not a verdict. It is a prioritized list: the regression, the implicated span, the correlated log entries, and a change that landed nine minutes earlier — proximity, not proof — each with the query that produced it, so the engineer can disagree with any step. The agent removed the assembly, not the judgement.

Why the Telemetry Has to be Disciplined First

Metrics should not use request identifiers as high-cardinality labels: Prometheus's own documentation warns against them, because every distinct value becomes another time series. So, the route from a metric anomaly to a single request runs through exemplars and trace IDs. Traces span only the services that propagate trace context, and one hop that drops the header fragments the trace permanently. Logs join only where the trace ID was injected at write time.

The agent correlates what the platform recorded. It cannot recover context that was never instrumented, and an estate without propagation or exemplars does not become investigable by adding an agent.

Security, and Where the Model Lives.

Operational telemetry is not trusted input. Headers, error messages, trace attributes and event text can carry content an attacker influenced, and that content becomes part of the model's context, the mechanism behind prompt injection, the top-ranked entry in OWASP's Top 10 for Large Language Model Applications across consecutive editions.

So, the controls live in the architecture around the protocol. MCP standardizes how tools are described, invoked and returned; authorization, validation and audit are enforced around it. Tenant isolation is enforced in the query layer rather than asserted in a prompt, and we log what the model was shown, not only the tool it called.

The same architecture supports either deployment model. Where a client's regulations or contracts require inference to stay inside the controlled environment, the models run locally; where policy permits a public service, we use one, and nothing else changes. Local inference is not itself a security control, it governs where content travels, not whether a tool call can carry data out, but how it satisfies a requirement that is often absolute.

Controlled Autonomy

This system automates investigation, not remediation. Its agents hold read access to operational evidence and cannot restart a pod, scale a service, roll back a deploy or modify a workload. That is a boundary we engineered rather than one we ran into: acting on infrastructure is a different risk class from reading it, and the controls that would make it safe — reversible actions, blast radius fixed by the credential, a measured false-lead rate — belong in place before that boundary moves.

What we built is an agent that reasons across real operational signals, drawn from systems that were never designed to be queried together. Which makes the interesting question not how autonomous it could become, but how much authority should be engineered into it, and under what controls.