All articles
AI Governance13 min read

AI Agent Monitoring That Actually Works in Production

Practical AI agent monitoring guide covering static analysis, privacy-aware tracing, blast-radius review, alerting, and incident response.

By Jonny Freedman, CEO, Agntz

By Jonny Freedman · Published
AI Agent Monitoring That Actually Works in Production

The strongest advice on ai agent monitoring still gets the sequence wrong. Teams start with dashboards, alerts, and per-call logs, then wonder why a workflow looked healthy while it failed a handoff, invoked the wrong tool, or touched data it shouldn't have seen. The better approach starts earlier, at the source-code and dependency layer, where hidden capabilities, vendor references, and unregistered components can be mapped before the first live run.

That inside-out view matters because runtime visibility alone can't tell a security lead what was possible in the first place. Agntz public copy says more than 10,000 open-source agent repositories have been scanned and classified. That scale claim does not establish which governance gap is most common, and current product evidence should not be read as a complete resolved transitive-dependency graph. The practical point is narrower: if the inventory misses a model, tool, vendor, or data path, every later metric is built on a blind spot. shadow AI in your codebase

Table of Contents

Why AI Agent Monitoring Has to Start Before Runtime

A runtime-first program assumes the agent already exists in a trusted state. That assumption breaks as soon as a repository pulls in an embedded model, a hidden package call, or an MCP connection that no one reviewed. Monitoring then becomes reactive, which is exactly where regulated teams lose control of capability drift and unapproved data paths.

The more useful frame is inside-out versus outside-in. Inside-out monitoring begins with source code, declared packages, detected vendor references, and available dependency evidence. It produces a capability map and an AI Bill of Materials that show what the agent can reach, which vendors are visible in the repository, and where sensitive data could move. Because complete transitive edges are not yet persisted, teams should corroborate indirect dependency coverage with their package-management and build evidence. Outside-in telemetry still matters, but only after the inventory exists.

Practical rule: if a capability can't be named before deployment, it can't be trusted after deployment.

Source-level visibility earns its keep. A repository scan can reveal a tool binding, a permission path, or an indirect package or vendor reference long before runtime traces ever appear. That matters for teams trying to answer a basic governance question, what AI capability exists at all, and which controls already apply.

Agntz fits that pre-runtime layer because it reads source code to inventory AI capabilities, map them to controls, and generate evidence-backed findings. For teams trying to produce the first capability map quickly, the free first scan at Agntz is one way to turn an unknown codebase into a reviewable capability map.

Telemetry You Cannot Afford to Skip

The minimum useful unit is the session, not the isolated model call. A practical ai agent monitoring workflow should start with session-level tracing rather than isolated LLM calls, capturing each run as a connected trace with the prompt, model response, every tool call plus its inputs, the tool response, and the final action taken, as described in Latitude's production monitoring guidance. Anything less leaves gaps between the model, the tools, and the outcome.

A complete trace should also carry the fields security teams need. That includes session ID, step-level latency, actor identity, data classification, tool or resource used, action taken, and whether something was redacted or blocked. Capture prompts, tool arguments, and results only under explicit data-minimization rules: redact sensitive values, restrict access, and define retention limits before collection. Without those controls, telemetry can create a second store of sensitive data instead of a reliable evidence trail.

A good baseline is easier to interpret when the trace is structured the same way every time.

Signal categoryWhat to captureWhy it matters
Trace structurePrompt, model response, tool calls, tool inputs, tool responses, final actionReconstructs the full path of a run
Identity and dataSession ID, actor, data class, redaction or block outcomeSupports audit and containment
PerformanceStep latency, token usage, tool-call errorsSeparates slow reasoning from broken integrations
OutcomeResolution success, policy compliance, abandonmentShows whether the agent achieved the goal

A trace should answer one question cleanly. What happened, in what order, and with what result. If the trace can't answer that, the alerting layer will guess. That guesswork is where most false confidence comes from.

Here's the main operational takeaway, instrument traces first, then derive metrics from those traces. Add targeted evaluation and alerts only after the baseline is stable. Teams that skip this step end up optimizing the wrong signal, usually per-call latency, while missing silent failures in the actual workflow.

A comparison chart showing the differences between static analysis and runtime tracing for AI agents.

Static Analysis Compared With Runtime Tracing

Static analysis and runtime tracing answer different questions, and teams need both. Static analysis reads the code and dependency graph before execution. Runtime tracing watches the agent while it's live. One gives inventory. The other gives behavior.

Static analysis is a strong control for capability discovery. It can map tools, permissions, data paths, and external services before the agent runs. That makes it useful for building an AIBOM, identifying hidden vendors, and attaching file, line, and call-path evidence to findings. It can also catch dormant or unregistered components that never surface in the live path.

Runtime tracing is the better control for execution evidence. OpenTelemetry's evolving GenAI semantic conventions provide common attributes and span patterns for models, agents, and tools, but the conventions remain under development. Teams should pin the version they implement and validate it against their own workflow model. That matters because agent failures usually happen in the middle of a tool chain, not at the request boundary.

DimensionStatic analysisRuntime tracing
ScopeSource, dependencies, permissionsLive execution, handoffs, tool use
Best answerWhat can this agent do?What did this agent do?
Audit valueInventory and provenanceReplay and root-cause evidence
WeaknessCan't prove live behaviorCan't reveal hidden capability without source context

The layered model is simple. Static analysis defines the blast area. Runtime tracing confirms what happened inside that area. Using one without the other creates a false sense of completeness.

Blast Radius Scoring That Security Teams Will Trust

Blast radius should be treated like a reach model, not a vague fear score. A security team needs to know which capabilities can touch which assets, how far a single failure can propagate, and whether that impact is reversible. The score only becomes useful when it is attached to concrete code paths and data classes.

Start with the capability map from source code. Then enumerate what each capability can read, write, invoke, or forward. After that, weight the result by sensitivity and reversibility. A capability that can read a ticket is not equal to one that can email a customer, change a record, and trigger another workflow.

A customer-support agent is a clean example. A calendar tool widens reach into scheduling data. A ticketing tool widens reach into case management records. An email tool adds outward propagation, because a bad action can leave the system boundary. Each added capability increases the set of assets that can be touched, and each asset adds a different audit burden.

The score should explain why a finding matters, not just that it exists.

The most defensible output is a finding attached to file, line, and call-path references. That gives auditors a reproducible trail and gives SOC analysts a way to isolate the capability that expanded exposure. A generic “high risk” label doesn't do that job.

For a separate, questionnaire-based triage step, the risk calculator can help structure an initial exposure discussion. It does not derive capability reach from source code and should not replace repository analysis or a capability map.

Monitoring for Prompt Injection and Indirect Prompt Injection

Prompt injection is not a side issue. It is a first-class monitoring problem because an agent can be steered into doing the wrong thing while still appearing to function normally. Direct prompt injection usually shows up in user input. Indirect prompt injection arrives through retrieved content, external documents, or other data sources that the agent treats as trustworthy.

The detection pattern is rarely subtle. Unexpected tool calls after a retrieval step are a red flag. So is instruction-like text embedded in content that should have been passive. Another warning sign is data egress that happens immediately after a document lookup or inbox read, especially if the agent had no reason to send that data anywhere.

The logging model needs to be strict. AI agent monitoring should produce a tamper-evident record of each agent action and each sensitive data element it touched, including who initiated the action, what data class was involved, which tool or resource was used, what action occurred, and what was redacted or blocked, as stated in Strac's monitoring guidance. That record is what lets investigators separate steering from exfiltration.

Three practical signals matter most:

  • Unexpected tool choice: the agent calls a tool that doesn't fit the task.
  • Instruction override in retrieved content: passive text starts behaving like a command.
  • Post-retrieval egress: data leaves the workflow right after a suspicious fetch.

The main control is not just blocking. It is traceability. Investigators need to see the actor, the data class, the tool, and the outcome so they can reconstruct intent and effect. The AI agent security risks guide is a useful companion reference for this threat class because it keeps the focus on code-path exposure and agent behavior together.

Alerting and SLOs That Reduce Noise

Alerting works only when it points to user harm or policy failure. That's why agent quality has to be treated as an SLO problem, not a generic uptime problem. A system can be up and still fail to complete the user's task, violate policy, or burn budget through retries.

Start with immediate alerts for known safety, security, and policy failures, then collect enough representative traffic to calculate p50, p95, and p99 for operational metrics before tuning SLO thresholds. Baselines help reduce noise, but they should never suppress alerts for unauthorized tool use, sensitive-data exposure, or containment failures.

Operational note: aggregate averages hide localized drops. Split by workflow type and failure mode or the worst regression will stay buried.

A useful metric set looks like this:

Signal categoryMetricWhat it tells you
QualitySession success rate by workflowWhether the agent completed the user goal
ReliabilityTool-call error and silent-failure ratesWhether integrations are failing without obvious crashes
PerformanceMulti-turn latency percentilesWhether long workflows are degrading
EfficiencyCost per completed sessionWhether the same outcome is getting more expensive
GovernancePolicy compliance and failure-mode coverageWhether the control set matches the risk profile

The control loop should combine sampled quality evaluations, human review of high-risk sessions, and feedback into the scoring model. That creates a path from detection to remediation. It also lets teams prioritize the sessions that are most likely to matter, instead of reviewing random traffic and hoping the signal shows up.

Incident Playbook for Cascading Agent Failures

A single bad action becomes serious when it propagates across agents, queues, or shared tools. Generic uptime playbooks don't fit that pattern. The right response is scoped containment, not fleet-wide panic. The team needs to know which agent acted, which dependency carried the error forward, and where to stop the spread.

A six-step process flow chart detailing the incident playbook for managing and mitigating cascading AI agent failures.

1. Detect the dependency chain

The first alert should come from cross-agent telemetry, handoff logs, or retry spikes in a workflow path. If only per-call metrics are watched, the chain will look like unrelated noise. Full-path telemetry is what reveals that one agent's output became another agent's input.

2. Scope impact with agent identity

Responder teams should use the agent identifier and its attached agent card to identify configuration, purpose, and expected behavior, as proposed in the agent card research. That makes it easier to distinguish a misbehaving subagent from a healthy upstream controller. It also shortens scoping when multiple agents share the same infrastructure.

3. Contain the smallest viable surface

Containment should focus on credentials, capabilities, or a specific workflow path. Pausing the whole fleet is usually too broad and creates unnecessary operational damage. Revoking a tool credential or disabling one action path is usually a cleaner first move when the blast radius is localized.

4. Roll back to the last known good state

Rollback only works if the team can replay what happened. Session traces, tool calls, and final actions become the evidence needed to reconstruct the last known good configuration. Teams should preserve the exact divergence point, not just the final failure.

5. Reconstruct root cause from the first bad span

Replay the logged interactions and isolate the first span where behavior diverged. That span usually tells the story. It might be a bad tool input, an unexpected retry loop, a corrupted retrieval result, or a permission path that reached too far.

6. Feed the failure back into the baseline

The incident should end with a named failure mode and a tracked control gap. That turns a one-off escalation into a measurable remediation item. A SOC lead can then see whether the issue was execution, governance, or both.

The gap this playbook fills is simple. Standard alerting shows that something went wrong. Cascading-agent response shows how the error moved, who it affected, and what to stop first. That's the difference between monitoring a service and defending an autonomous workflow.

How often should agent repositories be rescanned for drift?

Agntz currently checks active repositories daily, scanning changed source while reusing unchanged results where appropriate. Teams that require per-commit gates should add those controls in their own CI/CD workflow. The key is ensuring that changes in tools, vendors, or data paths re-enter review before release.

What if runtime traces contradict the source-level inventory?

Treat the contradiction as a governance signal, not a logging bug. The inventory may be missing an indirect dependency signal, an embedded model, or an unregistered component. Runtime traces may also show a path that only appears under certain inputs. Either way, the response is to reconcile source-level provenance with live behavior before trusting either view.

Are agent cards required for compliance?

They're not universally required by law, but they're useful for accountability and incident investigation. An agent card binds descriptive metadata to the agent identifier so responders can scope behavior faster. For regulated systems, that identity-linked provenance is often the difference between a clean evidence trail and a pile of raw logs that no one can interpret quickly.

Which agents should be instrumented first?

Instrument the agents with the widest data access, the most external tools, and the most branching logic first. Those are the ones most likely to create silent failures or downstream propagation. The next priority is any workflow that can read sensitive data, modify records, or trigger external actions that are hard to reverse.

Does Agntz replace runtime observability tooling?

No. It fills the pre-runtime inventory layer by reading source code, identifying declared packages and available dependency evidence, and mapping capability reach. Runtime observability still has to track traces, metrics, and incident behavior once the agent is live. The two layers solve different problems, and both matter in regulated production.


Agntz helps teams build the source-level capability map, AIBOM, and blast-radius evidence that should exist before an agent ships. If the current monitoring stack starts too late, visit Agntz for the free first scan and use it to get the inventory, provenance, and reach scoring that make runtime telemetry far more actionable.

#ai agent monitoring#agent observability#agent security#incident response#AIBOM

Own your AI risk

Estimate your exposure in under 3 minutes. No install, no access to grant.

Estimate Your Exposure