AI observability vs LLM observability
Most tools sold as observability for AI are really LLM observability: they capture the model call, the prompt, the completion, the token count, and the latency. That is useful, and it is where the category started, with products built around prompt logging and cost dashboards.
But an agent does far more than call a model. It runs shell commands, reads and writes files, calls tools over MCP, hits the network, and spawns sub-agents. AI observability is the broader practice of seeing all of it, not just the model turn in the middle. If your observability stops at the LLM call, you can see what the agent thought and none of what it actually did.
Logging vs monitoring vs observability
The three words answer different questions, and the distinction decides whether your setup can actually explain an incident.
| Logging | Monitoring | Observability | |
|---|---|---|---|
| Question it answers | What happened? | Is it healthy right now? | Why did this happen, and who did it? |
| The data | Discrete events, line by line | Metrics and alerts on known signals | Correlated traces across a whole session |
| For AI agents | A record of each call | Dashboards of latency, error rate, and cost | Query any agent’s full session, its tools, files, and delegation |
The full picture: every action an agent takes
AI observability means capturing the whole surface of what an agent does:
- LLM calls. Prompts, completions, tokens, and latency (time to first token and total).
- Tool and MCP calls. Which tool, with which arguments, returning what.
- File activity. Reads and writes, which matter enormously for a coding agent with disk access.
- Network and web requests. Where the agent reached, and what it sent or pulled back.
- Delegation. Which sub-agents it spawned, and how deep the chain went.
And the signal that ties it together, the one plain LLM logging never captures, is identity: which agent took each action, who owns it, and whether policy allowed it. Without that, you have a pile of activity and no way to say who is behind it.
Observability across every kind of agent
AI observability is not one integration, because agents run in very different places. It has to reach all of them.
Code agent observability
Coding agents like Cursor, Claude Code, and GitHub Copilot read repositories, run shell commands, edit files, and call MCP servers from inside the IDE and the CLI. Observing them means capturing those commands, file changes, and tool calls in the execution path, not just the model prompt, so you can see what a coding agent actually did to a machine.
Web AI observability
Agents that browse and act on the web navigate pages, fill forms, and pull data. The principle is identical: capture the actions the agent took and the data that moved, and tie each to the agent’s identity, so a web-acting agent is as accountable as one calling an internal API.
Custom agent observability
Most enterprises also run bespoke agents built on their own stack. These instrument cleanly with OpenTelemetry, emitting the same kind of trace data as everything else, so a custom agent’s LLM calls, tool calls, and actions flow into the same surface rather than a separate silo.
AI observability vs APM
Application performance monitoring (Datadog, New Relic, and the like) was built for a service: a request comes in, spans fan out across microservices, a response goes back. An agent is not a request. It is a reasoning loop that calls tools, touches files, and spawns sub-agents over a long session.
APM will show you the HTTP spans underneath an agent. What it cannot show you is “this agent, two hops into a delegation chain, acting for a deactivated user, wrote to this file and called the payments tool.” The gap is agent behavior and identity, and it is exactly where incidents become impossible to reconstruct.
OpenTelemetry for AI observability
OpenTelemetry is becoming the standard substrate for AI traces, with semantic conventions emerging for prompts, tokens, and tool calls. Building on it keeps your data portable: instrument once against the standard and you are not locked into one vendor’s format, and custom agents, code agents, and web agents can all emit into the same pipeline. It is the lesson distributed tracing taught the microservices era, arriving again for agents.
How Highflame approaches AI observability
Highflame Observatory is the evidence layer over the whole agent fleet: posture score, blast-radius graphs, and compliance evidence in one queryable surface, built on OpenTelemetry and ClickHouse so you query correlated data rather than reassemble logs after an incident. Because every action, an LLM call, an MCP tool call, a file write, already carries the agent’s identity and the authorization decision behind it, Observatory answers the question generic LLM observability cannot: not just what happened, but which agent did it, on whose authority, and whether policy allowed it. See it on the platform page.