MODULE 09 OF 12 · RELEASE 3

Observability & Agent Operations

Being able to answer "what happened?" about something that happened three weeks ago.

So what is this really about?

When a customer says "your system gave me a wrong answer last month", you either can reconstruct it or you cannot. That is decided long before the complaint, by whether each request left a trail: what came in, what was looked up, what the AI said, what it did.

The words you will hear

Trace

The full story of one request, start to finish, in order.

Span

One step inside that story — a single lookup or AI call, with how long it took.

Correlation id

A reference number stamped on everything belonging to the same request.

Runbook

Written instructions for a bad day, so nobody has to think it through at 3am.

By the end you will be able to

• Follow one request across every step it touched.
• Know what each request cost and where the time went.
• Be alerted about things that matter, not things that are merely busy.

Going deeper

The question you cannot answer later

A customer emails. Some time last month, your system told them something wrong, and it caused a problem. They do not have a screenshot, they are not sure of the date, and they would like to know what happened.

Whether you can answer was decided weeks ago, by whether each request left a trail worth following. If it did not, your honest answer is that you cannot reconstruct it — which is a bad thing to say to a customer, and a worse thing to say to a regulator.

Observability is not a dashboard. It is the ability to answer questions about the past that you did not think to ask at the time.

One number ties everything together

The single highest-value change is also the simplest. When a request arrives, generate an identifier, and attach it to every log line, every span, every database row and every outbound call that request produces.

Without it you have thousands of unrelated lines and no way to tell which belong together — particularly under concurrency, where ten requests interleave and the ordering means nothing. With it, one filter reassembles the entire story.

Pass it onward too. If your request triggers a queued job, the job carries the same identifier, so the trail does not stop at the boundary between the two.

What each step should record

For every model call and every tool call, record the same handful of facts:

{
  correlation_id,
  step,             // which model or tool
  started_at, ms,   // when, and how long
  tokens_in, tokens_out,
  outcome,          // ok | failed | timed_out
  attempt,          // 1 for the first try
}

Record the inputs and outputs too, with judgement. Full prompts and completions are enormously useful for debugging and enormously sensitive — they contain whatever your users typed. Decide deliberately: redact known-sensitive fields, set a shorter retention for this data than for the rest, and write down what you chose and why.

attempt is small and easy to omit. Without it, a retried call looks like two unrelated calls, and your latency numbers quietly include work that was thrown away.

Cost is derived, not estimated

Since every call records its tokens, cost is arithmetic rather than guesswork. That lets you answer the question that actually gets asked: not "what did we spend?" but "what did this feature cost, and is it worth it?"

Attributing by feature matters because totals do not support decisions. A total tells you the bill went up. A breakdown tells you one summarisation feature accounts for sixty percent of spend and is used by four people — which is an actionable fact.

Alert on conditions, not volume

The classic mistake is alerting on request count. Traffic doubling is not a problem; it might be excellent news. The things worth waking someone for are:

• Error rate above normal — as a proportion, not a count, so it works at any traffic level.
• Latency at the 95th percentile — averages hide the tail, and the tail is where users suffer.
• Spend rate — burning the daily budget by 10am is worth knowing at 10am, not tomorrow.
• Work arriving but not completing — queue depth growing while throughput holds steady means something downstream has stopped.

Each of those describes a condition a person would act on. An alert nobody acts on trains everyone to ignore alerts, which is worse than having none.

Write the runbook before you need it

Take the three most likely failures and write down, for each: how you notice, how you confirm, what you do first, and who to tell. It is a page each.

The value is not the document. It is that writing it forces you to discover you do not have a way to check something, at a time when discovering that is cheap. Then rehearse one, because a runbook that has never been followed is a hypothesis.

What good looks like

You can take a customer’s vague description of something from last month and produce the exact trace. You know what each feature costs. Alerts fire rarely and mean something when they do.

How it is put together

One trace per request, with a span for every model and tool call carrying tokens, latency, and outcome. Traces carry a correlation id that appears in every log line. Cost is derived from recorded tokens rather than estimated. Alerts fire on error rate, budget burn and latency percentiles, never on raw request count.

Where you start

A system with unstructured logs and no correlation between them.

The lab

1. Add a correlation id at ingress and thread it through every log line and span.
2. Emit a span per model and tool call with tokens, latency and outcome.
3. Derive cost from recorded tokens and attribute it per feature.
4. Add alerts on error rate, latency percentile and budget burn.
5. Write a runbook for the three most likely failures.

What goes wrong, on purpose

The scenario. A customer reports a wrong answer from three weeks ago and gives only an approximate time.

What should happen. The trace is locatable from the available detail and shows the inputs, retrieved context, tool calls and model outputs that produced the answer.

How you prove it works

• A test asserting every log line emitted during a request carries the correlation id.
• A test asserting recorded token counts match the provider response.
• A drill: reconstruct a specific past request using only stored telemetry.

What you walk away with

A traced request path, per-feature cost attribution, alerts on meaningful conditions, and a runbook that has been rehearsed.