CONCEPT 012bring a flashlight

Tracing:the requestleft receipts.

One answer can cross models, retrieval, tools, queues, and retries. A trace preserves that causal journey so you can find where the weird actually entered the system.

Open the black box Learn the anatomy

TRACE: one request's journey
SPAN: one operation inside it
PROOF?: evidence, not a verdict

NIGHT LINE / FLIGHT RECORDERREC

TRACE ID4bf92f35…4736

START23:14:07.081

a91fanswer.requestparent: root
2.84 s
b772retrieve.contextparent: a91f
1.92 s
c044model.generateparent: a91f
706 ms

DISPATCH NOTE

The request was slow. The trace says where: a remote retrieval span owned 68% of wall time.

01 / THE SHORT VERSION

A trace is the case file. Spans are the witness statements.

A flat log tells you an event occurred. A trace preserves which operation caused the next one, how long each took, and what evidence each operation carried.

THE MENTAL MODEL

Follow one request from entrance to exit. Every meaningful unit of work gets a timed span. Parent-child links reconstruct the causal route—even across process and network boundaries.

Not a replay tape

Sampled or redacted traces may omit payloads, and external state may have changed.

Not an eval

A clean trace can still produce a confidently wrong answer. It records behavior; it does not define quality.

Metrics ring the fire alarm. Logs report the smoke. A trace draws the path from toaster to flame.

02 / NIGHT SIGNAL LAB

Start with the symptom. Follow the lit track to the cause.

Same product, three complaints. Switch the observed symptom and inspect one request's causal record—not an aggregate dashboard.

DISPATCH CONTROL · LINE 012Which complaint reached the incident desk?

OBSERVED

The answer took 2.84 seconds—long enough to feel broken.

trace: 4bf92f35…4736
end to end: 2.84 s

+0 ms
a91fanswer.request
gateway · parent root
route=/answer
2.84 s
+42 ms
b772retrieve.contextCLUE
retrieval · parent a91f
region=eu-west · hits=6
1.92 s
+49 ms
c044embed.query
embeddings · parent b772
dimensions=1536
128 ms
+1.98 s
d883model.generate
model-client · parent a91f
input=2384 · output=186 tokens
706 ms
+2.03 s
e110audit.write
events · parent a91f
async=true
18 ms

LIKELY CAUSAL BREAK

Retrieval crossed a region boundary and owned the critical path.

The vector query spent 1.92 seconds in eu-west while the application ran in us-east. Nothing crashed; geography quietly ate the latency budget.

retrieval.region=eu-west · app.region=us-east · 68% of wall time

Critical path: the chain of dependent operations that determines end-to-end latency. Speeding up a span off that path may make a chart prettier while the user waits exactly as long.

03 / FIELD MANUAL

The nouns inside the black box.

A useful trace is structured enough to query and small enough to read. These are the pieces that earn their keep.

Trace

The full causal record for one logical operation, assembled from related spans.

Span

One timed operation: a model call, retrieval, tool execution, queue handoff, or application step.

Parent + child

The relationship that says this work happened because that work invoked it. The trace becomes a tree—or, more generally, a causal graph.

Attributes

Queryable key-value evidence such as model name, route, token counts, retry number, document version, or error type.

Events

Timestamped moments inside a span: a timeout, cache miss, exception, or checkpoint that does not deserve a separate operation.

Context

Trace and span identifiers carried across calls so downstream work can rejoin the same causal story.

PARENT–CHILD RECEIPT

One trace ID groups the journey. Each span ID names a particular operation; its parent ID reconnects the branch.

trace 4bf92f…
└─ answer.request        span a91f  parent —
   ├─ retrieve.context   span b772  parent a91f
   │  └─ vector.query    span c044  parent b772
   ├─ model.generate     span d883  parent a91f
   └─ audit.write        span e110  parent a91f  [async]

04 / FOUR WITNESSES

Observability signals answer different questions.

Correlate them. Do not promote one signal to a job it cannot do. Tracing is excellent at explaining a known failure path; evals still define whether the output was acceptable.

Metrics

“Error rate doubled.”

BEST AT: Is this widespread?
BLIND SPOT: Which individual path produced it.

Logs

“Tool call timed out at 23:14.”

BEST AT: What event was recorded?
BLIND SPOT: The whole cross-service causal chain unless correlated.

Traces

“This timeout retried the whole agent loop.”

BEST AT: Where did this request go, and what caused what?
BLIND SPOT: Whether the final answer met your quality bar.

Evals

“Policy accuracy fell six points.”

BEST AT: Was the behavior good enough?
BLIND SPOT: The production execution path behind a particular miss.

05 / INSTRUMENTATION RULES

Record enough structure to debug. Not enough autobiography to regret.

Good instrumentation preserves causal boundaries and decisive evidence. Bad instrumentation is either a blank map or a surveillance archive.

01
Span the decisions, not every function.
Create spans for operations whose timing, outcome, or boundary matters: model calls, retrieval, tools, retries, handoffs, and meaningful workflow stages.
02
Propagate context across every boundary.
HTTP, queues, workers, and tool services need the trace context. Lose it and one journey becomes a pile of unrelated fragments.
03
Name the operation; attach the instance.
Use stable, low-cardinality names such as tool.execute. Put tool name, model, route, attempt, and document version in attributes.
04
Make retries visible twice.
Record the logical operation and each physical attempt. Otherwise a successful retry hides the latency, cost, and original failure that caused it.
05
Record outcomes, tokens, and cost honestly.
Capture error status and exception events. Prefer provider-reported token usage; label estimates and custom cost calculations as estimates.
06
Correlate verbose logs instead of stuffing spans.
Put trace and span IDs on logs. Keep spans readable; use events or linked logs for detail that would turn the causal map into a junk drawer.

PROCESS Aagent-apitrace 4bf9 · span a91f

W3C TRACE CONTEXTtraceparent: 00-4bf9…-a91f…-01

PROCESS Btool-workertrace 4bf9 · parent a91f

A correlation ID can help you search related records. Trace context goes further: it carries the identity needed to create the downstream child relationship.

06 / KEEP THE SIGNAL, LOSE THE SECRETS

Privacy and sampling are part of the trace design.

Telemetry leaves the hot path, travels through collectors, and often lives longer than the request. Design its data exposure and volume before production does it for you.

REDACTION DESKCollect less on purpose.

Do not record raw prompts, tool arguments, retrieved documents, or model outputs by default.
Prefer template IDs, hashes, counts, classifications, and allow-listed metadata.
Redact before export and review what automatic instrumentation collects.
Treat baggage and propagated headers as data crossing trust boundaries.

Hashing a predictable identifier may still be reversible in practice. Data minimization beats clever cleanup.

SAMPLING OFFICEKeep representative traffic—and important weirdness.

HEAD

Decide early

Efficient and simple, often based on the trace ID and a probability. It cannot know that a later child will error.

TAIL

Decide after

Can retain slow or erroneous completed traces, but needs stateful infrastructure and a thoughtful policy.

Sample coherent traces, not random orphan spans. Monitor the sampler too; a telemetry pipeline can fail like any other system.

07 / PRACTICAL RECEIPT

Wrap the operation. Preserve the parent. End the span.

This OpenTelemetry-style TypeScript is intentionally small. In production, use automatic instrumentation for supported network libraries and add application spans around your AI-specific decisions.

answer-service.tsILLUSTRATIVE

import {SpanStatusCode, trace} from "@opentelemetry/api";

const tracer = trace.getTracer("answer-service");

return tracer.startActiveSpan("answer.request", {
  attributes: {
    "app.route": "/answer",
    "app.prompt.template_id": "support-v7",
  },
}, async (span) => {
  try {
    const result = await tracer.startActiveSpan(
      "retrieve.context",
      {attributes: {"app.retrieval.index": "policies"}},
      async (child) => {
        try {
          return await retrieve(question);
        } finally {
          child.end();
        }
      },
    );

    span.setAttribute("app.retrieval.hit_count", result.length);
    return await answer(question, result);
  } catch (error) {
    span.recordException(error as Error);
    span.setStatus({code: SpanStatusCode.ERROR});
    throw error;
  } finally {
    span.end();
  }
});

08 / CASES CLOSED, MOSTLY

Six ways to make a trace lie by omission.

The data can be technically valid and still lead the investigation astray. These are the usual traps.

“Every function needs a span.”

No. Excess granularity adds overhead and hides the operations people actually need to reason about.

“All green means correct.”

Green usually means operations completed. Output quality belongs to product checks and evals.

“A trace ID crosses services by telepathy.”

Instrumentation must inject and extract context across each transport boundary.

“Store the raw prompt. Future us will need it.”

Future you may instead need to explain why sensitive customer data entered a telemetry vendor.

“Sampling only loses boring traffic.”

A weak policy can drop rare failures or bias analysis. Sampling strategy is an engineering system, not a coupon.

“The longest span is always the fix.”

Only work on the critical path determines the wait. Parallel or asynchronous spans may be long without delaying the response.

THE OPERATING PRINCIPLE

Instrument the boundary where a decision becomes work. Carry the context. Keep the evidence you can defend. Then use the trace to locate failure—not merely admire that failure occurred.