Tracing connectseach operation inone request.
Tracing:one connected recordfor each AI answer.
One answer can cross models, retrieval, tools, queues, and retries. A trace preserves the causal path, timing, and attributes needed to locate delays and failures.One answer may involve several AI calls, searches, tools, and retries. A trace records that connected journey so people can find where a delay, error, or surprising decision began.
- TRACE
- one request's journeythe whole recorded journey
- SPAN
- one operation inside itone recorded step inside it
- PROOF?
- evidence, not a verdict
4bf92f35…473623:14:07.081- 2.84 s
a91fanswer.requestparent: root - 1.92 s
b772retrieve.contextparent: a91f - 706 ms
c044model.generateparent: a91f
The request was slow. The trace says where: a remote retrieval span owned 68% of wall time.
A trace groups the timed operations caused by one request.A trace connects the recorded steps for one request.
A flat log tells you an event occurred. A trace preserves which operation caused the next one, how long each took, and what evidence each operation carried.A normal record says something happened. A trace keeps the steps connected, so you can see what led to what, how long it took, and what information each step used.
Follow one request from entrance to exit. Every meaningful unit of work gets a timed span. Parent-child links reconstruct the causal route—even across process and network boundaries.Follow one request from start to finish. Record each meaningful step, how long it took, and which earlier step led to it.
Sampled or redacted traces may omit payloads, and external state may have changed.Private or bulky details may be left out, and the outside world may have changed since then.
A clean trace can still produce a confidently wrong answer. It records behavior; it does not define quality.A tidy sequence can still end in a wrong answer. A trace records what happened; it does not decide whether the result was good.
Metrics show how widespread a problem is. Logs record individual events. A trace connects those events into a causal path.A dashboard tells you something is wrong. Individual records show clues. A trace connects those clues into one path.
A trace connects a symptom to the request path that produced it.Start with the complaint. Follow one answer to the cause.
Same product, three complaints. Switch the observed symptom and inspect one request’s causal record—not an aggregate dashboard.The same product has three complaints. Pick one and follow the connected record for a single answer, step by step.
The answer took 2.84 seconds—long enough to feel broken.
- tracejourney
- 4bf92f35…4736
- end to endtotal time
- 2.84 s
- 2.84 s
a91fSTEP 1answer.requestWhole answer requestgateway · parent rootanswer service · starts the journey
route=/answerPart of the main path - 1.92 s
b772STEP 2retrieve.contextFind useful notesCLUEretrieval · parent a91fsearch service · follows an earlier step
region=eu-west · hits=6Strongest clue in this journey - 128 ms
c044STEP 3embed.queryPrepare the question for searchembeddings · parent b772search preparation · follows an earlier step
dimensions=1536Part of the main path - 706 ms
d883STEP 4model.generateAI writes the answermodel-client · parent a91fAI service · follows an earlier step
input=2384 · output=186 tokensPart of the main path - 18 ms
e110STEP 5audit.writeSave an activity recordevents · parent a91factivity recorder · follows an earlier step
async=trueRecorded in the background
The search ran in another region and caused most of the wait.
The app ran in the eastern United States, but its search ran in Europe. Nothing broke; the long trip quietly used most of the waiting time.
Search: Europe · app: US East · 68% of the total waitCritical path: the chain of dependent operations that determines end-to-end latency. Speeding up a span off that path may make a chart prettier while the user waits exactly as long.The path that matters: the connected steps that decide the total wait. Speeding up an unrelated background step can make a chart prettier while the person waits just as long.
Six parts of a useful trace.Six parts of a useful request record.
A useful trace is structured enough to query and small enough to read. These are the pieces that earn their keep.A useful trace is organized enough to search and small enough for a person to follow. These are the pieces worth keeping.
Trace
The full causal record for one logical operation, assembled from related spans.The complete connected record for one request, from beginning to end.
Span
One timed operation: a model call, retrieval, tool execution, queue handoff, or application step.One timed step, such as asking the AI, searching, using a tool, or saving a result.
Parent + child
The relationship that says this work happened because that work invoked it. The trace becomes a tree—or, more generally, a causal graph.The link that says one step happened because an earlier step started it. These links rebuild the path.
Attributes
Queryable key-value evidence such as model name, route, token counts, retry number, document version, or error type.Useful facts attached to a step, such as the AI version, document date, retry count, or error type.
Events
Timestamped moments inside a span: a timeout, cache miss, exception, or checkpoint that does not deserve a separate operation.Small moments inside a step, such as a timeout, failed lookup, warning, or checkpoint.
Context
Trace and span identifiers carried across calls so downstream work can rejoin the same causal story.A shared journey label carried between services so every later step can rejoin the same story.
One trace ID groups the journey. Each span ID names a particular operation; its parent ID reconnects the branch.
trace 4bf92f…
└─ answer.request span a91f parent —
├─ retrieve.context span b772 parent a91f
│ └─ vector.query span c044 parent b772
├─ model.generate span d883 parent a91f
└─ audit.write span e110 parent a91f [async]Give the whole request one journey label. Then record which step started each later step.
- 01Question arrives
- 02Useful notes are found
- 03The AI writes an answer
- 04The result is recorded
Observability signals answer different questions.Four kinds of evidence answer four different questions.
Correlate them. Do not promote one signal to a job it cannot do. Tracing is excellent at explaining a known failure path; evals still define whether the output was acceptable.Connect these records, but do not ask one kind to do every job. A trace explains how a known failure happened; a repeatable test still decides whether the result was good.
Metrics
“Error rate doubled.”“Errors doubled.”
- BEST AT
- Is this widespread?Is this happening a lot?
- BLIND SPOT
- Which individual path produced it.Which individual request caused it.
Logs
“Tool call timed out at 23:14.”“A tool timed out at 23:14.”
- BEST AT
- What event was recorded?What event was written down?
- BLIND SPOT
- The whole cross-service causal chain unless correlated.The full path unless records were linked.
Traces
“This timeout retried the whole agent loop.”“This timeout restarted the whole job.”
- BEST AT
- Where did this request go, and what caused what?Where did this request go, and what caused what?
- BLIND SPOT
- Whether the final answer met your quality bar.Whether the final answer was good.
Evals
“Policy accuracy fell six points.”“Policy answers got six points worse.”
- BEST AT
- Was the behavior good enough?Was the result good enough?
- BLIND SPOT
- The production execution path behind a particular miss.What happened inside one real failed request.
Record causal structure while minimizing sensitive data.Record enough to explain a failure while protecting people’s data.
Good instrumentation preserves causal boundaries and decisive evidence. Bad instrumentation is either a blank map or a surveillance archive.A good trace keeps the important steps and evidence. Too little gives you a blank map; too much turns useful records into a privacy problem.
- 01
Span the decisions, not every function.Record important steps, not every tiny function.
Create spans for operations whose timing, outcome, or boundary matters: model calls, retrieval, tools, retries, handoffs, and meaningful workflow stages.Keep the steps whose timing, outcome, or boundary matters: asking the AI, searching, using tools, retrying, and handing work to another service.
- 02
Propagate context across every boundary.Carry the journey label across every handoff.
HTTP, queues, workers, and tool services need the trace context. Lose it and one journey becomes a pile of unrelated fragments.When work moves to another service or waiting line, carry its shared label too. Without it, one journey becomes a pile of unrelated scraps.
- 03
Name the operation; attach the instance.Keep step names stable; attach the changing facts.
Use stable, low-cardinality names such as tool.execute. Put tool name, model, route, attempt, and document version in attributes.Call the same kind of work by the same name. Attach details such as which tool, AI version, attempt, or document was involved.
- 04
Make retries visible twice.Show every retry.
Record the logical operation and each physical attempt. Otherwise a successful retry hides the latency, cost, and original failure that caused it.Record the overall job and each attempt. Otherwise a successful second try can hide the time, cost, and first failure.
- 05
Record outcomes, tokens, and cost honestly.Record the outcome, usage, and cost honestly.
Capture error status and exception events. Prefer provider-reported token usage; label estimates and custom cost calculations as estimates.Save failures as well as successes. Prefer usage reported by the AI service, and clearly label any number you estimated yourself.
- 06
Correlate verbose logs instead of stuffing spans.Link the long notes instead of stuffing the map.
Put trace and span IDs on logs. Keep spans readable; use events or linked logs for detail that would turn the causal map into a junk drawer.Keep the main journey readable. Put bulky detail in linked records that carry the same journey and step labels.
trace 4bf9 · span a91fjourney 4bf9 · step a91ftraceparent: 00-4bf9…-a91f…-01carry journey 4bf9 and parent step a91f→trace 4bf9 · parent a91fsame journey · started by step a91fA correlation ID can help you search related records. Trace context goes further: it carries the identity needed to create the downstream child relationship.A shared label helps you find related records. Carrying the earlier step too shows which piece of work actually started the next one.
Privacy and sampling are part of the trace design.Choose what to record—and what to leave out—on purpose.
Telemetry leaves the hot path, travels through collectors, and often lives longer than the request. Design its data exposure and volume before production does it for you.These records often travel to other systems and live longer than the original request. Decide their privacy and volume before real use makes the decision for you.
- Do not record raw prompts, tool arguments, retrieved documents, or model outputs by default.Do not save full questions, tool details, source documents, or AI answers by default.
- Prefer template IDs, hashes, counts, classifications, and allow-listed metadata.Prefer safe labels, counts, categories, and a short approved list of details.
- Redact before export and review what automatic instrumentation collects.Remove private data before records leave the app, and inspect what automatic tracking collects.
- Treat baggage and propagated headers as data crossing trust boundaries.Treat any labels carried between services as information crossing a trust boundary.
Hashing a predictable identifier may still be reversible in practice. Data minimization beats clever cleanup.
Decide early
Efficient and simple, often based on the trace ID and a probability. It cannot know that a later child will error.Choose which journeys to save before they finish. It is simple, but it cannot know that a later step will fail.
Decide after
Can retain slow or erroneous completed traces, but needs stateful infrastructure and a thoughtful policy.Wait until a journey finishes, then keep slow or failed ones. This catches useful oddities but needs more storage and planning.
Sample coherent traces, not random orphan spans. Monitor the sampler too; a telemetry pipeline can fail like any other system.
Wrap the operation. Preserve the parent. End the span.
This OpenTelemetry-style TypeScript is intentionally small. In production, use automatic instrumentation for supported network libraries and add application spans around your AI-specific decisions.
import {SpanStatusCode, trace} from "@opentelemetry/api";
const tracer = trace.getTracer("answer-service");
return tracer.startActiveSpan("answer.request", {
attributes: {
"app.route": "/answer",
"app.prompt.template_id": "support-v7",
},
}, async (span) => {
try {
const result = await tracer.startActiveSpan(
"retrieve.context",
{attributes: {"app.retrieval.index": "policies"}},
async (child) => {
try {
return await retrieve(question);
} finally {
child.end();
}
},
);
span.setAttribute("app.retrieval.hit_count", result.length);
return await answer(question, result);
} catch (error) {
span.recordException(error as Error);
span.setStatus({code: SpanStatusCode.ERROR});
throw error;
} finally {
span.end();
}
});Six ways tracing data can mislead.
The data can be technically valid and still lead the investigation astray. These are the usual traps.
“Every function needs a span.”
No. Excess granularity adds overhead and hides the operations people actually need to reason about.
“All green means correct.”
Green usually means operations completed. Output quality belongs to product checks and evals.
“A trace ID crosses services automatically.”
Instrumentation must inject and extract context across each transport boundary.
“Store the raw prompt. Future us will need it.”
Future you may instead need to explain why sensitive customer data entered a telemetry vendor.
“Sampling only loses boring traffic.”
A weak policy can drop rare failures or bias analysis. Sampling strategy is part of the observability system.
“The longest span is always the fix.”
Only work on the critical path determines the wait. Parallel or asynchronous spans may be long without delaying the response.
Instrument the boundary where a decision becomes work. Carry the context. Keep the evidence you can defend. Then use the trace to locate failure and decide what to inspect next.