In 06-01 we left TechCorp with logs queryable by requestId and a POST /v1/orders p95 of 250 ms. What those signals do not tell us is where those 250 ms go: in the call to Catalog, in the call to Customers, in PostgreSQL, in Express itself? And when the request finishes with a 202, the saga carries on through RabbitMQ in three other services: each one's logs exist, but nobody joins them into a single journey. Distributed traces are the signal that draws that path. This lesson instruments TechCorp's services with OpenTelemetry, propagates context over HTTP and over RabbitMQ, sets up the Collector and Jaeger on Kubernetes, and teaches you to read a real trace of ord-88213.

1. The problem: one request, four services and a saga
2. Traces, spans and context: traceparent and its relationship with X-Request-Id
3. OpenTelemetry: API, SDK, instrumentations, Collector and exporters
4. Instrumenting Node.js: src/telemetry.js in the template and in the Dockerfile
5. Manual spans in the createOrder use case
6. Propagating context through RabbitMQ: outbox, consumers and links
7. The Collector and Jaeger on Kubernetes
8. Reading a trace: ord-88213 and the bottleneck
9. Traces ↔ logs and traces ↔ metrics correlation
10. Sampling and cost

Let's follow Ana Ruiz's order from the browser:

With what we have, each hop leaves an access log with its responseTime and an observation in the 06-01 histogram. But to answer "why did this request take 1.8 s?" you would have to open Loki, search for the requestId, write down the timings of four services by hand and subtract them. And the asynchronous part does not even share the requestId naturally: the Inventory consumer receives a message, not a request.

A trace does that work automatically: it records every operation as a span with start, duration, attributes and parent, and ties them together with a common identifier. Viewing them in Jaeger is seeing the sequenceDiagram above with real timings.

The standard format is W3C Trace Context: a traceparent header with four dash-separated fields:

And an optional tracestate for vendor data. Everything that speaks HTTP at TechCorp (gateway, services, fetch) must forward traceparent just as it forwarded X-Request-Id.

Does traceparent replace X-Request-Id? No: they coexist with different roles.

Decision: both are kept. The requestId is what a support operator asks the customer for and searches in Loki; the trace_id is what the developer opens in Jaeger. Both appear on every log line, so going from one to the other is one query.

OpenTelemetry (OTel) is the CNCF standard that unifies how traces, metrics and logs are generated and transported, independently of the vendor that stores them. Its pieces:

• API: what the application code uses (trace.getTracer, startActiveSpan, propagation.inject). It is stable and independent of the implementation: if no SDK is loaded, it does nothing.
• SDK: the implementation that actually creates the spans, applies sampling, groups them into batches and hands them to the exporter.
• Automatic instrumentations: monkey-patching of well-known libraries to create spans without touching our code: http (incoming and outgoing, including Node 20's global fetch via @opentelemetry/instrumentation-undici), express (one span per middleware/route), pg, mongodb, amqplib.
• Exporters: send over OTLP (OTel's native protocol, gRPC on 4317 or HTTP on 4318) to the Collector, or directly to a backend.
• Collector: an intermediate process that receives, processes (batching, tail sampling, enrichment with Kubernetes metadata) and re-exports to one or more destinations. It decouples the application from the backend: swapping Jaeger for Tempo means changing the Collector configuration, not redeploying seven services.

Automatic instrumentations patch modules when they are loaded, so the SDK must be initialized before Express, pg or amqplib. The clean way is a separate file loaded with --require, without touching server.js:

Explanation:

• resource describes who emits: service.name is what Jaeger shows as the service; service.version lets us compare the canary (05-04) with the stable version. OTLPTraceExporter() with no arguments takes its destination from OTEL_EXPORTER_OTLP_ENDPOINT, which goes into the service's ConfigMap (05-02).
• getNodeAutoInstrumentations enables the whole catalog; we disable fs and exclude the probes and /metrics from incoming spans for the same reason as in the 06-01 logs. enhancedDatabaseReporting: false: the pg span includes the SQL statement but not the parameters: Ana Ruiz's data must not travel to Jaeger (same rule as log redaction).
• The SIGTERM handler matters: spans are exported in batches every few seconds; without shutdown() the last batch of a pod that dies during a rolling update is lost. It coexists with the server's graceful shutdown from 04-02.

Standard OTel environment variables that we add to the configuration template (04-03) and to the ConfigMap:

Startup changes in package.json and in the Dockerfile from 05-01:

--require loads the module before the first line of server.js; from then on, every Express request, every outgoing fetch and every pg query produces spans with no extra code. With this, a POST /v1/orders already generates the gateway → orders → catalog/customers → pg trace in Jaeger. What is missing is what automatic instrumentation does not know: where our business logic starts and ends.

A span of our own marks the business operation and adds attributes that can later be searched in Jaeger. We modify the use case from 04-04:

• trace.getTracer('orders-service'): obtains a tracer from the API. We only import @opentelemetry/api: the use case knows nothing about the SDK, and in the 04-05 tests, with no SDK loaded, the spans are "no-op".
• startActiveSpan('createOrder', fn): creates the span as a child of the active one (Express's POST /v1/orders span) and keeps it active during fn, so that the fetch calls to Catalog and Customers and the pg INSERT hang off it automatically. That is the "active context" trick: automatic instrumentation looks for the active span in Node's AsyncLocalStorage and attaches to it.
• Attributes follow the domain.field convention (order.id, customer.id); they are high-cardinality values, which in traces are welcome (unlike the metric labels of 06-01): Jaeger lets you search for order.id=ord-88213.
• addEvent adds a timestamped marker inside the span; recordException + setStatus(ERROR) make the trace show up in red and with the stack (without this, a BusinessError that ends in a 503 would look like a green span that finishes early); and span.end() goes in finally always: an unclosed span is never exported.

The same three gestures (startActiveSpan, attributes, recordException) are applied in Inventory's createReserveStockUseCase and in Payments' charge.

The amqplib instrumentation injects traceparent into the headers when publishing inside an active span and extracts it when consuming. At TechCorp there is a trap: the order.created event is not published by the use case but by the outbox relay (04-04), at another moment and with no active span related to the request. If we do nothing, the saga starts a brand-new trace unrelated to the POST.

Solution: store the context in the outbox row when writing it and restore it when publishing.

propagation.inject takes the active context (we are inside createOrder, section 5) and writes the W3C headers into any object; that object is stored in the headers column (JSONB) next to the requestId we were already storing in 03-02.

In the relay (messaging/outboxRelay.js), when publishing each row:

There is a design decision here. We could make the publish span a child of the original POST /v1/orders, but that span ended seconds ago: the trace would show a child starting after its parent finished, and the traces of 15-minute sagas would be unmanageable. OTel has the concept of a link for this: the publish span belongs to its own trace (the relay's/the saga's) but links to the original span. Jaeger shows links and lets you jump from one trace to the other. The rule at TechCorp:

• Inside a synchronous (HTTP) request: parent-child relationship.
• Between the request and asynchronous processing (outbox, consumers): a link to the originating span and one trace per event.
• Inside a consumer, everything it triggers (queries, publishing the next event) is a child of the consume span, so each step of the saga is a small trace linked to the previous one: POST ⇢ publish order.created ⇢ consume order.created (Inventory) ⇢ publish stock.reserved ⇢ …

In the saga consumer (messaging/sagaConsumer.js) the amqplib instrumentation already creates an orders.saga process span with SpanKind.CONSUMER from the traceparent in the headers. We only add attributes and, if the message comes from a retry or from the DLQ (06-03), we keep the headers so as not to break the chain. When a service publishes without going through the outbox (Inventory publishes stock.reserved inside the consume span), automatic injection is enough.

The Platform team deploys the Collector in the observability namespace as a Deployment (two replicas behind an otel-collector Service with ports 4317/4318) and its configuration in a ConfigMap:

• receivers.otlp: accepts what the services send (gRPC 4317, HTTP 4318).
• processors: memory_limiter protects the Collector; k8sattributes tags every span with the originating pod and Deployment (same names as in Loki: they cross-reference well); tail_sampling decides which traces to keep after seeing them complete; batch groups before exporting.
• exporters.otlp/jaeger: Jaeger v2 accepts OTLP directly. Swapping it for Grafana Tempo means changing the endpoint; Tempo integrates better with Grafana/Loki, Jaeger has the better-known UI. TechCorp starts with Jaeger (jaeger-query published internally at jaeger.techcorp.internal).
• The spanmetrics connector generates RED metrics (traces_span_metrics_calls_total, traces_span_metrics_duration_milliseconds) from the spans; it exposes them on 8889 so that Prometheus (06-01) scrapes them. It is a second source of RED per service and operation, useful above all for outgoing calls.

In each service's ConfigMap (05-02) we add OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME; in the local compose.yaml (05-01) a jaegertracing/all-in-one container with OTLP enabled and the UI at http://localhost:16686 is enough for developing without a Collector.

Marta searches Jaeger for service=orders-service, order.id=ord-88213 and opens the trace. Jaeger shows it as a Gantt chart; in table form (fictional but realistic timings):

How to read it:

1. The gateway adds 3 ms of overhead: negligible.
2. Catalog (40 ms) and Customers (25 ms) run in parallel (Promise.all from section 5): both start at ms 9. If they were sequential, the trace would show the second one starting when the first finishes: the first optimization a trace usually reveals.
3. Database queries are fast: MongoDB 9 ms, PostgreSQL 6 and 12 ms.
4. Adding up: 40 (the slowest of the parallel pair) + 12 for pg = 52 ms of real work, but createOrder lasts 176 ms. There are 120 ms without a span between the COMMIT (ms 64) and the end of the span (ms 184). It is a gap: time spent inside our code without any instrumented operation. Looking at the use case, Luis finds that after persisting, a computeRecommendations function inherited from the monolith is called, which does a heavy synchronous computation. It blocks the event loop (06-04) and contributes nothing to the order. It is removed and the p95 of POST /v1/orders drops from 250 to 90 ms.

Practical reading rules: look for the longest span among siblings; look for gaps with no children (uninstrumented own code or a blocked event loop); check whether children are sequential when they could be parallel; and, in the asynchronous part, follow the links from one trace to the next to measure the whole saga (publish order.created at 10:21:04.6, consume order.confirmed at 10:21:07.9: 3.3 s, consistent with the saga_duration_seconds histogram from 06-01).

Traces ↔ logs. The @opentelemetry/instrumentation-pino instrumentation (enabled in section 4) automatically adds trace_id, span_id and trace_flags to every log line emitted inside an active span. The line from 06-01 becomes:

If you would rather not depend on that instrumentation, a mixin in createLogger does the same by hand:

With that, in Grafana you configure a derived field on the Loki datasource: expression "trace_id":"(\w+)" → link to Jaeger/Tempo by trace_id. And in the opposite direction, from a span in Jaeger, a link to Loki with {namespace="techcorp"} | json | trace_id="…". The operator no longer chooses between logs and traces: they jump.

Traces ↔ metrics. Exemplars allow a bucket of the http_request_duration_seconds histogram to carry the trace_id of a representative request; Grafana draws them as dots over the p95 graph and one click opens the trace. It requires prom-client with exemplar support and --enable-feature=exemplar-storage in Prometheus. We leave it as a mention: TechCorp will enable it once the stack is stable; the metric → logs → trace jump from the previous point covers the same case with one extra step.

Each POST /v1/orders generates about 12 spans; with the saga, about 30. At 3,000 orders/day that is little, but the catalog serves hundreds of thousands of GET /v1/products and on Black Friday ×20. Keeping 100% is expensive (storage and CPU in the Collector) and almost always unnecessary: "normal" traces all look alike. That is why you sample:

TechCorp combines both:

• In the services, OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 in production (10% of root traces; parentbased means that if the parent was sampled, so is the child: a trace never comes out half-done) and 1.0 in dev/staging.
• In the Collector, the tail_sampling policy from section 7: all traces with an error or longer than 1 s, and 10% of the rest. Since the head already cut down to 10%, the tail acts on that subset; so that errors are not lost at the head, the ratio of the critical services can be raised (Orders, Payments: 0.5) while leaving 0.1 for Catalog.
• Retention in Jaeger: 7 days. The requestIds in the logs stay 30 days, so an old incident is investigated with logs even though its trace has expired.

One last note, to close the thread with 05-05: a service mesh like Istio generates spans at every sidecar (entry and exit of each pod) and sends them to the same Collector, but it cannot propagate context inside the application: between the incoming request and the outgoing one there is our code, and only our process knows that the second is a consequence of the first. If TechCorp ever adopts the mesh, the application will still need exactly this lesson: forwarding traceparent, and with RabbitMQ, injecting and extracting it by hand.

• Loading the SDK after Express or pg. Nothing gets instrumented and there is no error. Always --require ./src/telemetry.js (or NODE_OPTIONS=--require), never a require inside server.js after other imports.
• Forgetting span.end() on an error path. The span is not exported and the trace stays "open" in Jaeger. Always use try/finally or the callback variant that closes it by itself.
• Losing context in setTimeout, internal queues or EventEmitter. AsyncLocalStorage follows await and promises, but a setTimeout or an emitter.on may run outside the context. context.with(ctx, fn) restores it.
• Making parent-child what should be a link. 15-minute traces with thousands of spans and parents that finish before their children. Asynchronous = links.
• Putting personal data in attributes (customer.email, request bodies). The same rules as for logs apply (06-01, 07-03).
• 100% sampling in production "so nothing is lost". What is lost is the Collector. Head 10% + tail for errors.
• Tip: locally, jaegertracing/all-in-one in compose.yaml and OTEL_TRACES_SAMPLER_ARG=1.0; seeing the trace of a curl to POST /v1/orders is the best way to understand the structure of a request.

A message from orders.saga went to orders.saga.dlq and is reprocessed manually (06-03). What must be preserved so that the new processing shows up linked to the original trace? Should it be a child or a link? Write the snippet that creates the reprocess span.

In a POST /v1/orders trace you see: createOrder 410 ms; fetch catalog starts at ms 5 and lasts 45 ms; fetch customers starts at ms 52 and lasts 30 ms; pg 15 ms starting at ms 84; nothing else until ms 410. List the two problems and what you would do about each one.

Exercise 1

You must preserve the original message headers (traceparent, requestId, eventId) when the script moves the message from the DLQ to the main queue. The reprocessing happens much later and on human initiative: it must be a link, not a child. Snippet from the reprocessDlq.js script:

The consumer will receive the reprocess span's traceparent and, through the link, the trace of the original failure can be reached.

Exercise 2

1. Sequential calls that could be parallel. Customers starts at ms 52, right when Catalog finishes. With Promise.all the validation phase would take 45 ms instead of 75. Change in the use case.
2. A ~310 ms gap with no spans after the COMMIT (ms 99 → ms 410). It is uninstrumented own code or a blocked event loop. It is located by wrapping the suspicious functions of the use case in manual spans (or with nodejs_eventloop_lag_seconds from 06-01 on that pod) and then removed, made asynchronous, or moved out of the request (into an event). Expected result: createOrder around 60-70 ms.

With this lesson, TechCorp finally sees the path of every request. Each service starts with node --require ./src/telemetry.js, which loads the OpenTelemetry SDK with automatic instrumentations for http, express, pg, mongodb, amqplib and pino, identifies itself with OTEL_SERVICE_NAME and service.version, and exports over OTLP to the Collector; the use cases add manual spans (createOrder, reserveStock, charge) with order.id/customer.id attributes and recorded exceptions; the W3C traceparent context travels over HTTP alongside X-Request-Id, is stored in the outbox row and re-injected in the relay, and the asynchronous stages of the saga are joined with links; the Collector applies k8sattributes, tail_sampling and batch and re-exports to Jaeger and Prometheus; and trace_id on every log line closes the triangle with Loki. Reading the trace of ord-88213 we found a 120 ms gap that no metric gave away. We can now see failures and slowness; the next lesson is about surviving them: the error handling and recovery patterns (timeouts, retries, circuit breaker, bulkhead, retry queues and DLQ, the saga watchdog) that modules 2, 3 and 5 kept deferring to 06-03.