Aurora Libros is built, optimized and hardened. What is missing is the thing that separates a platform that works from a platform you can operate: knowing what is happening right now, what happened last night at 3:14, and why the API container restarted four times on Tuesday.

1. The stdout/stderr rule, revisited
2. The logging drivers
3. Global and per-service configuration
4. Mandatory rotation: the full-disk arithmetic
5. Structured JSON logs
6. Centralized aggregation: the pattern
7. Loki, Promtail and Grafana in the observability profile
8. Useful LogQL queries
9. Metrics: docker stats and its limits
10. The daemon's metrics API
11. cAdvisor, node-exporter and Prometheus
12. The metrics that really get watched
13. Alerts
14. The rich health endpoint
15. Distributed tracing: the next step

A containerized application does not write log files: it writes to standard output and standard error, and lets the platform do the rest.

The reason is concrete: the container is ephemeral. A file in /var/log/app.log disappears when the container is recreated, forces you to mount a volume just for that, demands its own rotation and cannot be read with docker logs. By writing to stdout, the process only produces the events; collecting them, rotating them and shipping them where they belong is the logging driver's responsibility.

The process's outputs are pipes to the daemon, not files. At the other end, the driver decides their destination.

The limitation marked "No" is important, and it takes people by surprise at the worst possible moment: with syslog, fluentd, gelf or awslogs, docker logs stops working. If the central system is down, you have no way left to see what is going on.

The usual strategy avoids that dead end: keep json-file or local with rotation and collect the files with an external agent (Promtail, Fluent Bit, Vector). That way you keep docker logs for immediate diagnosis and get central aggregation for history.

The default for the whole host goes in /etc/docker/daemon.json, and anything specific goes in Compose:

After sudo systemctl restart docker, verify it with docker info --format '{{.LoggingDriver}}'. Careful: changing the driver only affects containers created afterwards; existing ones keep theirs until they are recreated.

Without rotation, json-file grows without limit. Let's do the arithmetic with Aurora Libros: the API logs one line per request, about 200 bytes of JSON, with 50 requests per second at peak and an average of 15/s across the day.

Almost 8 GB a month from a single service, in a file that is never truncated. With four services and a 40 GB disk, the platform goes down from a full disk in under two months. And a full-disk failure is particularly nasty: PostgreSQL stops accepting writes, Docker cannot create containers, and the logging system itself cannot record the problem.

With max-size: 50m and max-file: 5, the ceiling per container is 250 MB, and you always have the last few days. Two warnings: max-file without max-size does nothing, and the local driver ships with rotation enabled by default (100 MB across 5 files), which makes it the safest option if you forget to configure it.

A free-text log forces you to write regular expressions for everything. A JSON log is queried by field.

The req_id is the key piece: it travels through the x-request-id header, the response hands it back, and it appears on every line belonging to that request. When a user reports an error at 10:14 with that identifier, you recover the full trace of their request with a single query instead of reading ten thousand lines.

docker logs is fine for one container on one machine. With several services, several nodes and containers that get recreated, the events need to leave the host and outlive the container that produced them.

Loki indexes only the labels (service, level, host) and compresses the rest, which makes it far cheaper than a stack that indexes every word. The alternatives: ELK/OpenSearch (Elasticsearch + Logstash + Kibana), more powerful for full-text search and considerably hungrier for resources; and the clouds' managed services.

Having all of this live in a profile (lesson 04-06) is deliberate: docker compose up -d still brings up nothing but the platform, and observability is added when it is needed, without eating resources on anybody's laptop.

Security warning. Promtail mounts docker.sock read-only in order to read labels, and cAdvisor has broad access to the host. These are real concessions (lesson 05-03): in production, use a filtered socket proxy and validate this configuration with your security officer.

The second query is the one that justifies all the work in section 5: with an identifier handed to you by the user, you retrieve their exact request from among millions of lines in under a second. The last one turns an architectural decision —the cache-aside pattern— into an observable metric: if the hit ratio drops, something is wrong with Redis or with invalidation.

Its limits, and why it is not enough: it is a snapshot with no history, it has no alerts, it does not aggregate several hosts, and somebody has to be watching. It answers "what is happening now"; not "what happened last night".

The daemon itself exposes metrics in Prometheus format if you enable it in daemon.json:

These are the daemon's metrics, not each container's: how many containers are in each state, operation timings, engine version. Useful for watching Docker's own health. And expose that port only on your management network: it carries no authentication.

cAdvisor reads the cgroups (lesson 05-07) and publishes per-container metrics; node-exporter does the same for the host.

Grafana queries both sources, and you do not have to invent the panels: dashboards 193 (Docker/cAdvisor) and 1860 (Node Exporter Full) from the public catalog cover 90% of what you need.

Notice the pattern: what matters is almost never the absolute value but the relationship to the limit you set back in lesson 03-07. 400 MB of memory says nothing; 400 MB against a 512 MB limit says an OOM is close.

Two things make the difference between an alert that gets acted on and one that gets ignored. The first is for: without it, a two-second spike wakes somebody up in the middle of the night. The second is who it notifies: oncall for whatever demands immediate action (a service down, massive errors) and team for what can wait until tomorrow (high memory, restarts). An alert that leads to no concrete action should be deleted; the worst possible state for an alerting system is a team that ignores it out of habit.

/health started out returning {"status":"ok"}. That only tells you the process starts. A useful endpoint checks its dependencies:

The 503 is the important part: the Dockerfile's HEALTHCHECK detects it, the container goes unhealthy, depends_on: service_healthy will not start whatever depends on it, and the load balancer stops sending it traffic. With a /health that only ever returned ok, none of that would happen.

A design nuance: distinguish liveness (should this process be restarted?) from readiness (can it receive traffic?). If Redis goes down, the API should not restart in a loop, only stop advertising itself as ready; the explicit distinction arrives with Kubernetes (lesson 06-05).

Logs and metrics answer "what happened" and "how much"; traces answer "where did the time go". A request to /books crosses nginx, Express, Redis and PostgreSQL, and a trace measures each leg separately. OpenTelemetry is the standard: with Node's automatic instrumentation all you need is one package and a few variables (OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4318, OTEL_TRACES_SAMPLER_ARG: "0.1" to sample 10%).

The req_id from section 5 is, in fact, a hand-rolled precursor of the trace_id. If you adopt OpenTelemetry, replace it with the trace identifier and you will have logs, metrics and traces correlated by the same field.

Writing logs to files inside the container. They are lost when it is recreated and docker logs never sees them. Always to stdout/stderr.

Not configuring rotation. 8 GB a month per service and an outage from a full disk that also prevents the cause from being logged.

Switching the driver to fluentd or syslog and losing docker logs. If the central system fails, you go blind. Keep json-file/local and collect with an agent.

Logging personal data or secrets. A log containing passwords or customer data is a data-protection incident. Filter at the source and check it with your compliance officer.

Alerting on absolute values, or alerting without for. 400 MB means nothing without the limit next to it, and a two-second spike that wakes somebody at night teaches the team to ignore alerts.

A /health that only says ok. It does not distinguish "the process is alive" from "the service works". Check the dependencies.

Tip: set up observability before you need it. On the day of the incident there is no time to install Prometheus, and the data you need most is from the preceding hours, which nobody can recover anymore.

Exercise 1. Work out aurora-api's log growth with your own traffic: measure the log file's size, generate 500 requests, measure again and extrapolate to a month. Then configure rotation and prove that the ceiling is respected.

Exercise 2. Add the req_id and structured logging to aurora-api, generate traffic, and prove that you can reconstruct the full trace of one specific request from the identifier returned in the response header.

Exercise 3. Bring up the observability profile, drive aurora-api's memory usage close to its limit, and check in Prometheus that the alert's expression fires. Explain why the alert uses a ratio and not an absolute value.

Solution 1.

Seven gigabytes a month from a single service, measured rather than estimated. Now the limit:

The ceiling holds: three files, none above 10 MB, 25 MB in total no matter what. The oldest lines have been lost, and that is exactly the point: recent logs are the ones that help you diagnose, and history is Loki's responsibility, not the host disk's. The detail to remember is that max-size without max-file limits a single file and max-file without max-size limits nothing at all: you need both.

Solution 2.

Three lines that tell the whole story of that request: the cache did not have the key, PostgreSQL was queried and returned 9 rows in 24.8 ms, and the response went out with a 200 in 33.2 ms overall. With those numbers you can state that 75% of the time went into the database, without guessing anything.

What makes this genuinely operable is that the identifier goes out to the world in the x-request-id header. The user reporting a problem can give you that code —or the front end can include it in its error message— and you retrieve their exact request from among millions. And because the header is also accepted on the way in, if nginx or the front end already generates one, the API reuses it and the trace is chained end to end.

Solution 3.

The expression returns a result (94% of the limit) and the alert goes to pending. After five minutes of the condition holding —the rule's for: 5m— it would move to firing and reach Alertmanager. That delay is deliberate: a memory spike during a one-off import should not page anybody; a sustained trend at 94% should.

And the reason for using a ratio rather than an absolute value is twofold. The first is about meaning: 480 MB is catastrophic for aurora-api (a 512 MB limit, one step away from the OOM killer and exit code 137) and completely normal for aurora-db (a 2 GB limit). An alert with a fixed threshold either floods you with false positives or detects nothing. The second is about maintenance: the day you raise the API's limit to 1 GB, the ratio-based rule is still correct without being touched, whereas an absolute threshold would have to be remembered and changed. And that "remembered" is precisely what never happens.

Aurora Libros is no longer an opaque box in motion. You know why a containerized application writes to stdout/stderr and not to files, you know the eight logging drivers and the trap half of them hide: with syslog, fluentd or gelf, docker logs stops working, so the solid strategy is json-file or local with rotation plus an agent that collects. And you have done the arithmetic that convinces anybody: 214 bytes per request is 7 GB a month from a single service, and without max-size and max-file that ends in a full-disk outage which also prevents its own cause from being logged.

Your API now emits structured JSON logs with level, service, version and a req_id that travels in the x-request-id header and lets you reconstruct the full trace of one specific request from among millions of lines. On top of that you build aggregation with Loki, Promtail and Grafana in an observability profile that stays out of the way in development, with LogQL queries ranging from "errors in the last hour" to the cache hit ratio, which turns an architectural decision into an observable metric.

On the metrics side you know docker stats' limits, the daemon's Prometheus endpoint, and cAdvisor and node-exporter feeding Prometheus and Grafana. And you know what to watch: memory and CPU against the limit —never in absolute terms—, restarts, health, p95 latency, error rate and disk; with alerts that carry a for and that distinguish who they wake up. You close with a /health that checks PostgreSQL and Redis and returns 503 when something fails, making the HEALTHCHECK, depends_on and the load balancer react on their own, and with OpenTelemetry noted down as the next step.

In lesson 05-07, the last of the module, the circle closes: you will go down to the Linux kernel to see that a container does not exist. It is an ordinary process with three mechanisms on top —namespaces, cgroups and a layered filesystem— and you are going to touch each of them: the seven namespaces with lsns and nsenter, aurora-db's cgroups v2 files verifying that they match exactly the limits you set in 03-07, the real OverlayFS mount with its copy-on-write, capabilities decoded with capsh, and exactly what runc does when it starts a container.