As the previous lesson closed we were left with a question metrics cannot answer. When BookingsApiHighErrorRate fires at 03:14, we know that 7 % of requests are failing, on which route and with what latency. What we do not know is why: the specific exception, the message PostgreSQL returned, the line of code that blew up. That does not live in a metric. It lives in the logs.

And today, at Rutas Norte, the logs are an organised disaster: they are scattered across the nodes, they are lost when a pod is recreated, there is no way to search across components and they disappear entirely if a node dies. In this lesson we build the complete stack we announced in 06-02 when we deployed the collector as a DaemonSet: Elasticsearch, Fluentd and Kibana. We will see how logging works under the hood in Kubernetes, how logs are collected and enriched, why structured JSON is the single decision that improves the whole system most, and — critically for a platform that stores its customers' ID numbers and phone numbers — what must never end up written in a log.

1. Why kubectl logs is not enough
2. How logging works under the hood in Kubernetes
3. kubectl logs in depth: the front-line tool and its limits
4. The architecture of the per-node collection pattern
5. Fluentd versus Fluent Bit
6. The real collector configuration
7. Elasticsearch: indices, templates and life cycle
8. Kibana: index pattern, KQL and an error dashboard
9. Structured JSON logs: the decision with the best return
10. Correlating logs and metrics
11. What must NEVER be logged
12. Lighter alternatives and the real cost of a logging stack
13. Common mistakes and tips
14. Exercises

kubectl logs is the first tool we learned in 01-05 and it remains extremely useful. But it has four structural limits that make it insufficient as soon as the platform grows.

It is lost when the pod is recreated. A container's logs live on the node's filesystem, tied to the pod. When the pod is deleted — a deployment, an eviction, a scale-down — the files are deleted with it. If bookings-api went into CrashLoopBackOff at 03:14 and by 08:00 somebody had already restarted it by hand, the evidence is gone.

It does not search across components. A customer reports that their purchase failed. The request went through web-store, then through bookings-api, which queried redis-cache and bookings-postgres, called the external gateway and queued a message for notifications-worker. That is six different kubectl logs, on different pods, each with its own format and with no way of knowing which line in one matches which line in another.

It does not correlate. Even if you open the six terminals, you will have to line up timestamps by eye. With six bookings-api replicas you do not even know which one the request landed on.

It disappears with the node. If rutas-norte-worker-2 dies, all its logs die with it. And a node dying is exactly when they are most needed.

To those four you can add one of governance: there is no defined retention and no granular access control. Anybody with the pods/log permission sees everything the applications write, including the data that should not be there. We will come back to this in section 11.

Before collecting anything you have to understand where the logs physically are. Without this, the collector's configuration is magic.

Kubernetes defines no logging API for applications. The contract is Docker's and the twelve-factor one:

The application writes its logs to standard output (stdout) and standard error (stderr). It manages no files, no rotation and no destinations.

An application that writes to /var/log/myapp.log inside the container is doing it wrong: that file is invisible to Kubernetes, grows uncontrolled inside the container and disappears on restart.

1. The process writes a line to stdout.
2. The container runtime (containerd or CRI-O) captures that output.
3. The runtime writes it to a file on the node, in CRI format.
4. The kubelet manages that file: it rotates it and creates symbolic links with meaningful names.
5. kubectl logs asks the kubelet to read that file and returns it.

Let us go into one of our minikube nodes and look at them:

The file name is the key to everything. Its structure is:

From it the collector extracts, without asking anybody, the pod, the namespace and the container of every line. It is what makes the enrichment in section 6 possible.

Note too that bookings-postgres-0 has two files: one for the postgres container and another for the metrics-exporter sidecar we added in 06-04. Each container has its own log stream.

Each line has four space-separated fields:

The P tag appears when a line exceeds 16 KB: the runtime splits it. A badly configured collector will treat each fragment as a separate line. Fluentd and Fluent Bit know how to reassemble them, but it has to be enabled.

Without rotation, a chatty container would fill the node's disk and cause pod evictions. The kubelet handles it with two parameters in its configuration:

With the default values, each container keeps at most 50 MB of logs on the node. bookings-api under load generates around 200 MB a day per replica: that means it keeps less than six hours of history.

The direct and very concrete consequence: if the incident happened at 03:14 and you look at 11:00, the logs have already rotated and no longer exist, even if the pod has not restarted. It is an additional reason, on top of the four in section 1, to centralize.

Even with EFK in place, kubectl logs is still the first thing you run when there is a problem. It is worth mastering.

The --prefix flag puts the pod name in front of each line, essential when using -l:

Combinations that are useful day to day:

Rule of use: kubectl logs for what is happening now; the centralized stack for what happened.

There are three ways to collect logs in Kubernetes. Only one is right for the general case.

We choose the first, and we already have it deployed: in 06-02, while studying DaemonSets, we deployed a log collector with one pod per node, tolerations for the control plane and a hostPath mounted over /var/log/containers, announcing that the complete stack would be built here. The moment has arrived.

The collector's four responsibilities:

1. Read the files in /var/log/containers/, following new writes and remembering where it had got to after a restart.
2. Parse the CRI format and, if the content is JSON, expand it into fields.
3. Enrich with the Kubernetes metadata: namespace, pod, container, node, pod labels and annotations. These are not in the file: they are obtained by querying the API using the file name.
4. Send to the store, with buffering and retries so that nothing is lost if the destination goes down.

Enrichment is what turns a line of text into a queryable document:

Now you really can search for "every bookings-api error in pro between 03:00 and 04:00", which is what we needed.

We have put an aggregator between the agents and Elasticsearch. It is not mandatory, but in production it is justified:

• The node agents (Fluent Bit) stay extremely light: just read and forward.
• The heavy parsing, the masking of sensitive data and the routing happen in a single place, easier to audit and to change.
• Elasticsearch receives connections from 3 aggregators instead of 30 agents, which greatly reduces the pressure.
• The aggregator acts as a shock absorber: if Elasticsearch goes down for half an hour, the aggregator's buffer holds the data.

Both are CNCF projects and from the same family. The confusion about which to use is constant.

The rule of thumb the industry follows:

Fluent Bit as the agent on every node (DaemonSet): it consumes almost nothing, and multiplied by 30 nodes that difference matters a great deal.

Fluentd as the central aggregator (Deployment): where you need the flexibility of the thousand plugins, complex routing and expensive transformations.

For Rutas Norte with three nodes, honestly, Fluent Bit alone would be enough. We build both tiers because the masking of personal data in section 11 benefits greatly from being centralized, and because it is the architecture you will find in any serious cluster.

A cost comparison in our cluster:

We pick up and complete the DaemonSet from 06-02:

A security note, which we will pick up again in module 8. This DaemonSet mounts a hostPath over the node's filesystem. Even read-only, anybody who can run an exec in this pod can read the logs of every container on the node, including those of other namespaces. The collector is a high-value target: it must have its own namespace, minimal RBAC and very restricted access.

This is the detail that causes the most frustration and that is most appreciated when solved.

A Node.js exception reaches the log like this:

To the runtime these are five independent lines. Without the multi-line filter, Elasticsearch receives five documents:

• One with Error: connection pool exhausted, with no stack trace at all.
• Four with stack trace fragments, with no context about which error produced them.

And in Kibana, sorted by date among the logs of another five pods, reconstructing the exception becomes impossible. With twenty lines of stack trace, the problem multiplies.

The multiline filter recognises that lines starting with spaces and at are continuations of the previous one and joins them into a single document with the complete stack trace in one field. It is the difference between being able to debug and not.

The retry_forever true in production and false outside it is a conscious design decision: losing rutas-norte-dev logs during an Elasticsearch outage is acceptable; losing production ones is not.

The collector is critical infrastructure: if it fails silently, you lose your logs without noticing.

And the corresponding alert, following the criteria from 07-04:

Elasticsearch is a distributed search engine that indexes JSON documents. For our purposes, each enriched log line is a document.

Logs are time-based data that age: today's are queried constantly, those from three weeks ago hardly ever, and those from a year ago probably never. That is why they are not all kept in one index, but in daily indices:

The decisive advantages of this split:

• Deleting is instantaneous. Removing a whole index is a metadata operation. Deleting individual documents inside a large index is extremely slow and expensive.
• Date queries only touch the necessary indices. Searching the last 24 hours does not traverse 30 days of data.
• Each index can have a different configuration: the recent ones on fast disks, the old ones compressed.

A template defines how any new index matching a pattern is configured. Without one, Elasticsearch guesses each field's type, and it guesses wrong.

The distinction between keyword and text is fundamental and confuses everybody:

If level were text, you could not run a "how many logs per level" aggregation, which is exactly what you want on a dashboard. If message were keyword, you could not search for "pool" inside "connection pool exhausted".

refresh_interval: 30s is an important performance setting: the default is 1 second, which forces Elasticsearch to make every document visible almost instantly, at a high cost. For logs, a 30-second delay is perfectly acceptable and multiplies write throughput.

Index lifecycle management automates ageing. Without it, somebody has to remember to delete old indices, and nobody remembers until the disk fills up.

The critical point: the delete phase. The 30 days are not a technical number, they are a business and regulatory compliance decision. If the logs contain personal data (which they should not, but which is what happens in practice), the retention must match what the applicable regulation requires and what the company's data protection officer has defined. We come back to this in section 11.

This is the part that always gets underestimated.

And the minimum compute resources:

Elasticsearch needs three nodes, not one. With a single node there is no fault tolerance and the whole logging stack goes down when that pod restarts. With two there is a risk of split-brain. Three is the real minimum.

The golden rule for the heap: allocate at most 50 % of the container's memory and never more than 31 GB (above that threshold the JVM loses pointer compression and performs worse with more memory).

Approximate total: 24 GB of RAM and 300 GB of fast disk just to be able to read logs. That is the figure to put on the table before deciding, and the one that motivates section 12.

Before searching for anything you have to tell Kibana which indices to query. Under Stack Management → Data Views → Create data view:

The time field is what enables the time range picker and the histograms. Without it, Kibana cannot sort chronologically.

KQL (Kibana Query Language) is the search bar's language. It is far simpler than PromQL and takes ten minutes to learn.

A quick comparison with what we already know:

Let us go back to the scenario from section 1: the BookingsApiHighErrorRate alert fires at 03:14.

Step 1 — Narrow it down in time. Range picker: 2026-08-06 03:00 to 2026-08-06 04:00.

Step 2 — See the shape of the problem. A broad query and a look at the histogram:

The histogram shows at a glance whether the errors started all at once (a deployment, an outage) or grew progressively (a resource running out).

Step 3 — Identify the component. In the fields panel, click on kubernetes.labels.app to see the distribution:

Step 4 — Find the dominant message. Click on message.keyword:

Step 5 — Open a document and read the complete stack trace. Thanks to the multi-line filter in section 6, the exception is complete in a single document.

Step 6 — Correlate backwards. What happened just before the first error? Remove the level filter and look at the previous five minutes:

That is usually where the cause shows up: a deployment, a CronJob that started, a slow PostgreSQL query.

A complete diagnosis in five minutes. With kubectl logs across six pods, this would have taken an hour, and the logs would probably no longer exist.

Under Dashboards → Create dashboard, with these visualizations:

The heatmap by pod is especially revealing: if the errors concentrate on one pod out of six, the problem belongs to that pod (a node with trouble, a replica with corrupt state). If they are spread evenly, the problem is systemic (the database, an external dependency). That distinction, almost impossible to see with kubectl logs, jumps out here in a second.

Everything above works far better if the applications write JSON rather than free text. It is, by a wide margin, the lowest-cost, highest-impact change in the whole lesson.

Problems with this line, which looks perfectly reasonable:

• To filter by level you have to search for the substring [ERROR], which also matches a message saying "the user saw an [ERROR] on screen".
• The duration 4821ms is text: you cannot query duration_ms > 1000 or calculate percentiles.
• The user identifier is embedded in the sentence: you cannot filter by it.
• There is no trace identifier: you cannot follow this request through the other components.
• If somebody changes the message format tomorrow, every saved filter stops working.

Now you can write:

And aggregations: the average duration of failed requests, the ten journeys with most errors, the evolution of error_type over time.

A critical note about what is not in that JSON: the customer's name does not appear, nor their ID number, nor their phone, nor their email. What appears is trace_id, an opaque identifier. If you need to know which customer it was, you cross-reference the trace_id with the database, in a system with access control. It is a deliberate decision, and section 11 explains why it is mandatory.

Every platform component must emit these fields:

The rule about message: it must be constant for the same type of event, with the variable values in separate fields. "Failed to create booking" with duration_ms: 4821 is far more useful than "Failed to create booking after 4821ms", because it lets you aggregate by message.

And how it is used, with the trace identifier propagated:

Note the last detail: we return the trace_id to the client in the error body. When somebody calls customer support saying "I couldn't buy", that identifier locates the exact request and its whole journey in Kibana:

With no component filter: every log from every component that took part in that request, in order. That is real correlation.

In 06-04 we added a log adapter container to notifications-worker that converts its proprietary log to JSON. Here is where it becomes fully clear why.

notifications-worker uses an old mail-sending library that writes like this, and it cannot be modified without rewriting the component:

Two problems: it is not JSON, and it contains a customer's email address, which is personal data.

The adapter, a sidecar that reads that log and emits clean JSON:

Its output:

Two key transformations: the format becomes queryable JSON, and the email address is replaced by a hash. The hash lets you answer "how many emails have failed for the same recipient?" without the address appearing in any index.

Logs and metrics are two views of the same system, and their value multiplies when they can be cross-referenced. The key is using the same labels in both.

With that correspondence, the investigation flow is direct:

Three practical ways to link the two worlds:

1. A link from the Grafana panel. In the panel options, a Data link that opens Kibana with the filters already applied:

With that, a click on the spike in the graph takes you to that pod's logs in that exact window. It saves an enormous amount of time under pressure.

2. Metrics derived from logs. Fluentd can count events and expose them to Prometheus, which allows alerting on log patterns:

Watch the cardinality, exactly as in 07-03: never use the full message as a label.

3. Grafana as a unified viewer. Grafana can add Elasticsearch as an additional data source and show a metrics panel and a logs panel on the same dashboard. With the Loki data source (section 12) the integration is even tighter.

What is missing to close the circle completely is distributed tracing (OpenTelemetry, Jaeger, Tempo): the third pillar of observability, which records a request's complete journey with the timings of each hop. Our trace_id is a home-made and very useful version of that idea, but tracing is outside this course's scope.

This section is the most important in the lesson, and the one with the most consequences outside the technical sphere.

bookings-postgres stores personal data about Rutas Norte's customers: name, ID number, phone and email address. And as soon as you build a centralized logging stack, everything the applications write is copied, indexed and kept for 30 days in a system many more people can access than the database.

That is the real risk, and it is easy to underestimate. A log as innocent as this one:

has just turned your logging system into a personal data file with all the obligations that entails.

One case that always slips through: the access logs of the Ingress and of web-store. They record the complete URL of every request. If any route carries data in the query string, it gets logged without the application being involved at all. The fix: never put sensitive data in URLs, ever.

None is sufficient on its own. All of them are applied.

Layer 1 — In the code (the most effective). Make sure the data is never written. It is the only defence with no leaks, because if it never leaves the process there is nothing to filter. It requires review in pull requests.

Layer 2 — Redaction in the logging library. Like pino's redact from section 9: a safety net for slips.

Layer 3 — Masking in the collector. The last technical defence before the data is persisted.

That _masked field is more valuable than it looks. With a query in Kibana:

and aggregating by component, you get the list of components that are writing personal data and that need fixing in the code. It turns a defensive measure into an improvement tool.

An honest warning about regular expressions: they are not reliable. An ID written as 12.345.678-Z is not caught by the pattern above. A person's name has no detectable pattern at all. The only real defence is layer 1.

Layer 4 — Access control and retention. Even with no leaks, restrict who can see the production logs, with indices separated by environment (as we did in the aggregator) and Elasticsearch roles that grant access only to what is needed.

The 30-day retention from section 7 is a decision that combines three criteria:

And a rule that often comes as a surprise: if the logs contain personal data, keeping them "just in case" is not acceptable. Data protection regulation requires personal data to be kept only for as long as is necessary for the purpose that justified collecting it, and "in case we ever need to debug" is not usually a valid purpose.

This is a point that cannot be settled by technical judgement alone.

This lesson's configuration — what is logged, how long it is kept, who can consult it and from where — must be reviewed and approved by the organisation's compliance and data protection officer before going into production.

Rutas Norte processes personal data about its customers (name, ID number, phone and email address), which places the platform within the scope of the GDPR and of national data protection regulation. A centralized logging system that captures that data, even accidentally, has concrete consequences:

• It becomes a processing operation on personal data that must appear in the record of processing activities, with its legal basis and purpose documented.
• It requires a justified and technically enforced retention period, not indefinite retention "just in case".
• It obliges you to control and record access: who consults the production logs and for what purpose.
• It may require an impact assessment if the volume or nature of the processing justifies one.
• It considerably complicates the exercise of the rights of erasure and access: if a customer's ID number appears in twenty distributed indices, handling an erasure request becomes technically very expensive.
• If Elasticsearch is hosted outside the European Economic Area, it triggers the obligations on international data transfers.

What to do, in practice:

1. Document in writing which fields each component logs and present it to the person responsible for compliance.
2. Agree with them the retention period for each index, and apply it in the ILM policy.
3. Define who has access to the production indices, and review it periodically.
4. Audit with the _masked: true query which components are still emitting sensitive data and fix them in the code.
5. Include a log review in the checklist for any new feature that handles customer data.

The technical team provides the mechanisms — masking, retention, access control, separation by environment — but the decision about what is acceptable to log and for how long is not a technical decision.

Recapping section 7, the minimum EFK stack for Rutas Norte:

Plus the human cost: somebody has to maintain Elasticsearch, size the shards, watch the cluster's health, manage the upgrades and respond when it goes red. Elasticsearch is not a component you install and forget.

For a platform with six components and three nodes, that is a considerable cost. It is worth asking whether there is something lighter.

Loki is Grafana Labs' logging system, with a radically different design idea:

Loki does not index the log content. It only indexes the labels.

It is "Prometheus for logs": the same label model, the same query language (LogQL, very similar to PromQL) and storage in cheap object stores (S3) instead of fast disks.

When to choose each:

An honest recommendation for Rutas Norte: with six components, three nodes and a small team, Loki would be the more sensible choice. We have built EFK because it is what you will find in most established companies and because it teaches the concepts (indices, templates, ILM, mappings) that then apply in any system. But if you were starting from scratch tomorrow, start with Loki.

The third route is to run nothing yourself:

The advantage: zero operations. The drawbacks: a per-GB ingestion cost that skyrockets with volume, and — important for section 11 — the data leaves your infrastructure, which has to be reviewed with the compliance officer, especially if the provider stores it outside the EEA.

Before sizing anything, reduce the volume. It is free and it always works:

Complementary measures:

• Level info in production, debug only in rutas-norte-dev.
• access_log off; for the nginx health endpoint, as we already did in 07-01.
• Shorter retention for dev and pre (7 days) than for pro (30 days).
• Periodically review which component generates the most volume:

1. The application writes to a file instead of to stdout. It breaks the Kubernetes contract: the collector does not see it, the file grows inside the container and disappears on restart. If you cannot change the application, use an adapter sidecar like the one for notifications-worker.

2. Not configuring multi-line reassembly. Every exception stack trace becomes twenty disconnected documents, exactly what you most need to read whole during an incident.

3. The collector reads its own logs. An infinite loop: every log generated produces another log. The Exclude_Path from section 6 is not optional.

4. Elasticsearch with a single node. No quorum, no fault tolerance, and the whole logging stack goes down when that pod restarts. Three nodes is the real minimum.

5. A badly sized Elasticsearch heap. At most 50 % of the container's memory, and never more than 31 GB. Above that threshold the JVM loses pointer compression and performs worse with more memory.

6. No ILM policy. The indices pile up until the disk fills. When that happens, Elasticsearch goes read-only and you stop receiving logs exactly when they are most needed.

7. Mapping as text what should be keyword. Without keyword you cannot aggregate by that field, and aggregations are half the value of Kibana.

8. Logging personal data. The mistake with the greatest non-technical consequences. Apply the four layers of defence and, above all, do not write it in the code.

9. Not defining a field standard. If each component uses level, severity and lvl for the same thing, there is no query that works for all of them. Agree the standard before instrumenting.

10. Not propagating the trace_id. Without it, correlating a request across six components is impossible however big your logging stack is.

11. No buffer in the collector. If Elasticsearch goes down for ten minutes and there is no buffer, you lose ten minutes of production logs, probably the most interesting ones.

12. Not monitoring the logging stack. The collector can be silently dropping records. Use the PodMonitor and the alert from section 6: apply what you learned in 07-03 and 07-04 to the observability infrastructure itself.

13. Logging too much. The debug level in production multiplies volume and cost tenfold, and makes it harder to find what matters among the noise.

web-store (nginx) generates access logs in combined format:

1. List every problem with this line, including the data protection ones.
2. Write the nginx configuration that emits the same event in JSON, conforming to the Rutas Norte field standard.
3. Write the Fluent Bit parser you would need if you could not change the nginx configuration.
4. What would you do with the national_id parameter in the URL?

An internal audit reveals that the rutasnorte-pro-2026.08.* index contains 47,000 documents with customers' email addresses in clear text. The logs come from notifications-worker and from bookings-api.

1. Write the KQL query that locates the affected documents.
2. List, in order of priority, the actions to take in the next 24 hours.
3. Write the masking filter that prevents it happening again, and explain why it is not enough.
4. What is the compliance officer's role in this incident and at what point must they be involved?

Rutas Norte expands: it goes from 3 to 12 nodes, from 6 to 15 components, and the log volume multiplies by 5 compared with the section 7 estimate. The infrastructure budget does not multiply by 5.

1. Recalculate the daily volume and the storage needed for 30 days.
2. Propose three measures to reduce the volume without losing diagnostic capability, with an estimate of the saving from each.
3. Compare EFK and Loki for this specific scenario, with numbers.
4. Recommend a final architecture and justify it.

Solution 1

1. Problems with the line.

Format problems:

• It is not JSON: every field has to be extracted with regular expressions, fragile against any change.
• The date uses the nginx format (06/Aug/2026:03:14:22 +0000), not ISO 8601: it needs a specific parser.
• The duration 4.821 is in seconds, not milliseconds, and breaches the Rutas Norte standard (duration_ms).
• There is no trace_id: impossible to correlate this request with the bookings-api one.
• There is no component or level: you cannot filter by severity.

Data protection problems (the serious ones):

• The URL contains an ID number: ?national_id=12345678Z. It is recorded in the nginx log, in the Ingress log and in any intermediate proxy. This is the most serious problem.
• The client's IP (83.45.12.99) is personal data under the GDPR. It must be treated as such: anonymised or its retention justified.
• The User-Agent contributes to browser fingerprinting and, combined with other data, can be identifying.

2. The nginx configuration in JSON.

Key points of this configuration:

• $uri instead of $request: $uri is the path without the query string, so the ?national_id=... never reaches the log. It is the fix for the most serious problem.
• escape=json is mandatory: without it, a User-Agent containing quotes breaks the JSON and Fluent Bit cannot parse it.
• access_log off on the health probe, consistent with what we did in 07-01.

3. The parser if nginx cannot be changed.

And a Lua filter that normalises to the standard and strips the query string:

4. What to do with the national_id in the URL.

The correct answer has three levels, and only the first solves the problem at its root:

Level 1 (the real fix): change the application. An ID number must never travel in a URL's query string. URLs are recorded in the browser, in the history, in the Referer sent to third parties, in the nginx and Ingress logs and in any proxy. The query must be a POST with the data in the body, or use an opaque identifier.

Level 2 (immediate mitigation): do not log the query string. Use $uri instead of $request, as in solution 2. It takes minutes to implement and cuts off the leak into the logs.

Level 3 (safety net): mask in the collector. The Lua filter from section 11 detects the ID pattern and replaces it. It is the last defence, and it is not reliable on its own: an ID written as 12.345.678-Z slips past it.

Apply all three. And very importantly: the logs that already contain the ID number are still there. They have to be deleted, and that takes us to the next exercise.

Solution 2

1. The KQL query to locate the documents.

More precise, taking advantage of the masking flag:

And to quantify it and locate the source, an Elasticsearch aggregation:

That aggregation gives you exactly which component, how many documents and from what day: the three facts you need for the report.

2. Actions within 24 hours, by priority.

Hour 0–1 — Contain and notify.

1. Notify the compliance officer immediately. It is not a technical decision that can be deferred: breach notification deadlines are short and the clock starts running from the moment you become aware.
2. Restrict access to the affected indices to the incident team only, through Elasticsearch roles.
3. Document the scope: which components, which fields, how many documents, from what date, who has accessed those indices (the Elasticsearch audit logs).

Hour 1–4 — Stop the leak.

4. Deploy the masking filter in Fluent Bit (point 3) so that no new documents get in. It is quick and requires no changes to the applications.
5. Verify that new documents are now arriving masked, by checking the day's index.

Hour 4–12 — Fix the source.

6. Locate in the code the logger calls that emit the address. In notifications-worker it is in the legacy SMTP log; in bookings-api, probably in a log.info(booking) that dumps the whole object.
7. Fix the code: replace the address with a hash, as the adapter in section 9 does.
8. Add redact to the logger configuration as a layer 2 safety net.
9. Deploy to pre and then to pro.

Hour 12–24 — Clean up and prevent.

10. Delete or scrub the affected documents, according to what the compliance officer decides:

A warning: _delete_by_query over 47,000 documents is a heavy operation. If the indices are daily and heavily contaminated, deleting the whole index is far faster and safer, at the cost of also losing that day's clean logs.

11. Add an automatic check in continuous integration that rejects a pull request if it detects personal data patterns in logger calls.
12. Write a report for the compliance officer with a timeline, scope, root cause and measures.

3. The masking filter and why it is not enough.

Why it is not enough, for four reasons:

1. Regular expressions slip. marta.ruiz [at] ejemplo.example, an address split across two fields, or one with unusual characters do not match the pattern. Masking gives a false sense of security.
2. It only covers what you already know to look for. A person's name has no detectable pattern. Neither does a postal address. The filter protects against emails, ID numbers and cards; against the rest, nothing.
3. The data exists up to the filter. It leaves the process, is written to the node's file, travels to the collector. Anybody with access to the node (or to the collector's pod, which mounts a hostPath) sees it unmasked.
4. It is a mitigation layer, not a prevention one. The only real defence is that the data is never written. Everything else is a net that catches what escapes.

That is why step 7 (fixing the code) is the important one, and the filter only buys time while it is deployed.

4. The compliance officer's role.

When to involve them: within the first hour, before taking any clean-up action. It is a frequent mistake to "fix it first and tell them later": deleting the documents before they have documented the scope may destroy the evidence they need to assess the breach, and the notification deadlines run from the moment you become aware of the event, not from when you finish fixing it.

Their responsibilities in this incident:

• Classify the incident: determine whether it constitutes a personal data breach under the GDPR.
• Decide on notification: whether to notify the supervisory authority (in Spain, the AEPD) within the 72-hour legal deadline, and whether to communicate it to the individuals affected.
• Assess the risk to the rights and freedoms of the individuals affected, taking into account the volume, the nature of the data and who may have had access.
• Decide on the clean-up: what is deleted, what is kept as evidence and for how long.
• Record the incident in the internal register of security breaches, mandatory even when notification is not required.
• Approve the corrective measures and verify that they have been implemented.

And going forward, their role is preventive: reviewing and approving which fields are logged, the retention periods and the access policy, as set out in the warning in section 11. The technical team provides the mechanisms; the decision about what is acceptable to log is not a technical one.

Solution 3

1. Recalculating the volume.

An increase of roughly three times the original cost, plus a qualitative jump in operational complexity: with 5 data nodes you have to manage shards, rebalancing and continuous upgrades.

2. Three reduction measures.

Measure A — Sampling successful logs. Estimated saving: 45 %.

90 % of the logs are requests that went fine and that nobody will ever look at. Keeping 10 % of them is statistically enough to see trends, and 100 % of the errors.

The _sampled field lets you multiply the counts by 10 when aggregating and get correct figures.

Measure B — Tiered retention by environment and by level. Estimated saving: 30 %.

It is implemented with two different data streams, routed in the Fluentd aggregator by the level field, each with its own ILM policy.

Measure C — Discard noise at source. Estimated saving: 20 %.

The combined effect (the measures are not purely additive because they overlap):

And the important part: with no loss of diagnostic capability, because 100 % of the errors, 100 % of the slow requests and 100 % of anything touching bookings are kept.

3. Comparing EFK against Loki with numbers.

On the already optimised volume of 15.8 GB/day:

The decisive figure is in the comparison of the two search rows: Loki is slow at free-text searches across the whole history, but just as fast in the case that represents 95 % of real usage, which is "logs from this component, in this time window, filtered by level". And that is exactly how an incident is investigated: you never search for a string across 30 days of every component; you always narrow by component and by window, because the alert already gave you both.

4. The recommended architecture.

Recommendation: migrate to Loki, with the three reduction measures applied.

The justification, in five points:

1. Cost: €190 against €750 a month. The difference (€6,700 a year) is hard to justify when the predominant use case performs the same on both.
2. Operational complexity. Elasticsearch with 5 data nodes needs somebody who knows how to manage shards, rebalancing and upgrades. Loki, far less so. With a small team, that time is worth more than the cost difference.
3. Integration with what we already have. Grafana has been deployed since 07-04. Loki appears as one more data source, and the same dashboard can have a metrics panel above and a logs panel below, with the same time window. That saved context switch during an incident is worth a lot.
4. The label model is the one we already know. LogQL is so similar to PromQL that the team learns it in an afternoon, whereas KQL and the Elasticsearch aggregations are a separate body of knowledge.
5. The collection architecture is preserved. Fluent Bit remains the per-node agent, with the same parsing, multi-line and masking configuration. The migration only changes the output destination, which makes it low-risk.

When NOT to follow this recommendation:

• If the team already runs Elasticsearch for other things (the website's search engine, analytics), the marginal cost of adding logs is much lower.
• If there are audit requirements demanding free-text search over the whole history with guaranteed response times.
• If complex aggregations over log fields are needed routinely.

A low-risk migration plan:

1. Deploy Loki in parallel, without touching EFK.
2. Configure Fluent Bit with two simultaneous outputs for two weeks.
3. Rebuild in Grafana the error dashboards that were in Kibana.
4. Validate with a real incident that Loki answers the necessary questions.
5. Retire EFK, keeping a copy of the production indices until the retention agreed with the compliance officer runs out.

That last point is not a detail: you cannot delete EFK early if its retention is committed to compliance.

Rutas Norte no longer loses its history. In this lesson we have:

• Understood why kubectl logs is not enough: it is lost when the pod is recreated, it does not search across components, it does not correlate and it disappears with the node. And we have seen an additional, little-known limit: with the kubelet's default rotation, a chatty component keeps less than six hours of logs on the node.
• Seen how logging works under the hood: the application writes to stdout, the runtime stores it in /var/log/containers/ with a meaningful name from which the pod, namespace and container are extracted, and the kubelet rotates it.
• Mastered kubectl logs as a front-line tool, with --previous as the decisive flag in the face of a CrashLoopBackOff.
• Built the per-node collection architecture on top of the DaemonSet we deployed in 06-02: Fluent Bit as the lightweight agent on each node and Fluentd as the central aggregator, with the kubernetes filter that enriches every line and the multi-line reassembly that stops an exception turning into twenty disconnected documents.
• Configured Elasticsearch with daily indices, templates that distinguish keyword from text, and an ILM policy with hot, warm, cold and delete phases; and we have sized the stack with realistic numbers: some 24 GB of RAM and 300 GB of disk just to be able to read logs.
• Used Kibana with KQL to go from an alert to the root cause in five minutes, following a concrete workflow: narrow down, see the shape, identify the component, find the dominant message, read the stack trace and correlate backwards.
• Adopted structured JSON logs as the decision with the best return, with the Rutas Norte field standard and the trace_id that lets you follow a request across the six components; and we have closed the circle on the notifications-worker log adapter we introduced in 06-04.
• And, above all, we have established what must never be logged: credentials, payment data and customers' personal data. With four layers of defence — the code, the library, the collector and access control — knowing that only the first is truly reliable, and with the explicit warning that the logging configuration must be reviewed and approved by the compliance officer before it reaches production.
• Compared the real cost of EFK with the lighter alternatives, with the honest conclusion that for a platform the size of Rutas Norte, Loki would be the more sensible choice today.

We now have the three signals: the probes say whether a component is healthy, the metrics say how much and how well it is working, and the logs say exactly what happened. What we still do not have is a method for using them together.

Because when the web store returns 502 at 03:14 and the phone rings, knowing how to use Grafana and Kibana is not enough: you need a procedure that goes from symptom to cause without wandering, you need to know that Kubernetes events expire after an hour and have to be captured beforehand, and you need a mental table of symptom → probable cause → the command that confirms it. In 07-06, the module's last lesson, we will build that methodology and apply it, step by step, to a real Rutas Norte incident.