In 06-04 you built observability's first pillar. AlpinaShop now has a dashboard, uptime checks from four continents and three alerts that reach Marta's phone. It is an enormous leap from waiting for a customer's email.

But look at what happens when that alert really fires. It is eleven at night and Marta's phone shows: "5xx error rate above 2 % for 5 minutes". Marta opens the dashboard. Confirmed: 4.3 % of requests are failing, the p95 latency has gone from 400 ms to 3.2 seconds, and it started about twenty minutes ago. And now what?

The metric has told her that something is wrong. It does not tell her what. It does not tell her which exception is being raised, or which customers it is happening to, or whether it is in the catalogue or the basket, or whether the culprit is the application, the database or a call to an external API. For that you need the other two pillars.

This lesson builds them. And it ends with the complete journey through that incident, from the alert to the exact line of code, because that correlation — jumping from the alert to the metric, from the metric to the log and from the log to the trace — is the reason observability exists.

1. The structured log entry
2. Where logs come from automatically
3. Emitting logs properly from the Flask application
4. The Logs Explorer and its query language
5. Queries that solve real AlpinaShop incidents
6. Buckets, retention, views and scopes
7. Sinks: exporting to BigQuery, Cloud Storage and Pub/Sub
8. Exclusion filters: not paying for noise
9. Log-based metrics
10. Cloud Trace: what distributed tracing is
11. Instrumenting Flask with OpenTelemetry
12. Reading a waterfall and finding the slow query
13. Sampling and the cost of traces
14. Cloud Profiler: continuous profiling
15. The correlation that ties it all together: an incident end to end
16. Cost by volume and the three decisions that control it

A log is not a line of text. In Cloud Logging, a log entry is an object with fields, and understanding those fields is what turns logs from a rubbish tip into a queryable database.

The difference between textPayload and jsonPayload is the difference between being able to investigate and not being able to.

Without structure:

It is readable for a human and opaque to a machine. To answer "how many orders failed because of a gateway timeout in the last hour, and for what amount?" you have to parse text with regular expressions, and it only takes somebody changing the message for everything to break.

With structure:

Now that question is a filter. And the second-to-last line makes something even more powerful possible: jumping straight to the complete trace of that request, with everything that happened before and after. We will come back to it.

As with metrics, there is good news: a good part of the logs are already being collected without you having done anything.

Two cells deserve attention.

The load balancer's logs are not enabled by default, and they are among the most valuable there are: they contain every request to the shop with its latency, its code, its country of origin and the CDN cache result. To enable them:

The --logging-sample-rate=1.0 records 100 % of requests. That is right while the volume is moderate; with millions of daily requests you drop to 0.1 or 0.05 and accept losing detail in exchange for cost. Start at 1.0 and adjust when you see the bill.

VPC Flow Logs record network connections and are indispensable for diagnosing connectivity problems and for investigating security incidents. Their volume is high, so they are enabled with sampling:

And the audit logs deserve a mention with a forward reference: the admin activity ones — who created, modified or deleted a resource — are always on and are free. The data access ones — who read what — have to be enabled, they generate an enormous volume and they are charged for. The audit policy at the organization level, with its compliance implications, is the subject of 07-07; here it is enough to know they exist and that they answer the question "who deleted that resource?".

This is where AlpinaShop's team has the most room for improvement. The catalogue does this today:

It works in the sense that the text ends up in Cloud Logging — on GKE, everything that goes to stdout is collected. But it produces entries with textPayload, all with INFO severity even when they are errors, with no queryable field and no trace identifier.

The right way, with the client library:

And how it is used, with the data as fields and not as text:

The five rules of a useful log, which are worth more than any library:

The third rule is not negotiable and goes beyond good taste. Logs are retained, exported to BigQuery, read by several people and survive for months. An email address or a card number written into a log is a personal-data leak with GDPR implications, consistent with everything seen in 04-07 about DLP. If you need to identify a customer in the logs, use a stable hash that allows correlation without identification.

On severity, a practical criterion that avoids endless arguments:

The distinction between WARNING and ERROR is the one most often got wrong: a retry that then works is WARNING — the user never noticed; an exhausted retry that returns an error to the customer is ERROR.

The Logs Explorer is the query interface, and its language is simple but has details worth knowing.

The basic operators:

Three performance tips that change the experience a great deal, because a badly written query over weeks of logs takes minutes:

1. Always narrow the time first. It is by far the most efficient filter.
2. Filter by resource.type early. It cuts the search space down at a stroke.
3. Avoid global free-text search if you can filter on a specific field: searching in jsonPayload.causa is orders of magnitude faster than searching for the word across all the content.

From the command line, for scripts and for automation:

The theory is short; what is useful are the patterns. These five queries cover most real investigations.

Query 1 — The 500 errors since the last deployment. The first question in any incident:

And refined, to see whether the problem belongs to a specific version — which links up with $COMMIT_SHA from 06-01:

Query 2 — Isolate the request of a customer who has complained. A customer writes saying their purchase failed at 20:34:

Here you see the value of the personal-data rule: the hash lets you find the customer without ever having written their email into a log. Customer support turns the email into a hash with the same function, and the investigation works just the same.

Query 3 — Who deleted a resource. The audit logs, which answer the most uncomfortable question:

The answer includes protoPayload.authenticationInfo.principalEmail — who — and protoPayload.resourceName — what. It is the query that settles in thirty seconds arguments that would otherwise last an afternoon.

Query 4 — Slow load balancer requests with their origin. Over the access log from section 2:

And the same log answers a cost question from 03-03, the CDN hit ratio:

Query 5 — Errors from the image function. Closing the circle with 06-03:

And the one that would have caught the infinite loop from that lesson's exercise, counting invocations per hour:

Logs are stored in log buckets, which have nothing to do with Cloud Storage ones.

Every project has two by default:

And you can create your own buckets, which is what is needed when different kinds of log require different treatment:

Choosing the retention has two dimensions worth separating:

Keeping years of logs in a log bucket is the most expensive way of keeping them. For long retention, the right pattern is exporting to Cloud Storage with a cold storage class, which is the subject of the next section.

Log views let you grant access to a subset of a bucket. It is the piece that solves a real permissions problem: giving Lucía access to the application logs without exposing the audit logs or the payment ones to her.

Without views, log read permission is all or nothing, and "all" includes the audit logs. With views, access is narrowed, consistent with the least privilege from 03-04.

A sink is a rule that says: "entries matching this filter, send them to this destination as well". It is the mechanism that connects logs with the rest of the platform.

To BigQuery, to analyse. AlpinaShop's case: analysing the load balancer's access log with SQL, joining it with the data in alpinashop_analitica:

The writerIdentity step is forgotten constantly: the sink is created, it seems to work, and nothing arrives at the destination because the permission is missing. It is the first place to look if a sink is not delivering.

The --use-partitioned-tables matters for cost: it partitions by date, so a query over one day does not scan months, applying what was learned in 04-01.

And then you can do analyses that would be impossible with the explorer:

To Cloud Storage, to archive cheaply. The destination for long retention:

With a lifecycle rule on the bucket (02-02) dropping to Nearline at 30 days, Coldline at 90 and Archive at a year, seven years of payment logs cost a tiny fraction of what they would cost in a log bucket.

To Pub/Sub, to react. The most interesting sink conceptually, because it closes the circle with 06-03:

Every time somebody touches a firewall rule or an IAM policy, a message arrives on the topic, and a Cloud Function from 06-03 can post it to the security channel in Slack. From a log to an automatic reaction, with no human involvement.

A sink copies; an exclusion filter discards. And it is the most direct lever on the logging bill.

The reasoning is simple: some logs are generated in enormous volume and add nothing to diagnosis. At AlpinaShop the main ones are the health checks — hc-catalogo hits /salud every few seconds from several probes, generating tens of thousands of identical entries a day — and the requests for static assets served by the CDN.

The percent=95 parameter is very useful and little known: it excludes a random sample instead of everything. Keeping 5 % lets you still see trends and detect problems with the static assets, while paying a twentieth.

And the indispensable warning: an excluded entry is not stored anywhere and cannot be recovered. If you exclude something that later turns out to be necessary for investigating an incident, there is no way back. Review every exclusion with the question: could I need this during an investigation? When in doubt, sample rather than exclude outright.

Here the link with 06-04 is closed. A log-based metric turns entries matching a filter into a Cloud Monitoring metric, without touching the application's code.

It is the fastest technique for instrumenting something that is already being logged.

A counter metric, for counting occurrences:

With labels so it can be broken down, defined from a file:

Notice the version label: it lets you answer "did these failures start with yesterday's deployment?" immediately, correlating with the SHA from 06-01. And it respects the cardinality rule from 06-04: canal has three values, version a few dozen. Never extract id_pedido as a label.

A distribution metric, for numeric values:

That lets you plot the p50 and p95 of the average order value, a pure business figure obtained without writing a line of specific instrumentation.

The practical recommendation: start with log-based metrics to validate quickly what is worth measuring, and promote to custom metrics the ones that turn out to matter. And keep the risk in mind: if somebody renames a jsonPayload field, the metric silently stops counting and the associated alert stops firing. A change to a log's format is a change with consequences, and it deserves a mention in code review.

The metrics say the p95 is 3.2 seconds. The logs say there were errors. Neither says where those 3.2 seconds went.

That is the gap distributed tracing fills, and the need for it grows with the number of pieces. A request to AlpinaShop's catalogue today crosses: the load balancer, Cloud Armor, the CDN, the GKE pod, a query to Cloud SQL, a Firestore read for the basket and perhaps a call to the Vision API. If it takes three seconds, whose fault is it?

Two concepts:

• A trace represents one complete request through the whole system, identified by a trace_id.
• A span represents an operation within that trace, with a start, an end and a parent span. Spans form a tree.

Just looking at the diagram, the answer jumps out: SELECT stock_por_talla takes 3,020 of the 3,240 ms. No metric and no log would have pointed at that with such clarity.

Context propagation is what allows spans from different services to form a single trace. The service that starts the request generates a trace_id and sends it in a header; each service that receives it reads it, creates its spans as children and propagates it onwards.

The GCP load balancer adds X-Cloud-Trace-Context automatically to every incoming request, so AlpinaShop already has trace identifiers circulating without having done anything. What is missing is for the application to use them.

OpenTelemetry is the industry standard for instrumentation — metrics, traces and logs — and it is vendor-independent. Instrumenting with OpenTelemetry and exporting to Cloud Trace means that if AlpinaShop changes destination tomorrow, you change the exporter and not the instrumentation.

With those last three lines you already have useful traces without modifying a single business function. For your own detail, manual spans:

And the piece that changes everything: including the trace_id in the logs. It is what will make the journey in section 15 possible:

Those special keys logging.googleapis.com/trace and .../spanId are not just any fields: Cloud Logging recognises them and fills in the entry's trace and spanId fields. And with that, the console shows a direct link from each log to its trace, and from each trace to its logs.

With the instrumentation in place, this is how you read a slow trace from AlpinaShop's catalogue:

The diagnosis is immediate, and the three things to look at in any waterfall are always the same:

1. Which span takes the largest percentage? Here, stock_por_talla with 93 %. Optimising anything else is a waste of time.
2. Are there uncovered gaps? An interval inside the parent span that no child accounts for is usually waiting time: locks, connection pool contention or uninstrumented code.
3. Are there repeated spans? Fifty identical consecutive SELECT spans are the unmistakable signature of the N+1 problem: one query per element of a list, instead of one query for all of them.

With the SQL attribute added by the SQLAlchemy instrumentation, the trace includes the query:

And from there the investigation is a database one: EXPLAIN ANALYZE on that query, review indexes, look at Cloud SQL's slow query logs. The trace does not fix the problem; it locates it in thirty seconds instead of three hours, and that is exactly what is asked of it.

One pattern deserves a separate mention: if that same query takes 40 ms in a test environment and 3,020 ms in production, the problem is probably not the query but contention — the connection pool exhausted, with the request waiting for one to be released. There the Cloud SQL connections metric from 06-04 and the trace reinforce each other: the trace says where the waiting happens, the metric says why.

Tracing 100 % of requests has two costs: span ingestion, which is billed, and performance in the application, which although small is not zero. That is why you sample.

The ParentBased(TraceIdRatioBased(0.1)) combination from section 11 is the right default, and it deserves an explanation: TraceIdRatioBased(0.1) samples 10 % of new traces, and ParentBased guarantees that if a trace was sampled at the start, every service that receives it samples it too. Without ParentBased, each service would decide on its own and you would get incomplete traces with holes, which are worse than having no traces.

Tail-based sampling is what everybody would want — keeping 100 % of the slow or failing traces and 1 % of the normal ones — because the interesting traces are precisely the anomalous ones. It requires an OpenTelemetry collector that holds the spans until the end of the request in order to decide. For AlpinaShop today it is excessive complexity; it is worth knowing it exists for when the system grows.

Recommendation per environment:

Making the ratio an environment variable (TRACE_SAMPLE_RATE) is deliberate: it lets you raise it to 100 % during an investigation without redeploying code, and lower it afterwards.

A trace says a function takes 800 ms. It does not say what it does during those 800 ms. That is what profiling is for.

Cloud Profiler continuously collects CPU and memory samples from the application in production, with very low overhead — of the order of a small percentage — and shows a flame graph of where time and memory are going.

The three cases where Profiler solves what no other pillar can:

The third deserves a warning: intuition about where the bottleneck is is usually wrong. It is common to spend two days optimising a function that consumes 3 % of the total time while 60 % goes on a JSON serialisation nobody looked at. Profiler avoids that waste with data instead of guesswork.

And comparison by version is where it shines: with service_version set to the image SHA, you can compare two versions' profiles and see exactly what introduced a performance regression.

This section is the whole module's reason for being. We return to the moment the lesson started with.

23:14 — The alert. It reaches Marta's phone. It carries the documentation field with the first four steps, so she does not start from scratch.

23:15 — The dashboard. Confirmed: 4.3 % errors, p95 of 3.2 s against the usual 400 ms, and the MIG's CPU low. That last fact is informative in itself: if there were a traffic spike, the CPU would be high. It is not, so the problem is not one of capacity.

23:17 — The logs. First query, the one from section 5:

About 800 entries come up. Marta looks at one:

23:19 — The pattern. She refines the query grouping by cause and discovers that 94 % of the errors have causa="timeout_consulta" and all of them are on catalogue routes. It is not a general failure: it is one specific operation.

23:21 — The trace. And here is the jump that justifies all the instrumentation from section 11: in the console, that log entry's trace field is a link. One click and the complete trace of that exact request appears, the one belonging to the specific customer who suffered the error.

23:23 — The waterfall. The trace is the one from section 12: SELECT stock_por_talla consumes 3,020 of 3,240 ms. With the SQL query visible in the span's attributes.

23:25 — The cause. Marta runs query 3 over the audit logs and over the Cloud Build history: at 22:50 a database migration was applied that, among other things, dropped and recreated an auxiliary table without recreating the index on (sku, almacen). Without that index, the query goes from an indexed lookup to a full scan.

23:31 — The mitigation. The index is recreated, and in parallel the version rollback is prepared with the pipeline from 06-01. Latency returns to normal in three minutes.

Seventeen minutes from the alert to the root cause identified. Without this journey, the same investigation would have been: find out from a customer the next morning, look at the website, fail to reproduce the problem because night traffic is different, trawl through unstructured logs searching for text, suspect three wrong things and, with luck, hit on it by the end of the day.

The lesson to take away: no piece works on its own. An alert without logs only produces anxiety. Logs without structure are a rubbish tip. Traces without correlated logs force you to search blindly for which one to look at among millions. The value is in the links between the pieces, and the specific link that makes the whole journey possible is one line of code: including the trace_id in every log entry.

The model is simple: you pay per GB ingested, with a free monthly volume. Retention within the default period is not billed separately; extending it is. And always check the current prices in the official documentation.

The three decisions that control the bill, in order of impact:

Decision 1: what gets ingested. By far the most important, because it affects everything else. The exclusion filters from section 8 on health checks and static assets typically cut between 40 % and 70 % of the volume of a typical website, without losing any diagnostic value. And in the application: DEBUG off in production and no logs inside loops.

Decision 2: how long it is kept and where. Thirty days in the log bucket for diagnosis, and whatever has to be kept longer, exported. A sink to Cloud Storage with a lifecycle to Coldline and Archive makes seven years of payment logs cost orders of magnitude less than extending the bucket's retention.

Decision 3: how much is sampled. It applies to the load balancer log (--logging-sample-rate), to VPC Flow Logs (--logging-flow-sampling) and to traces (TRACE_SAMPLE_RATE). At high volume, 10 % is usually enough to detect patterns, because statistics do not need the complete census.

And the mistake to avoid above all others: excluding error logs to save money. They are a minimal percentage of the volume and 100 % of the value during an incident. The savings are in the repetitive noise, never in the exceptional.

A final perspective for calibration: for AlpinaShop, complete observability — logs, traces, metrics, alerts — comes to a small fraction of the cost of the infrastructure it observes. Compared with the seventeen minutes from section 15 versus a whole day of blind investigation, and with the orders saved by detecting an incident in five minutes instead of twelve hours, it is one of the platform's most profitable investments.

Using print instead of structured logging. The text does end up in Cloud Logging, yes, but with no severity, no queryable fields and no trace. It is the difference between being able to investigate and not being able to.

Putting the data in the message instead of in fields. f"order {id} failed" forces you to parse text. "Order failed" plus {"id_pedido": id} lets you filter and aggregate.

Writing personal data into the logs. Emails, names, cards, addresses. Logs are retained, exported and read by a lot of people. Use stable hashes.

Everything at INFO severity. It makes filtering by severity impossible and renders alerts based on severity>=ERROR useless. And a retry that works is WARNING, not ERROR.

Forgetting the writerIdentity when creating a sink. The sink is created without error and delivers nothing. It is the first place to look when a destination is empty.

Excluding logs without thinking twice. What is excluded is not stored anywhere and is not recoverable. When in doubt, sample with percent rather than excluding outright. And never exclude errors or audit logs.

Keeping years of logs in a log bucket. It is the most expensive way. For long retention, a sink to Cloud Storage with a lifecycle.

High-cardinality labels on log-based metrics. The same mistake as in 06-04: EXTRACT(jsonPayload.id_pedido) creates one time series per order. Extract dimensions with dozens of values, not thousands.

Tracing without ParentBased in the sampler. Each service decides on its own and you get incomplete traces with holes, which confuse more than they help.

Not including the trace_id in the logs. It is one line of code and it is the piece that turns three independent tools into a diagnostic system. Without it, jumping from the log to the trace is impossible.

Optimising by intuition instead of by profile. Two days improving a function that consumes 3 % of the time. Measure first with Profiler.

A final tip: the instrumentation is tested before the incident. Cause an error on purpose in alpinashop-dev, find its log, jump to its trace and check that the complete journey works. Discovering at eleven at night that the trace field is empty is discovering it at the worst possible moment.

This is the real code of a function in AlpinaShop's catalogue:

Rewrite it with structured logging fixing every problem you spot, justify each change, and write the Logs Explorer query that answers "how many returns were rejected for an incorrect status this week, and for what average amount?".

AlpinaShop generates approximately: 80 GB/month of load balancer logs, 15 GB/month of application logs, 30 GB/month of VPC Flow Logs, 2 GB/month of admin audit logs and 5 GB/month of Cloud SQL logs. Requirements: payment transaction logs must be kept for 7 years by regulation; the team needs 30 days for diagnosis; Lucía wants to analyse the access log with SQL over the last 12 months; and cost has to be reduced as much as possible without losing diagnostic capability. Design the complete configuration of buckets, sinks and exclusions, with a qualitative estimate of the saving.

A customer reports that a product page takes 8 seconds. Looking for the trace, Marta finds: root span GET /catalogo/crampon-02 of 8,100 ms; inside it, query_product of 120 ms and render_template of 90 ms; and nothing else. The remaining 7,890 ms are not covered by any child span. The logs for that request show only one INFO entry at the beginning. Explain what that gap means, list at least four hypotheses with what would tell them apart, and detail what instrumentation you would add so that this case is diagnosable next time.

Solution 1

Problems with the original code, seven in all:

Rewritten version:

Justification of the most important changes:

• Hashing the email solves the leak without losing investigative capability: customer support applies the same function and finds the entries.
• WARNING for the rejection by status. It is a deliberate decision: the system did the right thing, there is no fault. If it were ERROR, the alert from 06-04 would fire for perfectly normal returns, feeding the fatigue from section 11 of that lesson.
• The resultado field with a bounded set of values (completada, rechazada, error) lets you build a complete funnel with a single log-based metric.
• Specific exceptions before the generic one. Each records its causa, which lets you tell a gateway problem apart from a programming error, and they call for very different responses.
• raise on the real errors. The original code returned False whether the order was not returnable or the gateway was down, and that makes them impossible to tell apart from outside.
• trace_fields() on every entry. What makes the jump in section 15 possible.

The requested query:

For the average amount, the explorer counts but does not average well. Two options. The quick one, a log-based distribution metric over jsonPayload.importe_pedido_eur with that filter, which gives the p50 and the mean in Cloud Monitoring. The complete one, a sink to BigQuery and SQL:

And an observation that goes beyond the exercise: that breakdown by estado_pedido is product information, not infrastructure information. If it turns out that most of the rejections are orders in the en_transito status, the finding is not technical: it is that there are customers trying to return something they have not yet received, and the interface is probably not explaining that well. Good structured logging ends up answering business questions nobody had thought to ask.

Solution 2

Total volume: 132 GB/month. The load balancer log is 61 % and the Flow Logs 23 %: between them, 84 %. That is where all the headroom is.

Step 1 — Exclusions (the main lever):

Resulting ingested volume: around 70 GB/month, close to half. And without losing diagnostic capability: the health checks are already covered by the metrics from 06-04, the static assets keep a 5 % sample which is enough to see trends, and 25 % of the Flow Logs still detects connectivity patterns.

Step 2 — Buckets with differentiated retention:

The key decision, and it is counter-intuitive: the logs-pagos bucket is not configured with 7 years of retention. Keeping seven years in a log bucket is by far the most expensive option. The regulation requires keeping the data, it does not require it to be queryable in the Logs Explorer. Hence:

Step 3 — Sinks:

With the archive bucket's lifecycle: Nearline at 30 days, Coldline at 90, Archive at 365, deletion at 2,555 days (7 years). And in BigQuery, partition expiry at 365 days for Lucía's requirement.

Qualitative estimate of the saving:

The bulk of the saving comes from two places: half the ingested volume and, above all, moving the long retention from the expensive service to the cheap one, where the price difference per GB-month between a log bucket and Archive is two orders of magnitude.

And three warnings that complete the design. First: never exclude errors or audit logs; they are a minimal percentage of the volume and 100 % of the value. Second: verify that the payments sink captures everything the regulation requires before reducing the bucket's retention, because if the filter is wrong, the legal requirement is breached silently. Third: set an alert on the ingested volume with what you learned in 06-04, so you find out if somebody puts a log inside a loop before the bill arrives.

Solution 3

What the gap means. 7,890 of the 8,100 ms are not covered by any span. And that has a very specific meaning: something is happening that the instrumentation cannot see. There are only two families of explanation — uninstrumented code, or waiting time that is not an operation — and telling them apart is the whole exercise.

It is important to notice something: the two spans that do exist are fast. If the problem were in the query or in the rendering, you would see it. The time is going into the space between spans, and that is what has to be investigated.

Four hypotheses, with what tells them apart:

Which is the most likely and why. Hypothesis 1, and by a piece of reasoning worth spelling out: OpenTelemetry's automatic instrumentation covers Flask, requests and SQLAlchemy. If the code uses any other client — urllib3 directly, a payment provider's SDK, a Redis client, the Firestore library — those calls are completely invisible. And eight seconds with such a clean pattern smells of an external call timeout, very probably one configured at exactly 8 seconds.

Hypothesis 2 is the second candidate and it has a recognisable signature: it shows up under load and disappears when traffic drops. If the problem only happens at peak times, that is the one.

Instrumentation to add, in order of priority:

1. Instrument every outgoing client. The measure that solves the case:

2. An explicit span around every business operation. The general rule: if an operation can take time, wrap it in a span. It is cheap and it eliminates gaps by construction:

The acquire_db_connection span is especially valuable because it separates waiting time from working time, which is exactly what makes hypothesis 2 look like hypothesis 1.

3. A log on entering and leaving every long operation, with duracion_ms. Even with the span missing, two timestamped log entries bracket where the time went.

4. Explicit, aggressive timeouts on every external client, and a log when they fire:

This fourth measure is the most important in the long run, and it goes beyond diagnosis. Without an explicit timeout, most HTTP clients wait indefinitely or for the operating system's default, which can be very long. A service you depend on that degrades drags yours down with it, and eight seconds of waiting is exactly that. With a short timeout and graceful degradation, a downed ratings API produces a page without ratings in 3 seconds instead of a page that does not load in 8.

And the general lesson of the exercise: a gap in a trace is not a failure of the trace, it is information. It is telling you precisely that there is a part of the system you are not observing. The right reaction is not to distrust the tool, but to ask what the code is doing during that interval that nobody instrumented. In this case, almost certainly, waiting for somebody who is not answering.

AlpinaShop has observability's three pillars complete, and — more importantly — it has the links between them.

You know what a structured log entry is and why the difference between textPayload and jsonPayload is the difference between being able to investigate and not being able to. You know all its fields, and you know which is the decisive one: trace, the one that links the log with its trace.

You know where logs come from automatically and which ones have to be enabled explicitly: the load balancer's — among the most valuable there are, with their latency, their country and their CDN result — and VPC Flow Logs. And you know how to emit them properly from Flask, with the five rules: data goes in fields and not in the message, the severity must be right, never personal data, enough context to act on and no noise. With the severity criterion most often got wrong: a retry that works is WARNING; an exhausted one, ERROR.

You have mastered the Logs Explorer and its language — with the : that means "contains" and not "equals" — the three performance tips, and five query patterns that solve real incidents: the errors since a deployment, a specific customer's request located by their hash, who deleted a resource according to the audit logs, the load balancer's slow requests and the image function's errors.

You know log buckets with _Required and _Default, retention with its three horizons — diagnosis, analysis and compliance, each in its right place — and the views that let you grant access to a subset without exposing the audit logs. You know how to create sinks to BigQuery for analysing with SQL, to Cloud Storage for archiving cheaply with a lifecycle and to Pub/Sub for reacting with a function from 06-03, with the writerIdentity everybody forgets. And you know how to use exclusion filters with the percent that samples instead of discarding, with the warning that what is excluded is never recovered and that errors and audit logs are never excluded.

You have log-based metrics — counters and distributions — that instrument without touching the code and feed the alerts from 06-04, with the version label that answers "did this start with yesterday's deployment?" and the same cardinality rule as always.

You know what distributed tracing is: trace and span, context propagation with W3C's traceparent and the X-Cloud-Trace-Context the load balancer is already adding. You know how to instrument Flask with OpenTelemetry — a standard, not a proprietary tool — with automatic instrumentation of Flask, requests and SQLAlchemy, manual spans for your own detail, and the key piece: including the trace_id in every log. You know how to read a waterfall by looking at the three things that matter — which span dominates, whether there are gaps and whether there are repeated spans betraying an N+1 — and you know how to sample with ParentBased(TraceIdRatioBased(...)) so as not to end up with incomplete traces. You know Cloud Profiler and its three cases, with the warning that intuition about where the bottleneck is usually fails.

And you have the complete journey from section 15: from the alert to the metric, from the metric to the log, from the log to the trace and from the trace to the line of code, in seventeen minutes. With the lesson that sums up the whole module: no piece works on its own; the value is in the links, and the link that makes it possible fits in one line of code.

Finally, you know the cost and the three decisions that control it — what is ingested, how long it is kept and where, how much is sampled — with the mistake never to make: saving money by excluding errors, which are the smallest part of the volume and the largest part of the value.

Look now at where AlpinaShop is. The code is on GitHub and gets reviewed. The pipeline builds, tests and deploys on its own. The functions react to events. There are dashboards, alerts, checks from four continents, structured logs, traces and profiles. When something fails, it is known in five minutes and diagnosed in twenty.

And yet all the infrastructure that holds that up — the VPC, the load balancer, the cluster, the alerts you have just created, the sinks you have just configured — still exists because somebody ran the right commands. In 06-05 you saw the problem clearly, you understood what infrastructure as code is, you got to know Deployment Manager and you learned the migration procedure. What is missing is the tool.

In 06-07, the module's last lesson, comes Terraform: the state and why it never goes into Git, the plan and apply flow, AlpinaShop's real code in HCL, modules, importing everything created by hand and the automatic plan on every pull request. And with that, AlpinaShop's infrastructure finally stops living in a terminal's history.