Lucía has a question that SQL does not answer well: which products are bought together. If a customer puts a pair of crampons in the shopping cart, how likely are they also to buy an ice axe? With that matrix, AlpinaShop could suggest add-ons on the product page, group items into bundles and lay the categories out better. It is a calculation over every possible pair within each order, repeated across two years of history, and it is the kind of problem that starts out elegant in SQL and ends up as a JOIN of a table with itself that nobody wants to maintain.

It is also, as it happens, exactly the kind of problem Spark was invented for: iterative, algorithmic computation over large volumes, expressed in code rather than in queries.

Here it is worth being honest about why this lesson exists. If AlpinaShop were starting from scratch today, it would probably solve this with BigQuery and Dataflow and never touch Spark. But the real world does not start from scratch: there are tens of thousands of companies with Hadoop clusters in their data centres, with years of Spark code in production, with teams that know PySpark and do not know Beam, and with libraries — MLlib, GraphX, the whole scientific Python ecosystem — that have no direct equivalent. Cloud Dataproc is Google's answer to that reality: managed Hadoop and Spark, with the peculiarity that it turns the very idea of a cluster on its head.

In this lesson you will see what Hadoop and Spark are in essence, you will create the alpinashop-spark cluster, you will understand the pattern that makes Dataproc something other than rented Hadoop — the ephemeral cluster over storage in Cloud Storage — you will run the PySpark job that answers Lucía's question, and you will compare Dataproc, Dataproc Serverless and Dataflow with proper judgement.

1. Hadoop and Spark in one page
2. Why they still matter in 2026
3. What Dataproc adds
4. Anatomy of a Dataproc cluster
5. Creating alpinashop-spark
6. The key pattern: an ephemeral cluster over Cloud Storage
7. Submitting jobs: gcloud dataproc jobs submit
8. AlpinaShop's PySpark job: average basket and products bought together
9. Initialisation actions and image versions
10. Dataproc Serverless for Spark
11. Dataproc, Serverless and Dataflow: the decision table
12. Notebooks and Spark SQL over BigQuery
13. Migrating an on-premises Hadoop to Google Cloud
14. Cost and Spot VMs

Hadoop was born in 2006 to solve a concrete problem: processing more data than fitted on one machine, using many cheap computers that fail often. It has three pieces:

• HDFS, a distributed file system that splits files into blocks and replicates them (three times by default) across the disks of the cluster's machines.
• YARN, the resource manager that decides which process runs on which machine.
• MapReduce, the original programming model: you split the work into a map phase (transform each record) and a reduce phase (aggregate by key), and the framework handles the distribution and the failures.

MapReduce worked and it was extremely slow, for a design reason: it wrote to disk between every phase. An iterative algorithm that needs twenty passes over the same data did twenty rounds of writing and reading from disk.

Spark appeared in 2014 with the obvious correction: keeping the intermediate data in memory. For the same iterative algorithm, the improvement was one or two orders of magnitude. And it added a far more pleasant API.

Spark today is organised around:

The execution architecture, which you need to have in your head to understand what follows:

The driver runs your code, builds a graph of operations and chops it into tasks. The executors run those tasks over partitions of the data. Just as in Beam, transformations are lazy: nothing happens until you call an action (count(), collect(), write()). That laziness is what lets the optimiser reorder and fuse operations.

With BigQuery and Dataflow available, why learn this? Four concrete reasons and one consequence.

Existing code. A company migrating to the cloud with 200,000 lines of PySpark in production is not going to rewrite them. Rewriting adds no business value, introduces bugs and eats months. Dataproc lets you move those workloads without touching the code, and optimisation can come later.

People. The market has many more engineers who know Spark than engineers who know Beam. If AlpinaShop's data team hires tomorrow, the candidate is more likely to bring Spark with them. Choosing the technology your team knows how to use is a legitimate architectural decision, not a concession.

MLlib and the Python ecosystem. Algorithms such as ALS for recommendation, k-means at scale or FP-Growth for association rules are implemented, tested and distributed. Writing them in Beam would be absurd. And inside a Spark job you can use pandas, NumPy or scikit-learn over specific partitions.

Open formats and ecosystem. Hive, Presto/Trino, HBase, Kafka, Iceberg, Delta Lake, Hudi: a whole world of open tools that speaks Hadoop's language. If AlpinaShop wanted to leave Google Cloud, a data lake in Parquet over object storage processed with Spark travels to any provider unchanged.

The consequence: Dataproc is not a second-class service or a relic. It is the least traumatic migration route and the right tool when the problem is algorithmic and the team knows Spark.

Building a Hadoop cluster by hand — installing, configuring YARN, sizing HDFS, tuning executor memory, integrating authentication — is weeks of work and a permanent source of maintenance. Dataproc reduces it to one command.

The two minutes of creation time are not a marketing figure: they are what changes the mental model. If creating a cluster takes weeks, the cluster is a permanent installation that has to be looked after, shared between teams and kept switched on at all times just in case. If it takes ninety seconds, the cluster becomes disposable: it is created for one job, destroyed when it finishes, and each job can have its own with the version and the libraries it needs.

That is what we will see in section 6, and it is the most important idea in the lesson.

A Dataproc cluster has three types of node:

The secondary workers deserve attention. Since they do not take part in HDFS, they can disappear without any data being lost. That is why they can be Spot VMs (the same ones from 02-01, which Google can reclaim with 30 seconds' notice). If one disappears mid-task, YARN reassigns that task to another node and the job carries on, slower but correct.

That is the winning combination for AlpinaShop: a few standard primary workers to provide stability, and many Spot secondaries to provide cheap power.

Notice that the storage is outside the cluster. That is the next section.

And the cluster:

A review of what matters:

Adding autoscaling to the cluster requires a separate policy:

The gracefulDecommissionTimeout matters: without it, shrinking the cluster kills nodes with tasks in flight and those tasks have to be redone. With it, it waits for them to finish. And scaleDownFactor: 0.5 avoids the accordion effect of constantly going up and down.

Verification:

Here is the idea that changes everything.

In a traditional Hadoop, storage and compute live on the same machines. HDFS lives on the workers' disks. That had an excellent reason in 2006: moving data over the network was hugely expensive compared with reading it from the local disk, so the principle was "take the compute to the data".

But it has a devastating consequence: if you switch the cluster off, you lose the data. That is why Hadoop clusters are always switched on, even if they only work three hours a day. You pay for the hardware all twenty-four.

In Google Cloud, the internal network is so fast that the premise no longer holds: reading from Cloud Storage is not significantly slower than reading from a local disk. That allows the design to be inverted:

The Cloud Storage connector comes preinstalled in Dataproc and makes gs:// paths behave like HDFS paths for any Spark or Hadoop code:

The advantages of separating storage and compute are concrete:

• You pay for compute only when you compute. A cluster that lives 20 minutes a day costs 1.4 % of a permanent one.
• The data outlives the cluster. You can destroy it with complete peace of mind.
• Several clusters over the same data. Lucía's with Spark 3.5 and an external supplier's with an old version, simultaneously, without interfering with each other.
• Far higher durability. Cloud Storage replicates with eleven-nines guarantees; HDFS with three copies on three disks in the same rack does not come close.
• The data is reachable from outside Spark. BigQuery reads it as an external table, Dataflow processes it, the application downloads it.
• No upgrade ceremony. To move to a new Spark version, you create a new cluster. You migrate nothing.

The drawbacks, to be fair: somewhat higher latency on operations involving many small files, and the fact that Cloud Storage has no atomic directory rename, which affects certain write patterns. It is mitigated with columnar formats and reasonably sized files (128-512 MB), which is what you should be doing anyway.

AlpinaShop's rule: HDFS only as temporary working space inside a job. Nothing that has to outlive the cluster is written to HDFS. And --max-idle on every cluster, no exceptions. A cluster forgotten over a weekend costs more than all the rest of the month's analytics.

For scheduled jobs, you do not even keep a cluster: it is created, run and destroyed in a single step with workflow templates:

That final command is the one the orchestrator in 04-06 will invoke. Total cost: the minutes the job lasts. Zero for the rest of the month.

Dataproc accepts several job types:

Everything after -- is an argument for your program, not for gcloud. It is a very frequent confusion.

Useful options when submitting:

• --py-files: your own Python modules, packaged as .zip or .egg, distributed to every executor.
• --files: data files the job needs, reachable by name in the working directory.
• --jars: Java dependencies, such as the BigQuery connector.
• spark.sql.shuffle.partitions: the number of partitions after a shuffle. The default (200) is a poor fit for almost everybody: too many for small data, not enough for large. A reasonable rule of thumb is 2-3 times the cluster's total number of cores.

Follow-up:

The logs go automatically to Cloud Logging (06-06) and the Spark History Server interface stays reachable through the component gateway, even after destroying the cluster if a persistent history server is configured.

Now the real job. It reads the order lines from the lake, computes the average basket per month and country, and builds the product co-occurrence matrix.

F.broadcast() deserves attention: it tells Spark to replicate the small DataFrame on every executor instead of shuffling both sides over the network. It is the same optimisation as Beam's side input in 04-02, and it is the difference between a JOIN of seconds and one of minutes. It only works if the small side fits in memory (a few hundred MB at most).

How the three metrics are interpreted, because these are the ones Lucía will take to the meeting:

Lift is the one to look at. Confidence misleads with bestsellers: if 60 % of orders carry technical socks, any product will show high confidence towards them without any real relationship existing. Lift corrects for each product's popularity. Lift greater than 1 indicates a real association; lift close to 1, independence.

Three performance details that have to be internalised:

• cache() materialises the DataFrame in the executors' memory. Without it, since Spark is lazy, every later action would recompute the whole chain from the read of the lake. With two consumers, it saves half the work. And unpersist() at the end, to free the memory.
• coalesce(1) reduces to a single partition before writing. It is correct only for small results; with large data, concentrating it all in one executor would bring it down. For large volumes you use repartition(n).
• writeMethod=direct uses BigQuery's Storage Write API instead of going through temporary files in the bucket. It is faster and it avoids managing cleanup.

And the corresponding warning: this analysis uses cliente_id and order data. If identifiable personal data is incorporated at any point, the processing must comply with the GDPR and be reviewed by a compliance professional. For basket analysis you do not need to know who the customer is, only what was in the order; keeping it that way is data minimisation by design. All the data in this course is fictitious.

An initialisation action is a script that runs on every node when the cluster is created. It is used to install dependencies that do not come in the image.

Advice about initialisation actions:

• The script runs on every node, including the ones autoscaling adds. It must be idempotent and fast: a five-minute script multiplies each new node's start-up time by five.
• Use get_metadata_value attributes/dataproc-role to tell master from worker.
• set -euxo pipefail makes the script fail loudly instead of leaving the node half-configured.
• If there are many dependencies, it is better to build a custom image than to install them on every start-up.

About the image versions: each Dataproc version packages a specific set of Spark, Hadoop, Python and the operating system. The 2.2-debian12 series, for example, brings Spark 3.5 and Python 3.11.

For AlpinaShop: a pinned series in development, an exact version in the scheduled workflows. And in the workflow file, versioned in Git.

Although an ephemeral cluster is far better than a permanent one, you still have to size it: how many workers, which machine, how much memory per executor. Dataproc Serverless removes that decision: you submit the Spark job and Google takes care of everything.

There is no clusters create. There is no --max-idle because there is nothing to switch off. There is no --num-workers.

The criterion: if your job is pure Spark and you need neither Hive nor HBase nor a long-lived cluster, start with Serverless. It is less to administer and less to leave switched on by accident. Use a cluster when you need ecosystem components, long interactive sessions, or when the Spot VM discount on very large workloads outweighs the management.

For AlpinaShop, the reasoned decision is: the basket analysis goes to Serverless, because it is pure Spark, it runs monthly and nobody wants to remember to switch anything off. The alpinashop-spark cluster is kept solely as an exploration environment with Jupyter, with --max-idle=30m, and it is destroyed when it goes unused for a month.

AlpinaShop's final decision comes out like this:

• Streaming of orders and visits → Dataflow. Beam's model of windows and watermarks has no comfortable equivalent in Spark, and the Pub/Sub to BigQuery templates solve the base case with no code.
• New batch loads and transformations → Dataflow, for consistency with the above and because the team already has it set up.
• Algorithmic analysis: basket, co-occurrence, future models with MLlib → Dataproc Serverless.
• Interactive exploration over the lake → a notebook on alpinashop-spark, with --max-idle.
• Transformation expressible in SQL over data already in BigQuery → BigQuery, without moving anything.

With --optional-components=JUPYTER and --enable-component-gateway, the Dataproc console shows a link to JupyterLab protected by IAM. No opening ports and no SSH tunnels: whoever has the right role gets in, and whoever does not, does not.

The BigQuery connector for Spark lets you read and write tables directly:

The filter option is what makes the difference: it runs in BigQuery, not in Spark. Without it, the connector would bring the whole table over the network for Spark to filter, paying for the full read in BigQuery and wasting time. It is the same principle we saw with EXTERNAL_QUERY in 04-01: filter as close to the source as possible.

And the obvious question: if you can do this in Spark SQL, why not do it in BigQuery directly? Almost always you should. The connector makes sense when the SQL result feeds an MLlib algorithm, when you cross BigQuery tables with lake files that have not been loaded, or when the Spark code already exists. For a pure aggregation, BigQuery is faster and cheaper.

This is the scenario for which Dataproc is most justified. Suppose AlpinaShop absorbs a competitor with a 30-node Hadoop cluster.

What is preserved:

• The Spark, Hive and PySpark code: the vast majority works unchanged.
• The Hive queries: Dataproc includes Hive, and the metastore can be migrated.
• The data formats: Parquet, ORC and Avro are identical.
• The Oozie workflows, although it is worth replacing them with Composer (04-06).

What must be rethought, without exception:

The recommended strategy, and the only one that usually works out, is phased:

1. Copy the data to Cloud Storage with the Storage Transfer Service or hadoop distcp, touching nothing else. The on-premises cluster carries on working.
2. Stand up Dataproc Metastore and register the tables pointing at gs:// instead of hdfs://.
3. Run the existing jobs on Dataproc against the data already in Cloud Storage, comparing results with those from the old cluster. This dual-run phase is non-negotiable: it is the only way to prove that the numbers match.
4. Switch off the on-premises cluster when the comparison holds for several weeks.
5. Only then, optimise: move the Hive queries to BigQuery, the streaming jobs to Dataflow, adopt Serverless.

The classic mistake is attempting step 5 at the same time as step 3, that is, migrating and modernising simultaneously. When the numbers do not match, nobody knows whether it is because of the migration or the rewrite, and the project stalls for months.

Dataproc bills two things:

1. The underlying VMs (Compute Engine, disks and network), at the normal rate.
2. A Dataproc management fee, of the order of $0.01 per vCPU per hour, verifiable in the official documentation.

In other words, Dataproc's overhead compared with building Hadoop yourself on VMs is small: you pay very little for not administering anything.

An example with alpinashop-spark (1 master + 2 workers + 2 secondaries, all n2-standard-4, 20 vCPU in total), as an order of magnitude:

Verify current prices in the official documentation; what matters here is the factor of 100 between the first row and the third. That factor is the whole lesson.

Saving levers, in order of impact:

1. --max-idle, always. A cluster forgotten over a four-day break costs more than a year of Serverless.
2. Ephemeral clusters per workflow. It eliminates the problem at the root.
3. Spot secondary workers. Discounts of up to 80 %, with the caveat that they do not contribute HDFS and can disappear.
4. Serverless for occasional jobs.
5. Right-sized disks. With the data in Cloud Storage, HDFS is only shuffle space: 200 GB per worker is more than enough for almost everything.
6. A consistent region. Cluster and buckets in europe-west1: reading data from another region costs egress and latency.
7. Committed use discounts only if you end up with a permanent cluster, which this section suggests avoiding.
8. Columnar formats. Parquet with files of 128-512 MB reads far less and avoids the small-files problem, which is the biggest performance killer in Spark over object storage.

Creating a permanent cluster out of habit. It is the mental inheritance of on-premises Hadoop and it is the most expensive mistake. If the cluster has no --max-idle, it should not exist.

Writing important data to HDFS. It disappears when the cluster is destroyed. HDFS in Dataproc is working memory, not storage.

Not pinning the image version. Google updates the default version and a job that used to work stops working without anybody having touched the code.

Using collect() on a large DataFrame. It brings all the data back to the driver, which is a single machine. It is the number one cause of OutOfMemoryError in Spark. Use show(), take(n) or write to a file.

Forgetting cache() when a DataFrame is used several times. Spark recomputes the whole chain on every action. And the opposite mistake: caching everything that moves, until memory fills up and it starts spilling to disk.

Leaving spark.sql.shuffle.partitions at 200. With small data it generates 200 tiny tasks with more overhead than work; with large data, enormous partitions that do not fit in memory.

Lots of small files in the lake. Ten thousand 1 MB files are far slower to read than twenty of 500 MB, because every open has latency. Compact them.

Putting the data in one region and the cluster in another. Billed cross-region egress and added latency on every read.

Tip: use Serverless by default. Start there and create a cluster only when you discover you need something Serverless does not give you. It is the path with the least operational debt.

Tip: --dry-run does not exist, but the subset does. Before launching a job over two years, run it over one week. Logic errors show up just the same and cost a hundred times less.

Tip: always look at the Spark interface. The component gateway gives access to the Spark UI, where you can see the stages, the tasks and — most useful of all — the distribution of times across tasks. If one task takes a hundred times longer than the median, you have data skew, exactly as in Dataflow.

Create a cluster called alpinashop-spark-pruebas in europe-west1 that: lives in sn-datos-euw1 with no public IP, uses the sa-dataproc-analitica service account, has 1 n2-standard-2 master and 2 n2-standard-2 workers, adds 2 Spot secondary workers, pins the 2.2-debian12 image, self-destructs after 15 minutes of inactivity and in any case after 2 hours of life, and carries AlpinaShop's standard labels. Then check its status and delete it explicitly.

Write a PySpark job that reads gs://alpinashop-datalake/pedidos/cabeceras/ and gs://alpinashop-datalake/pedidos/lineas/ in Parquet, and computes per product category: the number of orders with at least one return, the return rate over the total orders in that category, and the average amount returned. The result must be written to alpinashop-datos.alpinashop_analitica.devoluciones_categoria. Apply at least two of the optimisations seen in the lesson and explain why you apply them.

For each of these five AlpinaShop assignments, choose between BigQuery, Dataflow, Dataproc with a cluster, Dataproc Serverless or bq load, and justify the choice in two or three sentences:

1. Loading a 4 GB Parquet file from the ERP into a BigQuery table every night, with no transformation whatsoever.
2. Computing the monthly sales ranking by category from tables that are already in alpinashop_analitica.
3. Processing the pedidos-nuevos events as they arrive, grouping them by the event's hour and tolerating 90 minutes of delay.
4. Training a recommendation model weekly with MLlib's ALS over two years of purchase history.
5. Running the PySpark code inherited from the absorbed competitor, 15,000 lines that use Hive and user-defined functions, while it is decided what to do with it.

Solution 1

The two lifecycle options are complementary and it is worth setting both:

• --max-idle=15m: it is destroyed if nobody uses it for 15 minutes. It covers the "I went to lunch" case.
• --max-age=2h: it is destroyed after 2 hours no matter what. It covers the "a job in an infinite loop keeps the cluster busy and --max-idle never fires" case. That is the failure that produces weekend invoices.

--no-address with --subnet=sn-datos-euw1 requires Private Google Access to be enabled on that subnet (03-01) or the nodes will not be able to download packages or talk to the APIs. If the cluster stays in CREATING and then fails, that is the first cause to check.

Solution 2

The optimisations and their justification:

1. Early column projection. Parquet is columnar; asking for four columns instead of twenty proportionally reduces the bytes read from the bucket and the executors' memory. It is the exact equivalent of not doing SELECT * in BigQuery.
2. broadcast() on both JOINs. The product catalogue is a few thousand rows and the period's headers are also small compared with the lines. Broadcasting them avoids shuffling the big table over the network, which is the dominant cost.
3. cache() on detail. Although in this final version there is only one aggregation, as soon as a second calculation is added (by country, by month) Spark would recompute the whole chain. It is the correct preparation; with unpersist() at the end so as not to hold memory.
4. Filter delegated to BigQuery (option("filter", "activo = true")). It runs over there and fewer rows come across.

A methodological note that has to be made explicit when presenting the result: an order with products from three categories counts as an order in all three, so the per-category figures do not add up to the total number of orders. That is correct for measuring a rate per category, but it has to be stated in the report or someone will subtract and the numbers will not add up for them.

Solution 3

1. Loading a 4 GB Parquet file every night with no transformation → bq load. There is no transformation, so no processing engine is needed. Batch loading into BigQuery is free, Parquet carries the schema built in and there is nothing to size. Using Dataflow or Spark here would mean paying for compute to make a copy. A single command, orchestrated in 04-06.

2. Monthly ranking over tables already in alpinashop_analitica → BigQuery. The data is already there and the operation is an aggregation with a window function, exactly what we did in 04-01 with RANK() OVER and QUALIFY. Taking the data out of BigQuery to process it elsewhere and putting it back in is the classic anti-pattern: read cost, compute cost, write cost and latency, for a worse result. If it has to be refreshed often, a materialized view.

3. pedidos-nuevos events by the event's hour with 90 minutes of tolerance → Dataflow. This is literally the use case Beam's model exists for: unbounded streaming, grouping by event time, fixed windows, watermark and allowed_lateness. Spark's Structured Streaming could do it, but with a less expressive time model and without native Pub/Sub integration. On top of that, Dataflow gives exactly-once semantics towards BigQuery with no extra work.

4. Training MLlib's ALS weekly → Dataproc Serverless. ALS is a distributed iterative algorithm implemented in MLlib, with no equivalent in Beam or in pure SQL. It is Spark from start to finish. Serverless rather than a cluster because it runs once a week: nobody wants to maintain or remember to switch off a cluster that works for an hour every seven days. (The next step, serving that model in production, is Vertex AI territory in module 5.)

5. 15,000 inherited lines of PySpark with Hive and UDFs → Dataproc with a cluster. Here the practical constraint rules: the code exists, it works and it uses Hive, which Serverless does not include. The priority is for it to carry on working with the minimum of change, so a Dataproc cluster with Dataproc Metastore, running the code as-is against the data already copied to Cloud Storage. Modernisation — moving queries to BigQuery, streaming to Dataflow — is a later and separate phase, never simultaneous with the migration: if the numbers do not match, you have to be able to tell whether it is the move or the rewrite.

You have toured the world of Hadoop and Spark with exactly the right perspective: what they are, what problem they solved — processing more data than fits on one machine, with machines that fail — why Spark displaced MapReduce by keeping the intermediate data in memory, and why in 2026 they still matter even though BigQuery and Dataflow exist: there is code already written, there are people who know how to use it, there are libraries such as MLlib with no equivalent, and there is an open ecosystem that provides real portability.

You have seen what Dataproc adds: clusters in under two minutes, configured and coherent, integrated with Cloud Storage, BigQuery, IAM and Logging. You know the anatomy — master, primary workers that hold up HDFS, secondary workers that only contribute compute and can therefore be Spot — and you have created alpinashop-spark inside sn-datos-euw1, with no public IP, with the sa-dataproc-analitica account, with the image pinned, with Jupyter reachable through the component gateway and with an autoscaling policy that only scales the secondaries and removes them gracefully.

But the important thing in this lesson is not a command, it is a conceptual inversion: the ephemeral cluster over storage in Cloud Storage. Because the internal network makes reading from gs:// comparable to reading from a local disk, the data no longer needs to live on the cluster. And if the data does not live on the cluster, the cluster can die. From that come --max-idle, --max-age, workflows with a managed cluster that is born and dies with the job, and the hundredfold difference on the invoice between a permanent cluster and an ephemeral one.

You have written the PySpark job that answers Lucía's question: the average basket per month and country with its median next to it, and the matrix of products bought together with support, confidence and lift — the metric that corrects for popularity and stops you concluding that everybody buys socks with everything — applying broadcast to avoid shuffle, cache to avoid recomputation, a cap on lines per order so that rare corporate orders do not dominate the calculation, and the sku_a < sku_b trick to count each pair only once. You know the initialisation actions, their risks and why pinning the image version is not a quirk.

You have met Dataproc Serverless, which removes even the sizing decision, and you have set AlpinaShop's criterion: the basket analysis on Serverless, the cluster only as an exploration environment with Jupyter and self-destruction. And you have the complete decision table between Dataproc, Serverless and Dataflow, with streaming and event time on Beam's side, algorithms and ecosystem on Spark's side, and SQL over already-loaded data on BigQuery's side, without moving anything. You know how an on-premises Hadoop is migrated in phases, with the golden rule of not modernising and migrating at the same time. And you know where the money is: --max-idle, ephemeral clusters, Spot, Serverless, right-sized disks and large files in a columnar format.

There is a promise that has been outstanding for two lessons. Everything you have built — the Dataflow pipeline, the Spark job, the BigQuery tables — works over data that is already somewhere. But the shop is still an island: when a customer confirms an order, the Flask application has to tell the warehouse to prepare it, billing to issue the invoice, the mail service to send the confirmation, and now analytics as well. If it does that by calling all four one after the other, the sale hangs waiting for the slowest, and if one fails, it is not clear what has happened to the other three. It is a fragile design that breaks on exactly the busiest sales day of the year.

In 04-04, Cloud Pub/Sub, we will break that coupling. We will finally create the pedidos-nuevos topic and the sub-almacen, sub-facturacion and sub-analitica subscriptions; you will understand messaging's real guarantees — at-least-once delivery, no guaranteed order — and why that forces your consumers to be idempotent; you will see dead letter topics, retries with exponential backoff, message replay with seek, filters by attribute and the direct BigQuery subscriptions that ingest without writing a line of code. And we will finally connect, properly, the alpinashop-catalogo bucket notifications that were promised back in 02-02.