The Project | About Us | Contribute | Donations | License

HOME

So far, AlpinaShop has not written a single line of modelling code. With SQL it put a heuristic recommender into production, and with AutoML it trained a cart classifier and an image one. For many companies, that is enough forever, and there is no shame in it.

But Marta wants something neither of the two routes gives. She wants a recommender that understands customers and products, not one that memorises pairs. One that knows that whoever buys crampons, an ice axe and a helmet is a different profile from whoever buys a 25-litre backpack and poles, and that can suggest something sensible to a new customer with two purchases because they resemble other customers who do have a history. And one that works with products that do not yet appear in productos_juntos because they arrived last month.

That is not classifying or predicting a number: it is learning a representation. There is no target column. There is no flat table. And there is no option without code.

This lesson is the descent to the level of the code. Not to turn you into a deep learning researcher, but so that you know when it is needed, what the minimum is that you have to understand, and how all of that runs on Google Cloud without setting up or maintaining a single machine.

When you need to go down to the code
TensorFlow and Keras: the bare essentials
Embeddings: the idea that makes the recommender possible
The two-tower architecture
AlpinaShop's model in code
tf.data: reading data without starving the GPU
TFRecord and reading from Cloud Storage
Training on Vertex AI: packaging the code
CPU, GPU or TPU: how to choose
Distributed training, conceptually
Checkpoints, Spot VMs and why they go together
Managed TensorBoard and Vertex AI Experiments
Saving and serving: SavedModel, Registry and endpoint
Optimising for inference
PyTorch, JAX and the golden rule of cost

When you need to go down to the code

Not always. Most of the time, no. The four cases where you do:

Situation	Why AutoML does not get there
Your own architecture	Two towers, siamese networks, attention over sequences: they are not in AutoML's catalogue
Custom loss function	Optimising margin instead of accuracy, or penalising errors asymmetrically
Specific preprocessing	Dynamically sampled negatives, particular data augmentation
Non-standard output	A 64-dimension vector instead of a class or a number

AlpinaShop's recommender falls into all four at once, which makes it an honest example. The output is not a class: it is a vector per customer and another per product. The loss is not "you got it right or you did not": it is "bring together the ones that go together and push apart the ones that do not". And the negative examples — products the customer did not buy — have to be generated, because they do not exist in the data.

And the reverse criterion, just as important: if your problem fits in a table with a target column, do not go down to the code. What you gain in control you pay for in weeks of work and in perpetual maintenance.

TensorFlow and Keras: the bare essentials

TensorFlow is a library for building and training numerical models. Keras is its high-level API, the one used today for almost everything. Four concepts are enough to follow the lesson.

Tensor. A typed multidimensional array. A scalar is a rank-0 tensor, a vector rank 1, a matrix rank 2. A batch of 32 vectors of 64 dimensions is a tensor of shape (32, 64).

import tensorflow as tf

t = tf.constant([[1.0, 2.0, 3.0],
                 [4.0, 5.0, 6.0]])
print(t.shape)   # (2, 3)  -> 2 rows, 3 columns
print(t.dtype)   # float32

Layer. A transformation with learnable parameters. Dense(64) multiplies the input by a weight matrix and adds a bias; Embedding(1000, 32) is a table of 1,000 vectors of 32 dimensions looked up by index.

Model. A composition of layers. Two ways of building one:

from tensorflow import keras
from tensorflow.keras import layers

# Sequential: one layer after another. Simple and limited.
seq_model = keras.Sequential([
    layers.Dense(64, activation="relu", input_shape=(20,)),
    layers.Dense(32, activation="relu"),
    layers.Dense(1,  activation="sigmoid"),
])

# Functional: a graph. Allows several inputs and several outputs.
input_a = keras.Input(shape=(20,), name="numericas")
input_b = keras.Input(shape=(1,),  name="categoria")
x = layers.Dense(64, activation="relu")(input_a)
y = layers.Flatten()(layers.Embedding(50, 8)(input_b))
output = layers.Dense(1, activation="sigmoid")(layers.Concatenate()([x, y]))
functional_model = keras.Model(inputs=[input_a, input_b], outputs=output)

The difference matters for this lesson. The sequential model only works for linear stacks. The recommender has two independent inputs — customer and product — that are processed along separate paths and combined at the end. That can only be expressed with the functional API or by subclassing keras.Model.

Compiling and training.

seq_model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="binary_crossentropy",
    metrics=["AUC"],
)

history = seq_model.fit(
    training_data,
    validation_data=validation_data,
    epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)],
)

The optimizer decides how the weights are adjusted. Adam is the sensible default choice.
The loss is what gets minimised. It is the mathematical translation of "what being wrong means", and it is where a model is genuinely customised.
An epoch is one complete pass over the data.
EarlyStopping stops when validation stops improving and restores the best weights. Without restore_best_weights=True you keep the last epoch's weights, which are worse. It is a classic mistake.

That is enough. You do not need to understand gradient descent to follow along.

Embeddings: the idea that makes the recommender possible

An embedding is a vector of numbers that represents an entity, learned in such a way that similar entities end up close together in that space.

The problem it solves is best seen through the alternative. To represent 2,400 products without embeddings, one-hot encoding uses a vector of 2,400 positions, with one 1 and 2,399 zeros. It is huge, it is sparse, and above all it says nothing: two backpacks are as far from each other as a backpack is from an ice axe, because all the vectors are equally different.

A 32-dimension embedding represents each product with 32 learned numbers. And learned with a criterion: if two products appear in similar contexts, their vectors end up resembling each other.

Aspect	One-hot	Embedding
Dimensions for 2,400 products	2,400	32–128
Content	Identity only	Learned similarity
New products	Trivial but useless	Requires retraining or using attributes
Memory cost	High and sparse	Low and dense

What is interesting is that the similarity emerges on its own, with nobody declaring that an ice axe resembles a set of crampons: it comes out of the purchase data. And once you have vectors, comparing is trivial. The dot product or the cosine similarity between two vectors is a number that measures affinity: if the ice axe vector is [0.21 −0.44 0.87 0.12] and the crampons one is [0.19 −0.40 0.91 0.09], their cosine is almost 1 — they point in almost the same direction — whereas against a pair of trekking sandals it comes out negative.

The embedding size is a hyperparameter with a clear trade-off: too few dimensions do not capture nuances, too many memorise and overfit. For 2,400 products, between 32 and 64 is a reasonable starting range.

The two-tower architecture

The intuition, without maths.

Imagine two functions. The first takes everything you know about a customer — their history, their categories, their average basket — and returns a vector of 64 numbers. The second takes everything you know about a product — its category, its price, its brand, its identifier — and returns another vector of 64 numbers in the same space.

If training has gone well, the dot product between a customer's vector and a product's is high when that customer would buy that product, and low when they would not.

flowchart TD
    A[Customer data<br/>history, categories, basket] --> B[Customer tower<br/>dense layers]
    C[Product data<br/>id, category, price, brand] --> D[Product tower<br/>dense layers]
    B --> E[Customer vector<br/>64 dim]
    D --> F[Product vector<br/>64 dim]
    E --> G[Dot product]
    F --> G
    G --> H[Affinity score]

Why two towers and not a single network that receives everything together. For a purely operational reason, and it is the key to the whole design: the two towers can be run separately.

The 2,400 product vectors are computed once and stored. When a customer arrives, only their vector is computed — one pass through the customer tower — and the nearest products are looked up among the precomputed ones. That is a nearest-neighbour search, which with the right indexes is a matter of milliseconds even with millions of items.

The alternative — a single network receiving the customer-product pair — would force you to evaluate the model 2,400 times per customer, once per product. With two towers, once.

The negative examples. AlpinaShop's data only contains purchases: positive customer-product pairs. For the model to learn to distinguish, it also needs examples of what was not bought, and those have to be generated. The standard technique is in-batch negative sampling: within a batch of 512 real pairs, each customer is paired with the products of the other 511 pairs and those are treated as negatives. It is efficient because nothing has to be searched for, and it works surprisingly well.

It has a known bias, and it is worth being aware of it: popular products appear more often in batches and are therefore penalised more as negatives. There are corrections for that; for a first AlpinaShop version, the complexity is not worth it.

AlpinaShop's model in code

The complete model, with comments. It is the heart of the lesson, so read it slowly.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

DIM = 64          # dimension of the shared space
N_CUSTOMERS = 45000
N_PRODUCTS = 2400
N_CATEGORIES = 18


def customer_tower():
    """Turns a customer's data into a vector of DIM dimensions."""
    customer_id  = keras.Input(shape=(), dtype=tf.int32,   name="cliente_idx")
    customer_num = keras.Input(shape=(4,), dtype=tf.float32, name="cliente_num")

    # Embedding: each customer has their own learned vector
    emb = layers.Embedding(N_CUSTOMERS, DIM, name="emb_cliente")(customer_id)

    # Numeric signals already normalised: pedidos_90d, ticket_medio,
    # dias_desde_ultimo, antiguedad_meses
    x = layers.Concatenate()([emb, customer_num])
    x = layers.Dense(128, activation="relu")(x)
    x = layers.Dropout(0.2)(x)
    x = layers.Dense(DIM)(x)

    # Normalise to norm 1: the dot product becomes cosine similarity
    output = layers.Lambda(lambda v: tf.math.l2_normalize(v, axis=1))(x)
    return keras.Model([customer_id, customer_num], output, name="torre_cliente")


def product_tower():
    """Turns a product's data into a vector in the same space."""
    product_id  = keras.Input(shape=(), dtype=tf.int32,   name="producto_idx")
    category    = keras.Input(shape=(), dtype=tf.int32,   name="categoria_idx")
    product_num = keras.Input(shape=(3,), dtype=tf.float32, name="producto_num")

    emb_p = layers.Embedding(N_PRODUCTS,   DIM, name="emb_producto")(product_id)
    emb_c = layers.Embedding(N_CATEGORIES, 16,  name="emb_categoria")(category)

    # precio_norm, ventas_30d_norm, puntuacion_media
    x = layers.Concatenate()([emb_p, emb_c, product_num])
    x = layers.Dense(128, activation="relu")(x)
    x = layers.Dropout(0.2)(x)
    x = layers.Dense(DIM)(x)
    output = layers.Lambda(lambda v: tf.math.l2_normalize(v, axis=1))(x)
    return keras.Model([product_id, category, product_num], output, name="torre_producto")

Three design decisions that explain the rest:

Each tower mixes an embedding with attribute information. A pure embedding memorises the specific entity; the attributes (price, category) make it possible to say something reasonable about a new product that has barely sold. It is the practical mitigation of the cold start.
Dropout(0.2) randomly switches off 20 % of the neurons at each training step. It is regularisation: it stops the model from depending too much on one particular signal and reduces overfitting.
The final L2 normalisation makes all the vectors have length 1. That way the dot product is exactly cosine similarity, bounded between −1 and 1, which stabilises training and makes the scores comparable with each other.

The complete model, with the loss:

class Recommender(keras.Model):
    def __init__(self, temperature=0.05):
        super().__init__()
        self.customer = customer_tower()
        self.product  = product_tower()
        self.temperature = temperature
        self.metric = keras.metrics.Mean(name="perdida")

    def train_step(self, batch):
        with tf.GradientTape() as tape:
            v_cus = self.customer([batch["cliente_idx"], batch["cliente_num"]])
            v_prod = self.product([batch["producto_idx"],
                                   batch["categoria_idx"],
                                   batch["producto_num"]])

            # Similarity matrix: each customer against each product IN THE BATCH
            similarities = tf.matmul(v_cus, v_prod, transpose_b=True) / self.temperature

            # The diagonal holds the real pairs (positives); the rest, negatives
            labels = tf.range(tf.shape(similarities)[0])
            loss = tf.reduce_mean(
                tf.nn.sparse_softmax_cross_entropy_with_logits(
                    labels=labels, logits=similarities))

        gradients = tape.gradient(loss, self.trainable_variables)
        self.optimizer.apply_gradients(zip(gradients, self.trainable_variables))
        self.metric.update_state(loss)
        return {"perdida": self.metric.result()}

How this loss works, in words. tf.matmul(v_cus, v_prod, transpose_b=True) produces a square matrix the size of the batch: cell (i, j) is customer i's affinity with product j. The real pairs are on the diagonal, because the batch is made up of pairs that did happen. The cross-entropy with labels 0, 1, 2, ... asks the model to make, for each customer, the correct product the highest-scoring one among all those in the batch. That is where the negative sampling lives, with no extra code.

The temperature (0.05) divides the similarities before the softmax. Low values make the distribution more "peaked" and training more demanding. It is a sensitive hyperparameter that is worth tuning.

This is exactly what AutoML cannot do: a custom loss over a custom architecture.

`tf.data`: reading data without starving the GPU

An expensive and frequent mistake: renting a GPU and discovering it is at 15 % utilisation because data reading cannot keep up. You pay for the whole GPU and use a fraction of it.

tf.data builds data pipelines that overlap with computation.

def build_dataset(file_pattern, batch_size=512, training=True):
    files = tf.data.Dataset.list_files(file_pattern, shuffle=training)

    ds = files.interleave(
        lambda f: tf.data.TFRecordDataset(f, compression_type="GZIP"),
        cycle_length=8,                        # 8 files in parallel
        num_parallel_calls=tf.data.AUTOTUNE)

    if training:
        ds = ds.shuffle(buffer_size=50000)     # shuffles within a window

    ds = ds.map(parse_example, num_parallel_calls=tf.data.AUTOTUNE)
    ds = ds.batch(batch_size, drop_remainder=True)
    ds = ds.prefetch(tf.data.AUTOTUNE)         # prepares the next batch
    return ds

Each piece, and why it is there:

Operation	What it does	Why it matters
`interleave`	Reads several files at once	A single file caps the throughput from Cloud Storage
`shuffle`	Shuffles within a buffer	Without this, batches would hold consecutive, correlated orders
`map` + `AUTOTUNE`	Parses in parallel	Adjusts the threads on its own, depending on the machine
`batch`	Groups into batches	The GPU is efficient with batches, not with loose rows
`prefetch`	Prepares the next one while training	The operation with the greatest impact: it overlaps CPU and GPU
`drop_remainder=True`	Discards the last incomplete batch	Necessary here: the loss assumes square batches

About shuffle: the buffer must be large. With 5,000 over a file ordered by date, the batches will still contain almost consecutive orders and the model will see the data in an artificial order. A buffer of 50,000 shuffled over already mixed files is reasonable.

About the batch size: larger means better GPU utilisation and, in this particular model, more negatives per example, which usually improves quality. The limit is GPU memory. Starting at 512 and raising it while it still fits is a sensible strategy.

TFRecord and reading from Cloud Storage

AlpinaShop's data is in BigQuery. To train you have to export it to a format TensorFlow reads quickly, and that format is TFRecord: a binary container of serialised records.

Why not CSV. CSV is parsed as text, line by line, and with millions of rows that is the bottleneck. TFRecord is binary, it is read sequentially and it compresses well.

bq extract \
  --destination_format=CSV \
  --compression=GZIP \
  'alpinashop-datos:alpinashop_analitica.reco_entrenamiento' \
  'gs://alpinashop-datalake/reco/entrenamiento/parte-*.csv.gz'

BigQuery does not export TFRecord directly, so the conversion is done with a Dataflow job (04-02) or, for moderate volumes, with a script. What matters is the partitioning: many medium-sized files, not one giant one.

Strategy	Result
1 file of 20 GB	A single sequential read, `interleave` useless, impossible to distribute
20,000 files of 1 MB	Opening overhead, latency dominates
200 files of 100 MB	The recommended balance

The usual rule: files of between 100 and 200 MB, and at least as many files as workers you are going to use, multiplied by a few times over.

Parsing each record:

SCHEMA = {
    "cliente_idx":   tf.io.FixedLenFeature([],  tf.int64),
    "producto_idx":  tf.io.FixedLenFeature([],  tf.int64),
    "categoria_idx": tf.io.FixedLenFeature([],  tf.int64),
    "cliente_num":   tf.io.FixedLenFeature([4], tf.float32),
    "producto_num":  tf.io.FixedLenFeature([3], tf.float32),
}

def parse_example(record):
    ex = tf.io.parse_single_example(record, SCHEMA)
    return {
        "cliente_idx":   tf.cast(ex["cliente_idx"],   tf.int32),
        "producto_idx":  tf.cast(ex["producto_idx"],  tf.int32),
        "categoria_idx": tf.cast(ex["categoria_idx"], tf.int32),
        "cliente_num":   ex["cliente_num"],
        "producto_num":  ex["producto_num"],
    }

An operational detail that causes a lot of grief: the data bucket and the training machine must be in the same region, europe-west1. Reading from another region adds latency and data egress cost, and in a long training run that gets multiplied by millions of reads.

Training on Vertex AI: packaging the code

The training code has to get to the machine somehow. Two routes, already seen in 05-01, now in detail.

Route A: training application in a pre-built container. You package your script in a .tar.gz and Vertex AI runs it on an official TensorFlow image. Quick to set up, little control over the dependencies.

Route B: your own container. You build the image and push it to Artifact Registry. AlpinaShop already has the registry set up, so it is the coherent option.

FROM europe-docker.pkg.dev/vertex-ai/training/tf-gpu.2-15.py310:latest

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY entrenador/ ./entrenador/

ENTRYPOINT ["python", "-m", "entrenador.principal"]

Important points about the Dockerfile. The official base image already brings TensorFlow with GPU support, CUDA and the drivers correctly matched: assembling that by hand is an endless source of version problems. And requirements.txt must carry pinned versions (pandas==2.2.1, not pandas), because otherwise the image you build in three months' time will not be the same one.

# Build and publish
gcloud builds submit \
  --tag=europe-west1-docker.pkg.dev/alpinashop-prod/alpinashop/entrena-reco:1.0.3 \
  --project=alpinashop-cicd

# Launch the training run
gcloud ai custom-jobs create \
  --project=alpinashop-datos --region=europe-west1 \
  --display-name=reco-two-tower-v3 \
  --worker-pool-spec="machine-type=n1-standard-8,accelerator-type=NVIDIA_TESLA_T4,accelerator-count=1,replica-count=1,container-image-uri=europe-west1-docker.pkg.dev/alpinashop-prod/alpinashop/entrena-reco:1.0.3" \
  --args="--datos=gs://alpinashop-datalake/reco/tfrecord/,--salida=gs://alpinashop-datalake/modelos/reco/v3,--epocas=25,--lote=512,--dim=64"

Cloud Build is used to build the image. Full integration with CI/CD, the triggers and the deployment pipeline are the subject of 06-01; here the command is enough.

The script must read an environment variable that Vertex AI injects and that many people ignore: AIP_MODEL_DIR, the Cloud Storage path where the platform expects to find the model. If you save there instead of in a path of your own, the later registration is direct and resumption works by itself.

import os
output_dir = os.environ.get("AIP_MODEL_DIR", args.salida)

Its siblings AIP_TENSORBOARD_LOG_DIR and AIP_CHECKPOINT_DIR play the same role for logs and checkpoints.

CPU, GPU or TPU: how to choose

Accelerator	When it suits	Example at AlpinaShop	Relative cost
CPU	Small models, shallow, tabular	Recommender prototype with one month's data	1×
GPU T4	Medium networks, embeddings, light vision	The complete recommender	3–5×
GPU L4 / A100	Large networks, heavy vision, tuning large models	Not applicable today	10–40×
TPU	Very large models with massive matrix operations	Not applicable today	Variable

The multipliers are indicative orders of magnitude and they change over time and by region: always check the official pricing calculator.

The practical criterion, in three rules:

Always prototype on CPU with a small sample. If the code has a bug, it is far cheaper to find out on a machine costing €0.20/hour.
Move up to GPU when epoch time is the bottleneck, and only if you have verified that the GPU is being used. A GPU at 15 % utilisation is money thrown away and usually means the problem is in tf.data, not in the accelerator.
TPU only with models that justify it. It requires adapting the code, it has restrictions on shapes and operations, and its advantage shows up on models with an enormous amount of matrix computation. For a recommender with 2,400 products and 45,000 customers, a T4 is more than enough.

To check utilisation you enable the TensorBoard profiler for a handful of batches (profile_batch=(20, 40) in the callback from section 12). It shows the time spent on data input versus computation: if the former dominates, the GPU is waiting and the solution is prefetch, interleave and better-partitioned files, not a bigger machine.

Distributed training, conceptually

When a model does not fit or takes too long on one machine, you spread it out. Two strategies:

Data parallelism (the usual one). Each machine has a complete copy of the model and processes a different chunk of the batch. The gradients are averaged across all of them and the weights are synchronised. It scales well as long as communication cost does not dominate.

Model parallelism. The model is split between machines because it does not fit in one. Necessary for enormous language models; for AlpinaShop, completely irrelevant.

In TensorFlow it is expressed with a strategy, and the rest of the code barely changes:

strategy = tf.distribute.MultiWorkerMirroredStrategy()

with strategy.scope():
    model = Recommender(temperature=0.05)
    model.compile(optimizer=keras.optimizers.Adam(1e-3))

model.fit(build_dataset(pattern, batch_size=512), epochs=25)

Everything created inside strategy.scope() is replicated automatically. On Vertex AI, it is enough to declare more replicas in the worker-pool-spec; the platform configures the communication between nodes.

The realistic warning: distributing has overhead. With four machines you do not train four times faster, and with small models it can even be slower than with a single one, because communicating gradients costs more than the computation. Only distribute when you have measured that one machine is not enough. For AlpinaShop's recommender, a T4 trains in under an hour: distributing would be complexity with no benefit.

Checkpoints, Spot VMs and why they go together

A checkpoint is a snapshot of the training state — weights and optimizer state — saved periodically.

ckpt_path = os.path.join(output_dir, "checkpoints", "ckpt-{epoch:03d}")

callbacks = [
    keras.callbacks.ModelCheckpoint(
        filepath=ckpt_path,
        save_weights_only=True,
        save_freq="epoch",
    ),
    keras.callbacks.BackupAndRestore(
        backup_dir=os.path.join(output_dir, "backup")
    ),
]

BackupAndRestore is the one doing the interesting work: if the process dies and restarts, it picks up at the epoch it was on instead of starting from scratch. There is no resumption logic to write.

And here comes the connection with money. Spot VMs cost a fraction of normal ones — the typical discount is very large, though variable — in exchange for Google being able to reclaim them at minimal notice. Without checkpoints, an interruption an hour and a half into training means losing everything. With per-epoch checkpoints, you lose a few minutes at most.

gcloud ai custom-jobs create \
  --project=alpinashop-datos --region=europe-west1 \
  --display-name=reco-two-tower-spot \
  --worker-pool-spec="machine-type=n1-standard-8,accelerator-type=NVIDIA_TESLA_T4,accelerator-count=1,replica-count=1,container-image-uri=...:1.0.3" \
  --config=config-spot.yaml

# config-spot.yaml
scheduling:
  strategy: SPOT
  restartJobOnWorkerRestart: true

Strategy	Relative cost	Risk	When to use it
Standard	100 %	None	Training with a strict deadline
Spot + checkpoints	A fraction of the above	Interruptions absorbed	Experimentation and periodic retraining
Spot without checkpoints	Cheap	High: you lose everything	Never

For AlpinaShop, whose retraining runs happen overnight and without urgency, Spot with checkpoints is the obvious choice.

Managed TensorBoard and Vertex AI Experiments

Training blind is the fastest way to waste time. Vertex AI TensorBoard is the managed version of TensorBoard: the logs go to Cloud Storage and are visualised without setting up a server.

callbacks.append(
    keras.callbacks.TensorBoard(
        log_dir=os.environ["AIP_TENSORBOARD_LOG_DIR"],
        update_freq="epoch",
        profile_batch=(20, 40),
    )
)

What to look at, in order of usefulness: the training and validation loss on the same chart — if the first goes down and the second goes up, there is overfitting and you have to stop; if the loss does not go down at all, the learning rate is usually the culprit (too high and it oscillates, too low and it does not move); and the profiler, to confirm the GPU is working.

Vertex AI Experiments adds the missing layer: comparing runs with each other and recording which parameters each one used.

from google.cloud import aiplatform

aiplatform.init(project="alpinashop-datos", location="europe-west1",
                experiment="recomendador-alpinashop")

with aiplatform.start_run(run_name="two-tower-dim64-temp005") as run:
    run.log_params({
        "dim": 64, "temperatura": 0.05, "lote": 512,
        "learning_rate": 1e-3, "epocas": 25, "dropout": 0.2,
    })
    history = model.fit(...)
    run.log_metrics({
        "perdida_final":       float(history.history["perdida"][-1]),
        "recall_at_10":        0.243,
        "recall_at_10_baseline": 0.198,   # the SQL from 05-01
    })

Notice the last metric: each run also records the baseline's result. That way, the comparison with the heuristic recommender from decision DA-003 is not an argument from memory, it is a column in a table.

A tip that saves weeks: the run_name should describe the configuration. two-tower-dim64-temp005 still means something three months later; prueba7 does not.

Saving and serving: SavedModel, Registry and endpoint

TensorFlow's serialisation format is SavedModel: a directory with the graph, the weights and the input and output signatures.

For the recommender, what gets served is not the complete model: it is the customer tower. The product vectors are computed once and stored.

# 1) Product vectors: computed once, stored in BigQuery
vectors = model.product.predict(catalogue_dataset)   # (2400, 64)

# 2) Only the customer tower is saved as a service
model.customer.save(os.path.join(output_dir, "torre_cliente"))

And the similarity search is resolved in BigQuery, with no extra infrastructure:

CREATE OR REPLACE TABLE `alpinashop-datos.alpinashop_analitica.reco_two_tower` AS
SELECT
  c.cliente_hash,
  p.sku,
  ROUND(( SELECT SUM(cv * pv)
          FROM UNNEST(c.vector) cv WITH OFFSET i
          JOIN UNNEST(p.vector) pv WITH OFFSET j ON i = j ), 4) AS afinidad
FROM `alpinashop-datos.alpinashop_analitica.vectores_cliente`  c
CROSS JOIN `alpinashop-datos.alpinashop_analitica.vectores_producto` p
QUALIFY ROW_NUMBER() OVER (PARTITION BY c.cliente_hash ORDER BY afinidad DESC) <= 20;

Registration and deployment, if you wanted to serve it online:

gcloud ai models upload \
  --project=alpinashop-datos --region=europe-west1 \
  --display-name=reco-torre-cliente \
  --artifact-uri=gs://alpinashop-datalake/modelos/reco/v3/torre_cliente \
  --container-image-uri=europe-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest \
  --version-aliases=candidato

The pre-built TensorFlow Serving container knows how to read a SavedModel and expose it over HTTP and gRPC without you writing any server code.

The alternative: serving on Cloud Run. You package the model in an image with a small server and deploy it as a service. Advantages: it scales to zero — you do not pay if nobody calls — it is the same technology the catalogue is already going to use because of decision DA-001, and the team will know it. Drawbacks: you take care of the server, the versioning and the model monitoring yourself. The full comparison is in 07-02; here it is enough to know it exists and that for intermittent loads it usually wins.

And the decision AlpinaShop already took in 05-01 still stands: batch every night, precomputed table. The customer tower would only be needed online if recommendations had to depend on what the customer is doing in this session, and today that is not the case.

Optimising for inference

A trained model is not optimised to answer quickly. Three techniques, from least to most effort:

Quantisation. Converting the weights from 32-bit floating point to 16 bits or to 8-bit integers: the model takes between half and a quarter of the space and answers faster, in exchange for a loss of precision that is usually small and that has to be measured, not assumed.

Pruning. Removing weights close to zero. It reduces size, but the speed benefit depends on the hardware. Graph optimisation. Fusing operations and removing unnecessary nodes; TensorFlow Serving does part of this on its own.

Technique	Size reduction	Latency gain	Quality risk
16-bit quantisation	~50 %	Moderate	Very low
8-bit quantisation	~75 %	High	Medium: it has to be measured
Pruning / graph	Variable	Low without specific hardware	Low or medium

About latency: the metric that matters is the 95th percentile (p95), not the average. If 95 out of every 100 requests answer in 40 ms and 5 take 800 ms, the average says 78 ms and sounds fine, but those 5 are customers with a frozen page. SLOs are defined over percentiles, and that is covered in depth in 07-06.

And the observation that closes the section: for AlpinaShop's recommender, with overnight batch prediction, none of this is needed. Optimising the inference of a model that runs once a day over 45,000 customers is optimising what does not hurt.

PyTorch, JAX and the golden rule of cost

Vertex AI is framework-agnostic. Custom training runs containers, and anything at all fits inside a container.

Framework	Pre-built containers	Comment
TensorFlow / Keras	Training and serving	Maximum integration with TF Serving and TFX
PyTorch	Training and serving (TorchServe)	Very widespread; first-class support
JAX	Training	Strong on TPU and in research
scikit-learn, XGBoost	Training and serving	For tabular, often the best real option

The choice of framework is made on what the team knows and what code can be reused, not on platform preferences. And an honest observation that closes the module's circle: for tabular problems, XGBoost usually beats a neural network with far less effort. Deep networks shine with unstructured data — images, text, sequences — and with representation problems like this recommender.

The golden rule of cost, without embellishment: do not leave GPUs switched on. A custom training job switches itself off when it finishes, and that is why it is safe. The dangers are two others:

A Workbench instance with a GPU created for a test and forgotten. It costs more than ten times one without a GPU.
An endpoint with a GPU deployed for a demo. It bills by the hour, with no traffic, indefinitely.

gcloud workbench instances list --project=alpinashop-datos \
  --format="table(name, state, gceSetup.machineType, gceSetup.acceleratorConfigs)"
gcloud ai endpoints list --region=europe-west1 --project=alpinashop-datos

And the preventive measures: idle-timeout-seconds on every notebook, a budget alert over the centro-coste:analitica label, and a monthly review in the calendar. Cost management in depth arrives in 07-05.

Common Mistakes and Tips

A GPU at 15 % utilisation. The problem is almost never the GPU: it is the data reading. Check prefetch, interleave and the TFRecord partitioning before asking for a bigger machine.

EarlyStopping without restore_best_weights=True. You keep the last epoch's weights, which by definition are worse than the best one's.

A shuffle buffer that is too small. If the data is ordered by date, a short buffer leaves the batches correlated and the model learns the ordering.

Data in one region and training in another. Latency, egress cost and an unnecessarily slow training run. Everything in europe-west1.

Distributing without having measured. With small models, four machines can be slower than one because of gradient communication.

Using latest in the container image. It breaks reproducibility, exactly as in 05-01.

Forgetting the negative examples. A model trained only with positives does not learn to discriminate: it learns to say "yes" to everything.

Serving the complete model when one tower was enough. It multiplies the inference cost by the number of products.

Tip: prototype with 1 % of the data on CPU. If the code works with 50,000 rows, it will work with 5 million. Discovering a tensor shape bug on a paid GPU is expensive and avoidable.

Tip: always record the baseline in the same experiment. Having each run carry the number from the 05-01 SQL alongside it turns "I think it has improved" into a fact.

Tip: fix every seed (tf.random.set_seed, numpy, python). An irreproducible training run is impossible to debug.

Exercises

Exercise 1

Dani launches the recommender's training on an n1-standard-8 machine with a T4 GPU. Each epoch takes 47 minutes. Looking at the TensorBoard profiler, he sees the GPU is busy 18 % of the time. Diagnose the problem, propose four measures in order of impact and estimate what improvement can be expected.

Exercise 2

Marta asks whether the two-tower recommender should be served on a Vertex AI endpoint with a GPU, on an endpoint with CPU, on Cloud Run or in batch. Analyse the four options for AlpinaShop's case and justify a recommendation. Include what would change the decision.

Exercise 3

After training, the model's recall@10 is 0.243 against 0.198 for the SQL baseline. Is it deployed? Argue your case and define what test you would run before deciding.

Solutions

Solution 1

Diagnosis: a bottleneck in the data pipeline. A GPU at 18 % means it spends 82 % of the time waiting for data. Training is not compute-bound, it is input-bound. Paying for a GPU so it can wait is the most common waste in this lesson.

Four measures, by expected impact:

1. Add prefetch(tf.data.AUTOTUNE) at the end of the pipeline. It is the measure with the greatest impact and the least effort. Without it, the cycle is strictly sequential: the CPU prepares a batch, the GPU processes it, the CPU prepares the next one. With prefetch, the CPU prepares batch n+1 while the GPU processes batch n. With this alone, utilisation can rise to 40-50 %.

2. Review the TFRecord partitioning. If the data is in a single 20 GB file, interleave can do nothing: there is one single sequential read. Repartitioning into around 200 files of ~100 MB and reading with cycle_length=8 multiplies the throughput from Cloud Storage. High impact, medium effort (a Dataflow job).

3. Check the bucket's region. If the bucket is multi-region or outside europe-west1, every read adds latency. It must be regional and match the machine.

4. Move the heavy preprocessing out of the map. If the map performs costly per-example transformations — computations, complex normalisations — that consumes CPU on every epoch. Precomputing them when generating the TFRecords runs them once instead of twenty-five times.

And a complementary measure: n1-standard-8 is 8 vCPUs to feed a T4. If after the four measures the CPU is still saturated, moving up to n1-standard-16 is reasonable — and it pays off, because the cost of the vCPUs is small compared with having the GPU idle.

Expected improvement: with measures 1 and 2, a GPU utilisation of 70-85 % is achievable, which would take the epoch from 47 minutes down to a range of 10-15. The check is to look at the profiler again, not to assume it.

Verification: before touching anything, measure the pipeline's throughput without the model — iterate 200 batches of the Dataset with a timer and divide examples by seconds. If reading alone is already slow, the model has nothing to do with it and the diagnosis is confirmed.

Solution 2

The four options:

Option	Cost	Latency	Complexity	Assessment
Endpoint with GPU	Very high (24×7)	Very low	Low	Ruled out
Endpoint with CPU	High (24×7)	Low	Low	Ruled out today
Cloud Run	Low, scales to zero	Low with cold start	Medium	Held in reserve
Batch prediction	Very low	High (irrelevant)	Low	Recommended

Ruling out the GPU is immediate. The customer tower is a tiny network: three dense layers over an embedding. On CPU it answers in a few milliseconds. A GPU for that is like using a lorry to deliver a letter.

Ruling out the 24×7 endpoint for the same reason as in 05-01: an endpoint bills per node-hour even if it receives no traffic. With recommendations that do not change from one minute to the next, it is paying for availability nobody uses.

Cloud Run is the reasonable middle option and deserves serious consideration: it scales to zero, so with no traffic it costs nothing; it is the same technology as the catalogue under DA-001, so the team is going to know it; and the cold start is mitigated with a minimum instance. Its drawback is that you have to manage the server, the versioning and the model monitoring by hand, without the integration with Model Registry or with Model Monitoring.

Recommendation: overnight batch prediction, consistent with decision DA-002.

The reasoning in three points. First, the recommendations do not depend on the session in progress: customer vectors change when their history changes, that is, when they place an order, not when they move the mouse. Second, 45,000 customers × 20 recommendations is 900,000 rows, a trivial table that Firestore or Memorystore (02-06) serve in microseconds and that furthermore does not depend on any model being alive. Third, availability improves: if the endpoint went down, the site would be left with no recommendations; with a precomputed table, there is nothing that can go down.

What would change the decision. Three specific scenarios:

Recommending within the session. If you wanted to react to what the customer is looking at right now — "they have spent five minutes looking at crampons" — the customer vector would have to be computed hot and online serving would be needed. That would be the moment for Cloud Run, not for a dedicated endpoint.
Anonymous customers. 60 % of the traffic is not identified and has no precomputed vector. For them, today the per-product heuristic recommender is used, which needs no customer. If you wanted to personalise for anonymous visitors based on session behaviour, we would be back to the previous case.
A much larger catalogue. With 2,400 products, precomputing is trivial. With 500,000, the customer × product table would stop being comfortable and Vector Search (05-06) would come into play for the nearest-neighbour search.

Solution 3

Not with that data. Not yet.

What the number says. An improvement from 0.198 to 0.243 in recall@10 is a 22.7 % relative gain, which is not negligible. But there are three reasons not to jump straight to production:

1. It is an offline metric. recall@10 measures whether the model would have got right what customers bought in the past, when the site recommended nothing. A recommender changes the very behaviour it is trying to predict: if it suggests a product, the probability of it being bought goes up by the fact of suggesting it. Offline metrics systematically under- or overestimate the real effect, and you do not know in which direction until you test.

2. It does not measure what matters to the business. What decides is the order value, not the number of hits. A recommender that gets a lot right suggesting €4 accessories can generate less margin than one that gets less right with €90 products. You also have to look at the business metrics: average order value, items per order, click rate on the recommendation, and the return rate (recommending badly increases returns).

3. There is a confidence interval nobody has calculated. How many customers are there in the test set? If there are 2,000, the difference between 0.198 and 0.243 could be within the noise. Without error bars, two numbers are not a comparison.

The test to run: an A/B test.

Element	Definition
Group A (control)	50 % of traffic, SQL heuristic recommender (DA-003)
Group B (treatment)	50 % of traffic, two-tower model
Primary metric	Average order value
Secondary metrics	Click rate on the recommendation, items per order, conversion
Guardrail metric	Return rate (it must not get worse)
Duration	At least 2-3 complete weeks
Assignment	Per customer, stable, not per session

Four details that make the test valid:

A duration of complete weeks. Purchase behaviour varies a great deal between weekends and weekdays. A three-day test measures the day, not the model.
Stable assignment per customer. If a customer sees one recommender on Monday and another on Tuesday, the experience is inconsistent and the measurement gets contaminated.
A guardrail metric. If the return rate rises significantly, the test is stopped even if sales go up: recommending badly generates logistics cost and dissatisfaction.
Sample size calculated in advance. You have to know before starting how many orders are needed to detect the expected improvement. If with AlpinaShop's traffic it would take three months to detect 2 %, better to know that before setting up the experiment.

And the decision criterion, written down before looking at the results, which is the part most often breached: the model is deployed if the average order value improves in a statistically significant way and the return rate does not get worse by more than one point. Setting it after seeing the data is the most elegant way of fooling yourself.

A scenario that has to be accepted in advance: it is perfectly possible that the test shows no difference. If that happens, the right decision is to stick with the SQL, because it is simpler, cheaper, more explainable and requires no retraining. Being willing to throw away weeks of work when the data does not back it up is what distinguishes a mature team.

Conclusion

You have gone down to the code, and you know when it is worth it: your own architecture, a custom loss, specific preprocessing or a non-standard output. AlpinaShop's recommender falls into all four cases, which is why it justifies the effort. If your problem fits in a table with a target column, the answer is still AutoML or BigQuery ML.

You have the minimum of TensorFlow and Keras needed to follow along: tensors, layers, the difference between the sequential and the functional model — which matters here because there are two independent inputs — and compilation with an optimizer, a loss and metrics, with EarlyStopping restoring the best weights.

You understand what an embedding is and why it changes the rules: 32 learned numbers instead of 2,400 empty positions, with the similarity emerging on its own from the purchase data. And you have built the two-tower architecture, understanding that the reason for splitting it in two is not theoretical but operational: the product vectors are computed once, the customer one is computed on the spot, and the search is a nearest-neighbour one and not 2,400 evaluations of the model. You have seen in-batch negative sampling, which turns a similarity matrix into free negative examples.

You know how to read data without starving the GPU with tf.data — interleave, shuffle with a large buffer, parallel map, batch and above all prefetch — and why TFRecord partitioned into files of 100 to 200 MB in the same region as the machine is the difference between a GPU at 18 % and one at 80 %.

You have packaged the training as your own container with a pinned version, launched with gcloud ai custom-jobs create, and you have the criterion for CPU, GPU or TPU: always prototype on CPU, move up to GPU only when the epoch is the bottleneck and the GPU is genuinely being used, and forget about TPUs until a model justifies them. You know what data parallelism is and why distributing without measuring can turn out slower.

You have brought together checkpoints and Spot VMs in the combination that makes experimentation affordable: BackupAndRestore picks up where it left off, and a Spot interruption stops being a disaster. You follow the training with managed TensorBoard and compare configurations with Vertex AI Experiments, always recording the baseline alongside so that the comparison is a fact and not a memory.

And you know how to save and serve: SavedModel, upload to the Model Registry, pre-built TensorFlow Serving or Cloud Run as an alternative (07-02), with the observation that here only one of the two towers is served. You know about quantisation and why latency is measured in p95 and not as an average. And you have the golden rule engraved: no GPU switched on with no work to do.

But look at what it has cost. A container, a data pipeline, an architecture, a custom loss, a training run, an experiment, a three-week A/B test, and still the honest possibility that the thirty lines of SQL win. That is the price of control, and sometimes it is worth it.

Sometimes it is not. Because AlpinaShop has two other outstanding questions: 8,000 customer reviews in free text that nobody has read, and the 60 GB of images that AutoML classified by category but from which nothing else has been extracted. For both there are already-trained models, built by people with more data and more resources than AlpinaShop will ever have, available with one API call and without a single minute of training.

In 05-04, the Natural Language API, we apply the opposite principle to this lesson's: do not train what is already trained. You will see exactly what the Natural Language API offers — sentiment, entities, classification, syntax — you will finally understand what score and magnitude mean, which is where everybody goes wrong, you will process the 8,000 reviews from Python with quota and cost control, you will cross the sentiment with the returns in BigQuery to answer whether the worst-rated products are returned more, and you will see honestly where it fails — irony, negations and mountain jargon — and when it pays to ask Gemini instead or to train your own classifier.

TensorFlow on GCP: Training and Serving Models

Contents

When you need to go down to the code

TensorFlow and Keras: the bare essentials

Embeddings: the idea that makes the recommender possible

The two-tower architecture

AlpinaShop's model in code

`tf.data`: reading data without starving the GPU

TFRecord and reading from Cloud Storage

Training on Vertex AI: packaging the code

CPU, GPU or TPU: how to choose

Distributed training, conceptually

Checkpoints, Spot VMs and why they go together

Managed TensorBoard and Vertex AI Experiments

Saving and serving: SavedModel, Registry and endpoint

Optimising for inference

PyTorch, JAX and the golden rule of cost

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Solution 1

Solution 2

Solution 3

Conclusion

Google Cloud Platform (GCP) Course

Module 1: Introduction to Google Cloud Platform

Module 2: Core GCP Services

Module 3: Networking and Security

Module 4: Data and Analytics

Module 5: Machine Learning and AI

Module 6: DevOps and Monitoring

Module 7: Advanced GCP Topics

Module 8: Final Project

TensorFlow on GCP: Training and Serving Models

Contents

When you need to go down to the code

TensorFlow and Keras: the bare essentials

Embeddings: the idea that makes the recommender possible

The two-tower architecture

AlpinaShop's model in code

tf.data: reading data without starving the GPU

TFRecord and reading from Cloud Storage

Training on Vertex AI: packaging the code

CPU, GPU or TPU: how to choose

Distributed training, conceptually

Checkpoints, Spot VMs and why they go together

Managed TensorBoard and Vertex AI Experiments

Saving and serving: SavedModel, Registry and endpoint

Optimising for inference

PyTorch, JAX and the golden rule of cost

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Solution 1

Solution 2

Solution 3

Conclusion

Google Cloud Platform (GCP) Course

Module 1: Introduction to Google Cloud Platform

Module 2: Core GCP Services

Module 3: Networking and Security

Module 4: Data and Analytics

Module 5: Machine Learning and AI

Module 6: DevOps and Monitoring

Module 7: Advanced GCP Topics

Module 8: Final Project

`tf.data`: reading data without starving the GPU