The churn service is deployed with 3 replicas and autoscaling. The next question will come from finance before it comes from engineering: how much does this cost, and could it cost less? And product will add its own: do we really always answer within the 300 ms? Inference optimization answers both, but it has one golden rule this lesson will repeat to the point of boredom: measure before you optimize. Optimizing without measuring is guessing, and in inference intuition fails spectacularly (spoiler: predict is almost never where the time goes, and a GPU is almost never the answer for a tabular model). In this lesson we'll measure CineClick's service with percentiles — not averages —, break down where every millisecond goes, and apply the optimizations in order of return: vectorization, caching, a lighter model, ONNX, and (ruling out) hardware. We'll close the module by taking stock of what's been built... and of everything we still do by hand.
Contents
- Measure before optimizing: percentiles, throughput, and cost
- Measuring CineClick's service: a load script
- Where the time goes in a prediction
- Optimization 1: vectorized prediction (request batching)
- Optimization 2: caching features and responses
- Optimization 3: a lighter model, with the loss quantified
- Optimization 4: ONNX, a format built for inference
- Optimization 5 (or not): CPU versus GPU
- The levers table and closing the module
Measure before optimizing: percentiles, throughput, and cost
Three metrics define an inference service's performance:
Latency by percentiles. The average lies, and it's worth understanding why with an example: if 99 requests take 20 ms and one takes 2,000 ms, the average is ~40 ms — "all good" — while one customer in a hundred suffers through 2 seconds staring at the cancellation screen. That's why you measure with percentiles:
- p50 (median): the typical experience. Half the requests respond faster.
- p95: the experience of the worst 1 in 20. Spike effects, GC, contention live here.
- p99: the long tail. In user-facing services, p99 is what breaks the budgets — and the first thing to worsen when something goes wrong.
CineClick's goal is stated like this: p95 < 300 ms (not "a 300 ms average": that would allow horrible tails).
Throughput: requests per second the service sustains without degrading latency. Latency and throughput trade off against each other: as you approach saturation, p99 shoots up long before the service "goes down".
Cost per 1,000 predictions: the metric that connects engineering with finance. It's computed from infrastructure cost and real volume:
With CineClick's numbers: 3 replicas requesting 0.25 CPU / 512 MiB on a managed Kubernetes cost on the order of €90/month; with ~600,000 predictions/month from the cancellation flow, that comes to €0.15 per 1,000 predictions. This number turns any optimization into a sentence finance understands: "the cache cuts the cost per thousand in half".
Measuring CineClick's service: a load script
You don't need a heavyweight tool for an honest first measurement. A script with httpx and asyncio that fires concurrent requests and computes percentiles:
# scripts/load_test.py — latency measurement with percentiles
import asyncio
import statistics
import time
import httpx
URL = "http://localhost:8000/predict"
TEST_CUSTOMER = {
"customer_id": "CLI-04217", "tenure_months": 3, "weekly_hours": 1.5,
"support_tickets": 4, "plan": "basic", "payment_method": "card",
"active_discount": False,
}
N_REQUESTS = 1000
CONCURRENCY = 20 # simultaneous requests: simulates real traffic, not a one-by-one queue
async def one_request(client: httpx.AsyncClient, sem: asyncio.Semaphore) -> float:
async with sem: # caps concurrency at the configured value
t0 = time.perf_counter()
r = await client.post(URL, json=TEST_CUSTOMER)
r.raise_for_status()
return (time.perf_counter() - t0) * 1000 # ms
async def main() -> None:
sem = asyncio.Semaphore(CONCURRENCY)
async with httpx.AsyncClient(timeout=10) as client:
t0 = time.perf_counter()
latencies = await asyncio.gather(
*[one_request(client, sem) for _ in range(N_REQUESTS)]
)
duration = time.perf_counter() - t0
latencies.sort()
def pct(p: float) -> float:
return latencies[int(len(latencies) * p) - 1]
print(f"requests: {N_REQUESTS} concurrency: {CONCURRENCY}")
print(f"throughput: {N_REQUESTS / duration:.0f} req/s")
print(f"p50: {pct(0.50):.1f} ms p95: {pct(0.95):.1f} ms p99: {pct(0.99):.1f} ms")
print(f"mean (for comparison): {statistics.mean(latencies):.1f} ms")
if __name__ == "__main__":
asyncio.run(main())Result against the local container from 04-03 (a development laptop; the absolute numbers will vary on your machine — what matters are the proportions):
requests: 1000 concurrency: 20 throughput: 210 req/s p50: 38.2 ms p95: 92.5 ms p99: 148.7 ms mean (for comparison): 47.1 ms
Reading it: a p95 of ~93 ms — inside the 300 ms budget with room to spare. Notice that the mean (47 ms) is 23% higher than the p50: the tail is already showing. For more serious loads there are dedicated tools — locust (Python, programmable scenarios) and k6 (JavaScript, widely used on platform teams) — that add load ramps, virtual users, and reports; we don't cover them here, but the concept is the same: controlled concurrency + percentiles. And one scope distinction: this is a point-in-time measurement of performance, a snapshot; the continuous watching of these same metrics in production is monitoring, and it arrives in lesson 06-01.
Where the time goes in a prediction
Before optimizing anything, let's break a request down. We instrument each segment of the endpoint with time.perf_counter() (temporarily, or better: leaving the per-segment timings in 04-02's structured log) and average over a thousand requests:
| Segment | What it does | Typical time | % of total |
|---|---|---|---|
| Deserialization + validation | JSON → validated Pydantic object | ~1 ms | 3% |
| DataFrame construction | dict → 1-row pd.DataFrame |
~8 ms | 21% |
build_features |
ratio_tickets, encodings... over 1 row | ~9 ms | 24% |
predict_proba |
the RandomForest's 300 trees | ~18 ms | 47% |
| Response serialization | object → JSON | ~1 ms | 3% |
| Network and framework | HTTP, routing, threadpool | ~1 ms | 3% |
Two surprises almost everyone gets the first time:
- More than half the time is NOT the model. Creating a one-row DataFrame and pushing features through pandas costs almost as much as the 300 trees: pandas is designed to operate over thousands of rows, and its fixed per-operation overhead eats lone rows alive.
- The 1-row
predict_probatakes ~18 ms... and the 1,000-row one takes ~50 ms. Not a thousand times more: three times more. That chasm is the first optimization.
The moral: if we had "optimized the model" without measuring, we'd have attacked the 47% while ignoring the 45% spent preparing one row. Profiles rule.
Optimization 1: vectorized prediction (request batching)
The demonstration with numbers, runnable as-is:
# scripts/vectorization_demo.py
import time
import mlflow
import pandas as pd
from cineclick_churn.features import build_features
model = mlflow.sklearn.load_model("models:/churn-cineclick@champion")
df = pd.read_csv("data/churn_customers.csv").drop(columns=["customer_id", "churned"])
X = build_features(df).head(1000)
# A) 1000 predictions of 1 row (what the API does with individual requests)
t0 = time.perf_counter()
for i in range(1000):
model.predict_proba(X.iloc[[i]])
print(f"1000 x predict(1 row): {time.perf_counter() - t0:.2f} s")
# B) 1 prediction of 1000 rows
t0 = time.perf_counter()
model.predict_proba(X)
print(f"1 x predict(1000 rows): {time.perf_counter() - t0:.2f} s")A factor of ~370x. The reason: every predict_proba call pays a fixed cost (input validation, array conversion, dispatch to the trees) that dominates when the payload is one row; over a thousand rows, that cost gets amortized and the trees process vectorized. How to exploit it at CineClick:
- The weekly batch already does it right: 04-01's
score_batch.pyscores the entire base in a singlepredict_proba. Scoring 100,000 customers takes seconds — never write that job as a loop of API calls (100,000 × ~40 ms ≈ over an hour, plus the cost of the API enduring it). - The
/predict-batchendpoint (04-02's exercise) gives internal consumers with N customers the vectorized route: one call, onepredict_proba. - For the cancellation flow (1-row requests arriving individually) there is an advanced technique, adaptive micro-batching — accumulating the requests of a few-ms window and predicting them together — which specialized serving frameworks ship out of the box. With our p95 of 93 ms we don't need it; knowing it exists is enough.
Optimization 2: caching features and responses
The second lever: never compute the same thing twice. It's 04-01's hybrid pattern (precompute + serve from cache) applied in miniature, inside the service:
- Cache the full response keyed by the input's hash (we already compute that hash for the log in 04-02!): if the same customer with the same data asks twice within a short window — frontend retries, a double click, the user bouncing back and forth from the cancellation screen — the second response comes from memory in <1 ms.
- A short, deliberate TTL: the cache expires (say, after 300 seconds) because freshness was the online pattern's whole reason for existing. A 5-minute TTL is an explicit compromise: "we accept serving a prediction up to 5 min old in exchange for cutting cost"; a TTL of hours would bring us back, through the back door, to the stale-batch problem.
# in-memory cache sketch with TTL (per replica; to share across replicas: Redis)
import time
CACHE: dict[str, tuple[float, "PredictionOutput"]] = {}
TTL_SECONDS = 300
def get_cached(key: str):
entry = CACHE.get(key)
if entry and time.time() - entry[0] < TTL_SECONDS:
return entry[1] # hit: a response in microseconds
return None
def store(key: str, output) -> None:
CACHE[key] = (time.time(), output)The key is 04-02's input_hash(payload). With a 20% hit rate (measurable: log hits/misses), 20% of requests go from ~40 ms to <1 ms and — more relevant for finance — stop consuming CPU: fewer HPA replicas at peak. If the expensive segment were the features (aggregations over history, calls to other services), you'd cache the features per customer instead of the response; at CineClick, build_features is cheap and the full response is the natural unit.
Optimization 3: a lighter model, with the loss quantified
The champion's 300 trees with max_depth=20 — are they all necessary in serving? Laura retrains reduced variants (same data, same dvc repro pipeline, everything tracked in MLflow's churn-cineclick experiment, as module 3 mandates) and we measure the three dimensions that matter:
| Variant | Size on disk | p95 latency (1 row) | Recall | Precision | F1 |
|---|---|---|---|---|---|
| Champion v2 (300 trees, depth 20) | 145 MB | 92 ms | 0.68 | 0.55 | 0.61 |
| 150 trees, depth 20 | 74 MB | 71 ms | 0.67 | 0.55 | 0.60 |
| 100 trees, depth 12 | 21 MB | 58 ms | 0.65 | 0.54 | 0.59 |
| 50 trees, depth 8 | 4 MB | 49 ms | 0.60 | 0.52 | 0.56 |
(Metrics on the same evaluation set from module 3; latencies from the load script, same machine.)
The right way to read this table is not technical but business: going from 300 to 150 trees saves 20% of latency and half the memory in exchange for one point of recall (0.68 → 0.67) — a point of recall is customers who churn without anyone making them an offer. Is it worth it? With p95 = 92 ms against a budget of 300, there is no latency problem that justifies paying recall: CineClick keeps the champion intact. A different decision would be legitimate in another context (a 50 ms budget, or expensive memory): what matters is that it's now a decision quantified on both sides of the scale, not an "I trimmed trees because it felt slow". If a lighter variant is ever adopted, it enters through the usual path: a new version in the registry, human review, alias.
Optimization 4: ONNX, a format built for inference
ONNX (Open Neural Network Exchange) is a standard model-interchange format: you export the trained model to a graph of operations, and an optimized runtime (onnxruntime, in C++) executes it. For serving it brings two things:
- Faster inference, especially in the case that hurts us — few rows per call — because it removes the Python/sklearn overhead per prediction.
- Serving without scikit-learn: the inference container only needs
onnxruntime(tens of MB versus the sklearn+scipy+pandas stack), which slims the image and eliminates the whole class of "different sklearn version at deserialization" bugs.
# scripts/export_onnx.py — export the champion to ONNX
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType
import mlflow
model = mlflow.sklearn.load_model("models:/churn-cineclick@champion")
# ONNX requires declaring the input: N rows x n_features of type float
n_features = model.n_features_in_
input_type = [("input", FloatTensorType([None, n_features]))]
onx = convert_sklearn(model, initial_types=input_type)
with open("models/churn.onnx", "wb") as f:
f.write(onx.SerializeToString())
# inference with the runtime (how the service would use it)
import onnxruntime as rt
import numpy as np
session = rt.InferenceSession("models/churn.onnx")
X_sample = np.random.rand(1, n_features).astype(np.float32) # illustrative
probs = session.run(None, {"input": X_sample})[1] # probabilitiesIn tests with CineClick's model, onnxruntime cuts the 1-row predict from ~18 ms to ~2-4 ms. The tolls, to weigh it honestly: it's one more piece to maintain (an export step in the pipeline, verifying the numerical equivalence of sklearn vs. ONNX predictions before trusting it, the occasional unsupported operator depending on the model), and the features remain pandas — ONNX speeds up the predict segment, not the whole pipeline. CineClick notes it down as the next lever if the latency budget tightens; today, with p95 at 92 ms, it doesn't pay for its complexity.
Optimization 5 (or not): CPU versus GPU
Let's say it bluntly: for a RandomForest over tabular data, a GPU doesn't pay off. Decision trees are branches and comparisons — irregular work at which the CPU excels — not the massive matrix multiplications the GPU exists for. Adding a GPU to the churn service would multiply the cost of each replica (a GPU instance costs several times a CPU one) to gain little or nothing, and would complicate the image, drivers, and Kubernetes scheduling.
When IS the GPU a conversation to have? Deep learning at inference: transformers, vision, embedding models — and even then only with batching that fills the GPU (requests arriving one at a time waste it; hence the specialized inference servers with micro-batching). The heuristic for your day-to-day:
| Model | Reasonable inference hardware |
|---|---|
| Tabular models (trees, linear, boosting) | CPU, always. Optimize with levers 1–4 |
| Small networks / embeddings at low volume | CPU is usually enough; measure first |
| Transformers/vision at volume or with demanding latency | GPU with batching (or managed endpoints) |
And the last cost lever you already have from 04-04: autoscaling matches replicas to traffic (paying for actual demand), and for the weekly batch job, ephemeral compute or serverless with scale to zero — a job that runs 10 minutes a week shouldn't have a server waiting for it the other 10,070 minutes.
The levers table and closing the module
The lesson's executive summary:
| Lever | Typical gain | Implementation cost | Does CineClick apply it? |
|---|---|---|---|
| 1. Vectorized / batch prediction | 10–400x in bulk throughput | Nearly zero (you already write the batch and /predict-batch right) |
Yes, by design |
| 2. Response/feature cache (TTL) | Eliminates 100% of compute on every hit | Low (hash already exists + dict/Redis); watch freshness | Yes, TTL 300 s |
| 3. Lighter model | 20–50% latency, 50–95% memory | Medium: retrain and pay in metric (business decision) | No — the recall is worth more than the ms |
| 4. ONNX / optimized runtime | 3–10x on the predict segment |
Medium: export, equivalence verification, extra piece | Noted for the future |
| 5. GPU | Only deep learning with batching | High (instance cost, drivers, complexity) | No — tabular = CPU |
| Autoscaling / scale to zero (04-04) | Pay for actual demand | Already in place (HPA); serverless for the batch | Yes |
The order is not accidental: it's decreasing return. The first two levers cost almost nothing and come free with good architecture; the last ones demand investment and are only justified with measurements in hand. And that's the complete discipline: measure (percentiles) → localize (per-segment breakdown) → apply the cheapest lever that solves it → measure again.
Let's take stock of module 4. CineClick chose its patterns with criteria (batch for marketing, online for cancellation), built the FastAPI service with a Pydantic contract, the registry champion, and single-source features, froze it into the cineclick/churn-api:0.1.0 image, deployed it on Kubernetes with 3–10 replicas, probes, and two rollback levels, and now knows it answers with a p95 of 92 ms at €0.15 per thousand predictions — measured, not assumed. But look back at how all of that reached production: Laura ran the tests by hand, you built the image by hand (docker build), pushed it by hand, applied the manifests by hand (kubectl apply), and the weekly batch job... someone still launches it by hand on a Monday at 6:00. Every "by hand" is an oversight waiting to happen.
Common Mistakes and Tips
- Mistake: optimizing without a profile. Intuition says "the model is the slow part"; measurement says half the time was pandas preparing one row. Without a per-segment breakdown, you'll optimize the wrong segment.
- Mistake: reporting (and promising) average latencies. The mean hides the tail; users live at p95 and p99. Budgets and SLOs always in percentiles.
- Mistake: bulk scoring by calling the API in a loop. 100,000 customers through
/predictis an hour of requests and useless load; the same work vectorized is seconds. Each pattern (04-01) through its own route. - Mistake: caching without a TTL (or with a carefree one). An eternal cache turns your online service into a stale batch in disguise — exactly what the cancellation case can't tolerate. The TTL is a freshness decision; write it down and justify it.
- Mistake: trimming the model without quantifying the loss. "I went down to 50 trees and it flies" — and recall dropped from 0.68 to 0.60 without anyone putting that cost on the table. The size-latency-metric table is mandatory before deciding.
- Mistake: requesting a GPU for a tabular model. It's the most expensive and least useful optimization on the list. CPU + vectorization + cache get you a very long way.
- Tip: keep the measurement runnable in the repo (
scripts/load_test.py): measuring performance should cost one command, because in 05-01 we'll want it to be just another step that runs on its own. - Tip: accompany every optimization proposal with its cost per 1,000 predictions before/after. It's the common language between engineering and business.
Exercises
Exercise 1
A service reports these 10 latencies (ms) under load: 22, 24, 25, 25, 26, 28, 30, 31, 45, 950. Compute the mean, p50, and p90 (with the lesson's method: the value at position int(n × p) - 1 of the sorted list). What story does each metric tell, and which would you use to verify the 300 ms budget?
Exercise 2
Marketing proposes reusing /predict to score the campaign's 80,000 customers "because the API is already deployed and scaled". With the lesson's numbers (p50 ≈ 38 ms per request, throughput ≈ 210 req/s at concurrency 20; vectorized predict_proba of 1,000 rows ≈ 0.05 s), estimate how long each approach would take and write the recommendation.
Exercise 3
The latency budget drops from 300 ms to 60 ms (p95) because the cancellation screen is being redesigned. Using the per-segment breakdown and the lesson's tables, propose a plan in order of return to go from p95 = 92 ms to p95 < 60 ms, stating what you would NOT do yet.
Solutions
Solution 1
They're already sorted. Mean = (22+24+25+25+26+28+30+31+45+950)/10 = 120.6 ms. p50 = position int(10×0.5)-1 = 4 → 26 ms. p90 = position int(10×0.9)-1 = 8 → 45 ms (and that 950 ms request lives at the p100/maximum; with only 10 samples, p99 can't be estimated seriously — another lesson: high percentiles need many samples). Stories: the mean (120 ms) suggests a service 4-5 times slower than what the typical user experiences (p50 = 26 ms) — it's been hijacked by a single outlier; the p90 says 9 out of 10 requests come in at ≤45 ms. For the 300 ms budget you verify the committed percentile (p95/p99 with enough samples): here the only potential violation is that 950 ms tail, which is exactly what should be investigated and what the mean dilutes and the p50 ignores.
Solution 2
- Via the API, sequentially: 80,000 × 38 ms ≈ 3,040 s ≈ 51 minutes of requests, occupying the service (and triggering the HPA: you'd pay for extra replicas to inflict load on yourself).
- Via the API at the measured concurrency: 80,000 / 210 req/s ≈ 380 s ≈ 6.3 minutes at maximum throughput — that is, saturating the service that attends the real users of the cancellation screen for 6 minutes. Unacceptable during service hours.
- Via the vectorized batch (
score_batch.py): 80 batches of 1,000 rows × 0.05 s ≈ 4 seconds ofpredict_proba(plus data reads/writes: on the order of a minute total), without touching the API or its infrastructure.
Recommendation: the campaign gets scored with 04-01's batch job — which already exists, is three orders of magnitude more efficient, and doesn't compete with the online traffic. "The API is already deployed" is not an argument: the API is sized (3-10 replicas, latency budget) for another consumption pattern.
Solution 3
With p95 = 92 ms and a target of < 60 ms, we need to shave off ~35 ms. In order of return according to the breakdown (DataFrame ~8 ms + features ~9 ms + predict ~18 ms dominate):
- Response cache with TTL (already designed): every hit responds in <1 ms; at a 20% hit rate the p95 barely moves (the high percentiles are the cache misses), so it helps cost but does not solve the p95 by itself — the cache-miss path must be attacked.
- Trim the pandas segment (~17 ms): for one row, build the features with dict/numpy operations instead of a DataFrame — but careful: that would brush against the feature reimplementation forbidden by 03-03; the correct route is optimizing
build_featuresinsidefeatures.pyitself (making it accept the fast path too), preserving the single source and with equivalence tests. Estimated gain: 8-12 ms. - ONNX for the
predict(lever 4): from ~18 ms to ~2-4 ms on the model's segment, with numerical-equivalence verification before promoting. Gain: ~14 ms. With steps 2+3, the estimated p95 lands comfortably under 60 ms. - What I would NOT do yet: sacrifice trees/recall (lever 3) — you don't pay in business metric while purely technical levers remain — and, of course, a GPU (lever 5), which doesn't apply to a tabular model. And after every change: rerun
load_test.py, because estimates are hypotheses until the percentile confirms them.
Conclusion
Module 4 ends where the course began to promise: Laura's model no longer lives in a notebook — it takes real requests. The full path is standing and measured: patterns chosen by consumer (weekly batch for marketing, online for cancellation), a FastAPI service that validates with Pydantic, loads models:/churn-cineclick@champion, and imports the features from the single source, a reproducible, secret-free cineclick/churn-api:0.1.0 image, Kubernetes with 3-10 replicas, probes against /health, and two well-separated rollbacks (image and model), and verified performance of p95 = 92 ms at €0.15 per thousand predictions, with the optimization levers prioritized and the discarded ones — GPU included — discarded in writing. But look at the glue holding all of this together: hands. Tests that run if someone remembers, docker build and push from whoever's laptop, kubectl apply typed carefully, a batch job "somebody launches on Mondays". Every manual step is slow, unauditable, and — sooner or later — will get skipped exactly on the day it shouldn't. Module 5 automates the glue: CI that runs code, data, and model tests on every change, CD that builds, publishes, and deploys the image with no hands, orchestration that launches the weekly batch with retries and alerts, and release strategies (shadow, canary, A/B) so the new version earns trust with real traffic before taking it all. The pieces already exist; now it's time they moved on their own.
MLOps Course
Module 1: MLOps Fundamentals
- What MLOps is and why models die in the notebook
- The lifecycle of an ML model in production
- MLOps maturity levels and team roles
- The course project: from notebook to production
Module 2: From Notebook to Reproducible Code
- Structuring an ML project: from notebook to package
- Reproducible environments and dependency management
- Data versioning with DVC
- Reproducible training pipelines
Module 3: Experiments and Model Registry
- Experiment tracking with MLflow
- Model registry: versioning and promoting models
- Feature stores: when and what for
Module 4: Serving Models in Production
- Deployment patterns: batch, online and streaming
- A prediction service with FastAPI
- Packaging with Docker
- Scaling and deployment: Kubernetes and serverless
- Inference optimization: latency and cost
Module 5: Automation: CI/CD and Orchestration
- CI for ML: testing code, data and models
- CD: automating model deployment
- Orchestrating ML pipelines
- Release strategies: shadow, canary and A/B