The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Monitoring services and models

Next ►
►

Module 5 left CineClick with a system that moves on its own but still doesn't watch: the shadow gets switched on by hand, and the only way to know whether the champion is aging is for someone to remember to check. This lesson builds the system's eyes. We'll see that an ML system is watched on two distinct planes — the service plane (does it respond?) and the model plane (does it get things right?) — and that the second can be completely broken while the first glows green on every panel. You'll learn to expose service metrics with prometheus-client, to define operational proxies for watching the model while there are no labels yet, to persist an auditable prediction table, and to close the loop when the ground truth finally arrives.

Contents

  1. The two planes of monitoring
  2. Service plane: RED metrics and /metrics
  3. Prometheus and Grafana in broad strokes
  4. Model plane without labels: operational proxies
  5. The prediction table: the log that becomes data
  6. When the labels arrive: real metrics
  7. One dashboard, two planes

The two planes of monitoring

In lesson 01-02 we planted an idea that it's now time to make concrete: an ML system has two independent ways of failing.

  • Service plane: does the system respond? Latency, error rate, resource saturation. It's exactly what you'd monitor in any microservice; nothing here is ML-specific.
  • Model plane: does the system get things right? Is it receiving data similar to what it saw in training? This is exclusive to ML, and it's the plane traditional infrastructure tools do not see.

The trap is that they are genuinely independent:

Situation Service plane Model plane Does an infrastructure dashboard catch it?
Pod down, timeouts Red — Yes
p95 at 800 ms after an undersized deployment Red Green Yes
The data team changes the unit of weekly_hours to minutes Green Red No
A marketing promotion attracts a new customer cohort Green Red (degrading) No

In the last two rows, CineClick's API returns 200 OK in 38 ms, the HPA is calm, Kubernetes reports 3/3 healthy replicas… and the model is predicting garbage. A "green" service with a broken model is the characteristic failure mode of ML systems, and the reason this module exists.

flowchart LR
    subgraph Service plane
        A["Does it respond?<br/>latency, errors, saturation"]
    end
    subgraph Model plane
        B["Does it get it right?<br/>normal data? normal predictions?"]
    end
    A -->|"standard tools<br/>(Prometheus, Grafana)"| C[Dashboard]
    B -->|"prediction table,<br/>proxies, ground truth"| C

Service plane: RED metrics and /metrics

For the service plane we'll use the RED pattern, a de facto standard for request/response services:

  • Rate: requests per second.
  • Errors: rate of error responses (4xx from the client, 5xx ours).
  • Duration: the latency distribution (p50, p95, p99 — not the mean, which hides the tails).

The canonical way to expose them is a GET /metrics endpoint in Prometheus format. We add prometheus-client to the FastAPI service we built in 04-02:

# src/cineclick_churn/api/metrics.py
from prometheus_client import Counter, Histogram

# Counter: a value that only grows. Prometheus computes rates from it.
PREDICTIONS_TOTAL = Counter(
    "cineclick_predictions_total",
    "Number of predictions served",
    ["result"],  # label: "at_risk" (p >= threshold) or "no_risk"
)

# Histogram: bins observations into buckets to compute percentiles.
# The buckets are chosen around what we care about: the 300 ms budget.
PREDICTION_LATENCY = Histogram(
    "cineclick_prediction_latency_seconds",
    "End-to-end prediction latency",
    buckets=[0.01, 0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1.0],
)

Piece-by-piece explanation:

  • A Counter only adds. We don't store "requests per second": we store the accumulated total, and Prometheus derives the rate with rate(). That makes it robust to restarts and missed scrapes.
  • The result label gives us a model metric inside the service plane for free: the positive rate (the proportion of "at_risk" predictions), which we'll use as a proxy later.
  • A Histogram doesn't store each individual latency, but how many observations fell into each bucket. That's why the buckets must surround the critical values: with the real p95 at 92 ms and the budget at 300 ms, we place fine buckets below 300 ms and coarse ones above.

We instrument the 04-02 endpoint without touching its logic:

# src/cineclick_churn/api/routing.py (fragment)
import time
from .metrics import PREDICTIONS_TOTAL, PREDICTION_LATENCY

@router.post("/predict")
def predict(customer: CustomerInput):
    t0 = time.perf_counter()
    probability = model.predict_proba(build_features(customer))[0, 1]
    result = "at_risk" if probability >= CHURN_THRESHOLD else "no_risk"

    PREDICTIONS_TOTAL.labels(result=result).inc()
    PREDICTION_LATENCY.observe(time.perf_counter() - t0)
    # ... the JSON logging from 04-02 stays as is ...
    return PredictionOutput(churn_probability=probability, prediction=result)

And we expose the endpoint in main.py:

# src/cineclick_churn/api/main.py (fragment)
from prometheus_client import make_asgi_app

app.mount("/metrics", make_asgi_app())

With this, curl http://localhost:8000/metrics returns plain text with the counters and the histogram buckets. Nothing has to be sent anywhere: Prometheus is the one that comes to read.

Prometheus and Grafana in broad strokes

We're not going to turn this into a Prometheus tutorial (that's what platform operations is for); just enough to get the system wired up. The mental model is pull: Prometheus visits each pod's /metrics every few seconds (scrape), stores the time series, and Grafana draws them.

On Kubernetes, with the Prometheus Operator CineClick's platform already uses, declaring what should be scraped is enough:

# deploy/k8s/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: churn-api
  namespace: cineclick
spec:
  selector:
    matchLabels:
      app: churn-api          # the Service from module 4
  endpoints:
    - port: http
      path: /metrics
      interval: 15s           # scrape frequency

The two service-plane alerts come straight from the guardrail metrics we defined in 05-04 for the canaries — here we make them permanent for 100% of the traffic:

# deploy/k8s/service-alerts.yaml (PrometheusRule fragment)
groups:
  - name: churn-api-service
    rules:
      - alert: P95LatencyExceeded
        # histogram_quantile computes the percentile from the buckets
        expr: histogram_quantile(0.95,
                rate(cineclick_prediction_latency_seconds_bucket[5m])) > 0.3
        for: 10m
        labels: {severity: warning, channel: ml-alerts}
      - alert: ErrorRate5xx
        expr: (sum(rate(http_requests_total{status=~"5.."}[10m]))
               / sum(rate(http_requests_total[10m]))) > 0.01
        for: 10m
        labels: {severity: critical, channel: ml-alerts}

How to read it: if the p95 exceeds the 300 ms budget for 10 minutes, or 5xx responses exceed 1% of traffic, a notice lands in #ml-alerts. These are exactly the same thresholds that triggered the canary's automatic rollback: keeping "what watches a release" consistent with "what watches the day-to-day" avoids having two different truths.

Model plane without labels: operational proxies

Here begins the ML-specific part, and with it this lesson's key concept: ground truth delay. To know whether a churn prediction was right, you have to wait for the customer to cancel… or not. At CineClick that means weeks. Today we cannot compute the recall of today's predictions; we'll only be able to do it when the future arrives.

Does the model plane go blind in the meantime, then? No: we watch operational proxies — signals that don't measure accuracy but that, if they move, indicate something odd is happening before the ground truth confirms it:

Proxy What it watches Alarm signal at CineClick
Distribution of emitted probabilities The "shape" of what the model thinks The score histogram changes shape from one week to the next
Positive rate (p ≥ 0.5) The volume of customers flagged at risk Outside the expected 10%–25% band (the same guardrail band from 05-04)
Volume by plan and payment_method The input mix The basic plan goes from 40% to 70% of requests
% of outlier values in features The quality of the incoming data weekly_hours > 100, tenure_months = 0 with support_tickets = 50…

Notice the logic: none of these proxies requires knowing who cancelled. The positive rate we already have in Prometheus thanks to the counter's result label. The rest require looking at the predictions in more detail — and for that we need to persist them.

One scoping clarification: if a proxy moves, how much has it moved and is it statistically relevant? That question — formal drift, with PSI and tests — is lesson 06-02. Here we just leave the signals switched on.

The prediction table: the log that becomes data

Since 04-02 the service already writes one JSON line per prediction (input_hash, probability, model_version, latency_ms, and since 05-04 the shadow/canary fields). Until now that log was only for debugging. This lesson's leap is treating it as first-class data: the prediction table is the raw material of all model monitoring, of 06-02's drift analysis, and of the real evaluation we'll see in the next section.

Where does it go? A collector (at CineClick, the platform's own log stack) gathers the pods' JSON lines and materializes them as Parquet partitioned by date — the same format the weekly batch already uses:

storage/predictions/online/date=2026-07-06/predictions.parquet

What must each row contain so that three months from now we can audit and evaluate?

Field What it's for later
prediction_id, timestamp Identifying and ordering; the temporal window of the join with labels
input_hash Detecting duplicates and auditing without storing the raw payload
features (the ones that entered the model, including ratio_tickets) Reproducing the prediction; per-feature drift analysis
probability Score distribution, positive rate, real evaluation
model_version, alias Attributing each prediction to a specific model (v2 or the challenger?)
shadow_probability, shadow_model_version Comparing champion vs. challenger offline
group (canary/control/stable) Not mixing populations when evaluating
latency_ms Correlating service degradations with model changes

Golden rule: store the features exactly as they entered the model, not just the raw input. If features.py changes tomorrow, the raw input no longer reproduces what the model saw.

When the labels arrive: real metrics

Weeks later, the ground truth arrives: CineClick's billing system knows which customers cancelled. Now we can answer the question no proxy answers: what REAL recall and precision is the champion delivering in production?

The mechanism is a job (at CineClick, one more task inside the Prefect ecosystem that already orchestrates the Monday scoring) that matches predictions with cancellations:

# flows/ground_truth_evaluation.py (fragment of the main task)
import pandas as pd

def evaluate_week(predictions: pd.DataFrame, cancellations: pd.DataFrame,
                  window_days: int = 30) -> dict:
    """Matches predictions with real cancellations and computes production metrics.

    predictions: prediction table for ONE week (customer_id, timestamp,
                 probability, model_version, group).
    cancellations: confirmed cancellations (customer_id, cancellation_date).
    """
    df = predictions.merge(cancellations, on="customer_id", how="left")

    # Real label: the customer cancelled WITHIN the window after the prediction.
    # The window matters: a cancellation 6 months later doesn't validate today's prediction.
    df["actual_churn"] = (
        (df["cancellation_date"].notna())
        & (df["cancellation_date"] > df["timestamp"])
        & (df["cancellation_date"] <= df["timestamp"] + pd.Timedelta(days=window_days))
    ).astype(int)

    df["prediction"] = (df["probability"] >= 0.5).astype(int)

    tp = int(((df.prediction == 1) & (df.actual_churn == 1)).sum())
    fp = int(((df.prediction == 1) & (df.actual_churn == 0)).sum())
    fn = int(((df.prediction == 0) & (df.actual_churn == 1)).sum())

    return {
        "real_recall": tp / (tp + fn) if (tp + fn) else None,
        "real_precision": tp / (tp + fp) if (tp + fp) else None,
        "n_predictions": len(df),
        "n_cancellations": int(df.actual_churn.sum()),
    }

Three details that make or break this job:

  1. The join is by id AND by temporal window. Without the window, any historical cancellation "validates" any prediction and the metrics come out inflated.
  2. Filter by group. Customers in the permanent control group (05-04) don't receive the retention discount: they are the only population whose label isn't contaminated by our own intervention. The "clean" metrics are computed on them.
  3. Always compare against the offline reference. The champion v2 promised recall 0.68 and precision 0.55 on the test set. If production delivers a stable 0.62/0.51, that's the normal offline→online degradation. If real recall falls week after week, the model is aging — and that trend is the input to 06-03's retraining policy.

The result of each run gets written as a time series (real_recall, real_precision per week and per model_version), ready to plot alongside everything else.

One dashboard, two planes

The classic mistake is having the service dashboard in Grafana and "the model stuff" in a notebook someone runs when they remember. CineClick builds a single dashboard with both planes, because real incidents cross the border (a latency spike coincides with a change in the input mix; a drop in the positive rate coincides with a deployment). Panel sketch:

Row Panels Source
Service req/s, 5xx rate, p50/p95/p99 vs. the 300 ms line, replicas/HPA Prometheus
Model (proxies) probability histogram per day, positive rate with the 10–25% band, mix by plan/payment method, % outliers Prediction table
Model (ground truth) real recall and precision per week vs. the offline lines (0.68 / 0.55), with deployment annotations Real evaluation job
Context active model version (/version), release events (shadow/canary) Registry + CD

The context row is what turns the dashboard into a diagnostic tool: when something goes sideways, the first question is always "what changed and when?".

Common Mistakes and Tips

  • Monitoring only the service plane. It's mistake number one: the platform team "already monitors" the API… and nobody checks whether the model gets things right. Remember the table at the start: the typical ML failure mode is green service, red model.
  • Using mean latency instead of percentiles. With p50 at 38 ms and p95 at 92 ms, the mean will say ~45 ms and hide that 5% of your users wait twice as long. Alert on p95/p99, always.
  • Logging the raw payload "just in case". Besides the privacy problem (we come back to it in 06-04), it bloats storage. The input_hash + the processed features cover auditing and analysis.
  • Not storing model_version in each prediction. When champion, shadow, and canary coexist, a table without versions is useless: you won't know whom to credit each hit or miss to.
  • Matching predictions with labels without a temporal window. It inflates recall and precision silently. Define the window according to the business (in churn, the prediction's horizon) and document it.
  • Alerting on everything. If #ml-alerts receives ten notices a day, it will stop being read. Start with a few actionable alerts (the two service ones + positive rate out of band) and add more only when an absence has hurt you.

Exercises

  1. Instrumenting the batch. The weekly-churn-scoring flow (05-03) also makes predictions, but it's not an HTTP service: Prometheus can't scrape it continuously. Which model-plane metrics would you record on each weekly run, and where would you persist them so they show up on the same dashboard?

  2. Diagnosis with the prediction table. One Monday you observe: p95 = 85 ms, 5xx = 0%, positive rate = 4% (it had been between 12% and 16% for months). The platform team says "everything's green". List, in order, the first three checks you'd make using the prediction table and the /version endpoint.

  3. Evaluation window. The ground truth job uses window_days=30. Marc proposes lowering it to 7 to "get real metrics sooner". What would happen to real_recall and real_precision with that window, and why? What compromise would you propose?

Solutions

  1. On each flow run: the batch's probability distribution (score percentiles), positive rate over the total scored, number of rows scored (a collapse indicates an upstream problem), and % of outliers per feature. They get persisted as rows in the same prediction table (the batch already writes data/churn_predictions.parquet; it's enough to add the same fields as the online path: version, timestamp, features) and/or pushed to Prometheus via Pushgateway, which exists precisely for ephemeral jobs. That way the single dashboard shows online and batch together, distinguished by a source label.

  2. First, GET /version: did model_version or alias change recently? An accidental promotion or rollback is the fastest cause to rule out. Second, in the prediction table, compare this week's probability histogram with the previous weeks': if the whole distribution shifted downward, the model "thinks differently" — likely a change in the input data. Third, compare the input mix (distribution of plan, payment_method, and statistics of weekly_hours, tenure_months, ratio_tickets) against previous weeks to locate which feature moved; a constant or null value gives away a failure in the data producer. Note: the service is healthy — the problem is data or model, exactly the "green on the outside" scenario.

  3. With 7 days, many cancellations the prediction anticipated correctly would fall outside the window and count as false positives: real_precision would drop artificially, and real_recall would be computed over far fewer cancellations (only the very immediate ones), becoming noisy and unstable. The usual compromise: keep the window aligned with the business horizon (30 days) for the official metrics, and additionally publish a "7-day partial" metric clearly labeled as a provisional preview, useful as an early signal but never as a decision criterion.

Conclusion

CineClick now has eyes. The service plane is covered with RED metrics exposed at /metrics, scraped by Prometheus and alerted with the same guardrail thresholds from 05-04 (p95 > 300 ms, 5xx > 1%). The model plane — the one no infrastructure tool sees — is covered in two stages: operational proxies while the ground truth hasn't arrived (score distribution, positive rate within the 10–25% band, input mix, outliers) and real metrics when it does, thanks to the job that matches the prediction table with confirmed cancellations and compares production's recall/precision against the 0.68/0.55 promised offline. The centerpiece is that prediction table: the JSON log from 04-02 elevated to first-class data, with features, version, and group on every row. But so far we've only said that the proxies "move" — not how much, nor with what significance. Putting rigorous numbers on that movement, telling a weird Monday apart from a real change in the population, and automating that vigilance with Evidently is exactly the topic of the next lesson: data drift and concept drift.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved