Module 4 ended with an uncomfortable confession: CineClick's whole system works, but the glue is human hands. The tests run "if someone remembers", and that means a git push on a Friday afternoon can break features.py without anyone noticing until Monday's batch produces absurd predictions. Continuous integration (CI) solves exactly that: every change automatically triggers a battery of checks, and the change doesn't get integrated if anything fails. But on an ML project, classic CI falls short, because the system's behavior doesn't depend on the code alone: it depends on the code, on the data, and on the model. In this lesson we'll build the real CI of cineclick-churn: we'll extend the existing code tests, add data validation with pandera, write model tests (smoke, behavioral, and metric-threshold), and tie it all together with a GitHub Actions workflow.

1. What changes in CI when ML is involved: three things to test
2. The testing pyramid for ML
3. Code tests: features and API
4. Data tests: schema and expectations with pandera
5. Model tests: smoke, behavior, and metric thresholds
6. The workflow: .github/workflows/ci.yml explained block by block
7. What runs on each event

In a traditional web application, CI means: on every push, run the linter and the tests; if they pass, the code can be merged into main. The implicit premise is that code is the only source of behavior: if the code doesn't change, the system behaves the same.

In ML that premise is false. The behavior of CineClick's churn service is the product of three ingredients, and any of the three can degrade independently:

• Code: features.py, train.py, the FastAPI API. It changes when Laura or Marc commit. It's tested as always: unit and integration tests.
• Data: churn_customers.csv and its future updates. They change when the data team exports a new version, even if nobody touches a line of code. A silent schema change (a renamed column, a new plan, nulls where there were none) can poison training without producing any Python error at all.
• Model: the artifact registered in MLflow. A model retrained with the same hyperparameters can be worse than the previous one if the data has changed, and pytest has no idea what a recall is.

The practical consequence: our CI needs three families of tests, not one. And each family fails for different reasons and protects against different risks:

The classic testing pyramid (lots of cheap unit tests at the bottom, few expensive end-to-end tests at the top) still applies, but in ML it grows extra layers. Ordered from cheapest and most frequent to most expensive and sporadic:

The golden rule: the cheap stuff always runs; the expensive stuff runs when it earns its keep. It makes no sense to evaluate the challenger's recall on every push to a working branch, but it is mandatory before that challenger aspires to @champion.

In module 3 we decided that features.py is the single source of truth for features: training, the online API, and batch scoring all use it. That makes it the most critical file in the repo — a bug here propagates to all three consumers at once. It deserves the most exhaustive tests.

Recall the detail we pinned down back then: ratio_tickets is computed as support tickets per month of tenure, with clip(lower=1) on the denominator to avoid division by zero for brand-new customers. That's exactly the kind of decision a teammate accidentally "simplifies" away in a refactor. We armor it with tests:

Points worth understanding well:

• _base_customer as a fixture factory: instead of repeating a 7-field dictionary in every test, one valid customer is defined and each test changes only what's relevant. The tests read like sentences: "a customer with tenure 0 and 3 tickets has a ratio of 3".
• The one-hot test encodes a design decision, not just a computation: build_features must use a fixed list of known categories (for example pd.Categorical with explicit categories), because bare pd.get_dummies generates columns from whatever it sees, and a new plan in production would produce a DataFrame with columns different from training's — the model would fail or, worse, predict over shuffled columns.
• The docstrings explain the why. When this test fails a year from now, whoever reads it will know what it protects.

The API already has its validation test from module 4: checking that a malformed request returns 422 without invoking the model. This test is gold in CI because it needs neither MLflow nor artifacts — it runs in seconds on any runner:

In CI this test validates the service's input contract: the schemas in schemas.py are the boundary between the outside world and the model, and any accidental relaxation (someone removes a Pydantic constraint) gets caught here.

You can validate a DataFrame with loose asserts (assert df["plan"].isin(...)), and for trivial cases that's fine. But a declarative schema with pandera brings three things asserts don't: all the errors at once (not just the first), a readable report of what failed and in which rows, and a schema object you can import from anywhere (CI, training, batch scoring) — once again, a single source of truth, this time of the data contract.

We define it in the package, not in the tests, precisely so it can be reused:

Breaking down the decisions:

• Types and ranges: tenure_months between 0 and 240 (20 years; CineClick didn't exist before that), weekly_hours between 0 and 168 (there are no more hours in a week). Impossible ranges are the most common trap of a broken export.
• Closed categories: plan and payment_method admit only the known values. If a legitimate new plan appears, this check must fail: it's the signal to update features.py, retrain, and update the schema, in that order and deliberately — not in silence.
• Null proportion: weekly_hours allows nulls (some customers have no telemetry), but if they exceed 5% something is wrong in the instrumentation. It's a statistical expectation, not a per-row rule.
• Churn rate ≤ 30%: the historical dataset hovers around 15%. If a 45% suddenly shows up, the most likely explanation isn't a customer stampede but a broken filter in the extraction. Validating the shape of the target variable keeps you from training on garbage.
• strict=True turns "someone added a column" into a visible error rather than a surprise.

Validating is one line: customer_schema.validate(df, lazy=True) (with lazy=True pandera accumulates all the errors into a single report instead of stopping at the first).

There's an important subtlety here: the full data is not in the repo (it lives in the DVC storage remote), and CI shouldn't download gigabytes on every PR. The solution has two legs:

1. In CI: a versioned sample is validated (data/samples/churn_customers_sample.csv, about 2,000 rows, tracked with DVC just like the full dataset). It detects schema and contract changes at minimal cost. The workflow does dvc pull on that sample only.
2. Before every training run: the first action of the prepare_data stage in dvc.yaml (and, as we'll see in 05-03, the second task of the scoring flow) is validating the full dataset with the same customer_schema. Same schema, two locations: CI watches the contract; the pipeline watches the day's real data.

The most profitable test in this whole lesson: checking that the training pipeline works end to end on tiny synthetic data. It doesn't validate quality — it validates that nobody has broken the data → features → fit → predict chain:

Note the tiny hyperparameters (n_estimators=10): we don't want the production model, we want speed. This test takes seconds and would have caught half of the "works on my machine" episodes from module 1.

This idea comes from the CheckList paper (Ribeiro et al., 2020), originally conceived for NLP models, and its intuition carries over perfectly to tabular data: besides asking "what metric does the model score?", you have to ask "does the model behave according to what we know about the problem?". A model can have good overall recall and still have learned an absurd relationship that will explode on future data. Two kinds of check:

• Invariance: some inputs shouldn't affect the prediction. customer_id is an arbitrary identifier; if changing it changes the churn probability, the model (or more likely features.py) is leaking the identifier as a feature — a serious, silent bug.
• Directionality: some changes have a direction we know even if we don't know the magnitude. An otherwise identical customer with more support_tickets is, at the very least, just as dissatisfied: their churn probability should not go down. If it does, either the features are badly built or the model learned noise in that region.

The challenger_model fixture loads models:/churn-cineclick@challenger from the registry (it needs MLFLOW_TRACKING_URI, which is why these tests carry the @pytest.mark.model marker and don't run on just any push). Note the -0.02 tolerance in the directionality test: a RandomForest isn't monotonic by construction, and demanding strict monotonicity at a single point would produce false positives; what we're hunting is a clear inversion of the relationship.

In module 3 the promotion criterion was written in prose: "precision ≥ 0.50 if recall > 0.60". Prose doesn't block merges; tests do. We turn it into the final gate, evaluating the challenger against the versioned test set (data/processed/test.csv, the split with test_size=0.2 and random_state=42 from params.yaml, tracked by DVC — if the test set changed on every run, metrics wouldn't be comparable across candidates):

For reference: the current champion (v2) scores recall 0.68 and precision 0.55, so it passes with room to spare. A challenger that doesn't pass this test never even makes it to the promotion conversation — the human review from module 3 still exists (we pick it up again in 05-02), but now it reviews only candidates that already cleared the automatic bar.

Everything above needs an engine that runs it on its own. This is the repo's actual workflow:

Block by block:

• on: — two triggers with different intent. push to working branches gives fast feedback while you develop; pull_request toward main is the gate with every control in place.
• actions/checkout@v4 — clones the repo onto the runner. Without this there's nothing to test.
• setup-python with cache: pip — installs Python 3.11 (the version pinned in module 2) and caches downloaded packages; the second run installs in seconds.
• pip install -r requirements.lock — we install from the pip-tools lock, not from loose dependencies: the runner reproduces exactly the production environment (scikit-learn 1.5.2, pandas 2.2.3, FastAPI 0.115.5). CI with floating versions is CI that lies.
• ruff + pytest -m "not model" — linter plus all the tests that need neither registry nor data. The model marker (declared in pyproject.toml) separates the cheap from the expensive.
• dvc pull with credentials in secrets — the credentials for the storage remote live in the GitHub repository's secrets (Settings → Secrets and variables → Actions), never in the YAML or the code. The workflow injects them as environment variables only in the step that needs them.
• dorny/paths-filter + label — the model tests only run if the PR touches src/** or if someone applies the model-tests label (useful for forcing them on a data or configuration PR). It's the mechanism that implements "the expensive stuff runs when it earns its keep".
• needs: — chains the jobs: if the linter fails, we don't spend time (or runner minutes) downloading data.

And one row is missing on purpose: building and publishing the Docker image. That's no longer integration but delivery, and it's exactly the topic of the next lesson.

• Installing in CI with a bare pip install pandas scikit-learn. The runner will resolve the versions of the day and your tests will validate an environment that isn't production's. Always from requirements.lock.
• Behavioral tests that are too strict. Demanding exact monotonicity from a RandomForest produces intermittent (flaky) tests, and a flaky test ends up ignored — which is worse than not having it. Use small tolerances and representative "typical" customers.
• Validating the data only in CI. The CI sample watches the contract, but the fresh production data gets validated in the pipeline (the prepare_data stage and, from 05-03 on, in the scoring flow). If you only validate the sample, on Monday you'll train on whatever happened to arrive.
• Putting the DVC credentials in the YAML "just to try it out". They stay in the Git history forever. GitHub Secrets from minute one, and rotate the key if it ever escaped.
• Metric thresholds against a test set that changes. If every run makes its own split, two candidates are never compared on the same ground. The test set is a DVC-versioned artifact, period.
• Turning CI into a 40-minute wall. If every push takes half an hour, people batch up enormous changes or look for ways around it. Protect the fast loop (pyramid levels 1-2 in under 5 minutes) and push the expensive stuff to PRs.
• Tip: when a bug reaches production despite CI, the right retrospective isn't "who did it?" but "which test was missing?". Add that test in the same fix. That's how you grow a suite that reflects the real failures of your system.

Marc proposes adding the feature days_since_last_ticket. Write two unit tests for it in the style of test_features.py: (a) a customer who has never opened a ticket (the source field last_ticket_date is null) must produce a sentinel value of 9999, not a null or an error; (b) the value can never be negative even if a future date arrives through a data error.

Extend customer_schema with two new checks: (a) a country column with allowed categories ["ES", "PT", "AD"] and no nulls; (b) a dataset-level expectation: no customer with active_discount = 1 may have plan = "basic" (the discount only applies to higher paid plans). Write only the schema fragment.

The model-tests job takes 8 minutes and the team complains that documentation PRs (changes only under docs/) wait for it too. Do they really wait, with the workflow written as it is? Reason out the answer by looking at the YAML and explain the role dorny/paths-filter plays.

Solution 1

The key to part (a) is the double assert: not failing isn't enough — you have to check the specific sentinel and the absence of nulls (a null here would break scikit-learn later, far from the problem's origin). In (b), the test pins the design decision (clip(lower=0)) just as we did with ratio_tickets.

Solution 2

and in the schema's checks list:

The second one is a cross-column business rule: it can't be expressed as a single-column check, which is why it goes in the DataFrame-level checks. This kind of rule is what detects logical corruption that types and ranges can't see.

Solution 3

They (almost) don't wait: the job starts, but dorny/paths-filter evaluates the files the PR touches and, since docs/** doesn't match the src/** filter, the steps.changes.outputs.src output is 'false'. All the heavy steps carry an if: on that output (and on the model-tests label), so they get skipped: the job finishes in seconds having done only the checkout and the filter. The real cost of a docs PR is the fast lint-and-tests job, which does always run — and that's correct: even a docs PR can break an import if it touches one file too many.

The CI of cineclick-churn no longer depends on anyone's memory: every push runs the linter and the code tests in minutes; every PR toward main additionally validates the data contract with pandera over a versioned sample; and when the change touches src/, the candidate proves that it trains (smoke), that it reasons (invariance with respect to customer_id, directionality of support_tickets), and that it performs (recall ≥ 0.60 and precision ≥ 0.50 against the versioned test set — module 3's business criterion turned into code that blocks merges). The complete pyramid, with the cheap stuff always running and the expensive stuff running when it earns its keep. But notice where it ends: at a green tick. The validated code still doesn't reach production on its own — the image still gets built on a laptop and the kubectl apply still gets typed by hand. The next lesson closes that gap with continuous delivery, and with a peculiarity that is pure ML: at CineClick there isn't one deployment pipeline, there are two — one for the service and one for the model — and confusing them gets expensive.