Throughout the course you have used NumPy, pandas, scikit-learn, XGBoost, LightGBM and Keras without stopping to look at the full map: each tool showed up when you needed it. Before taking the MercaFresh churn model to production, it is worth taking that pause. In this lesson we organize the Python machine learning ecosystem into layers, compare the options that compete with each other (TensorFlow versus PyTorch, the three boosting libraries), give you sound criteria for choosing, and, above all, learn how to pin and reproduce the working environment — because a model that only works "on my machine" cannot be deployed, and that is exactly the job of the next lesson.

1. The Python ML ecosystem, layer by layer
2. The scientific base layer: NumPy, pandas, SciPy and matplotlib
3. Classical ML: scikit-learn and statsmodels
4. Boosting: XGBoost, LightGBM and CatBoost
5. Deep learning: TensorFlow/Keras versus PyTorch
6. NLP and computer vision: transformers, spaCy and OpenCV
7. Summary table: which library for which task
8. Criteria for choosing a library
9. Environment management: the foundation of reproducibility

Python dominates machine learning not because of the language itself, but because of its ecosystem: hundreds of libraries built on top of one another, forming layers. Thinking of them as layers avoids two typical mistakes: believing you have to learn them all, and not knowing which one to reach for when a new problem appears.

Reading the diagram from the bottom up:

• Layer 1 — scientific base: arrays, tables, statistics and plots. Everything else is built on top of it.
• Layer 2 — classical ML: the algorithms from modules 4 and 5, with scikit-learn as the de facto standard.
• Layer 3 — boosting: independent libraries specialized in gradient boosting (module 7), which mimic the scikit-learn API so they can plug into it.
• Layer 4 — deep learning: deep neural networks (lesson 07-04), with their own compute engine (GPU, automatic differentiation).
• Layer 5 — domains: libraries for text and images that package ready-to-use deep learning models.

For the MercaFresh churn model — tabular data, tens of thousands of customers — layers 1 to 3 are enough. Layers 4 and 5 come into play with unstructured data, as you will see in the image classification (09-02) and sentiment analysis (09-03) projects.

You have used them throughout the course; here we simply pin down the role each one plays:

• NumPy: the n-dimensional array and vectorized operations. It is the lingua franca: when scikit-learn, XGBoost or Keras receive data, underneath they are NumPy arrays. Its speed comes from the fact that the loop runs in C, not in Python.
• pandas: the DataFrame, a table with typed, labeled columns. It is the tool of the cleaning and exploration phases (module 3): loading CSVs, grouping, joining tables, computing each customer's RFM.
• SciPy: statistics and engineering on top of NumPy. You used it in module 2 (distributions, hypothesis tests with scipy.stats) and you will reuse it in lesson 08-03 to detect drift with the Kolmogorov-Smirnov test.
• matplotlib (with seaborn on top): visualization. Histograms, ROC curves, confusion matrices — every plot in this course comes from here.

One practical consequence: when a higher-layer library fails with a strange type or dimension error, the cause usually lives in this layer (a DataFrame with an unexpected object column, an array with NaN). Knowing how to drop down to the base layer to debug is a skill, not a failure.

Scikit-learn is the library you have used the most, and it is worth making explicit why it feels so comfortable: the entire library shares three verbs.

This consistency is what makes possible the Pipeline you built in module 3: since every object responds to the same verbs, you can chain interchangeable pieces, and GridSearchCV (07-05) can optimize any combination without special code.

Switching from logistic regression to Random Forest means changing one line. That uniformity — which other ecosystems lack — is why scikit-learn is the recommended starting point for almost any tabular problem.

Scikit-learn answers "what does the model predict?"; statsmodels answers "what does the model say about the data?": per-coefficient p-values, confidence intervals, hypothesis tests (the whole apparatus of module 2). If the question at MercaFresh were "does the number of delivery incidents have a statistically significant effect on churn, controlling for tenure?", statsmodels is the right tool; for predicting which customers will leave, scikit-learn. Prediction and inference are different goals, and each library is optimized for its own.

In lesson 07-03 you saw that gradient boosting dominates tabular problems, and you worked with XGBoost and LightGBM. The three main libraries implement the same idea with different emphases:

All three offer a scikit-learn-compatible interface (XGBClassifier, LGBMClassifier, CatBoostClassifier with fit/predict), so they fit into the churn pipeline without touching the rest of the code. In practice, on medium-sized tabular data all three perform similarly when well tuned; the choice usually comes down to training speed and how comfortable each one is with your specific data.

For deep networks (07-04) there are two dominant ecosystems, and an honest comparison is in order:

Which one to choose? For learning and for standard models, Keras (as you did in 07-04) remains the fastest path from idea to trained model. If you plan to read and implement recent papers, or want fine-grained control over training, PyTorch is the majority choice today. The good news: the concepts (layers, activation functions, gradient descent, epochs) are identical; switching frameworks is a matter of days, not months. It is neither an irreversible decision nor the most important one in your project.

A brief introduction — you will really use them in module 9:

• Hugging Face transformers: the entry point to pretrained language models (BERT, and the families you saw at the end of 07-04) and vision models. Its pipeline("sentiment-analysis") pattern gives you a working sentiment classifier in three lines; it will be central to project 09-03.
• spaCy: classic "industrial" NLP: tokenization, named entities, lemmatization. Fast and built for production; ideal for preprocessing text before vectorizing it.
• OpenCV: image processing (reading, resizing, filtering, contour detection). It is usually the preparation layer before a CNN; it will appear in project 09-02.

AutoML deserves a separate mention (auto-sklearn, FLAML, and cloud services): tools that automate the model and hyperparameter selection you did by hand in 07-05. Useful as a quick baseline, but no substitute for understanding what is going on — everything they automate is exactly what you learned in modules 6 and 7.

When faced with a shiny new library, apply three filters before adopting it:

1. Maturity and maintenance: does it have years of stable releases, well-kept documentation, and recent commits? An abandoned library is technical debt: it will end up tied to old versions of NumPy or Python.
2. Community: are there answers on Stack Overflow, attended issues, third-party tutorials? When something breaks at eleven at night, the community is your technical support.
3. Real need: does it solve a problem your current tools do not, or is it just novelty? For MercaFresh's churn, swapping scikit-learn + LightGBM for the framework of the moment would improve nothing and add risk. The boring, proven tool is usually the professional decision.

An important corollary for the next lesson: every library you add is a dependency you will have to install, version and maintain in production. The bill for a dependency is not paid when you install it, but when you deploy and maintain it for years.

Here we move from the libraries themselves to how to manage them, and it is the most important part of this lesson with deployment in mind. The problem: you train the model with scikit-learn 1.5 on your laptop; the server has 1.2; the model loads incorrectly or behaves differently. This kind of silent failure is one of the most common causes of production errors.

The solution has two pieces:

1. Isolated virtual environments — each project with its own versions, without polluting the system Python or other projects:

2. Versions pinned in writing — a requirements.txt that declares exactly what the project needs:

• pip freeze > requirements.txt dumps the exact versions of the current environment.
• pip install -r requirements.txt rebuilds the same environment on another machine.
• The == (pinning the exact version) is deliberate: in production you do not want tomorrow's pip install to pull a different version than today's. Reproducibility is worth more than being on the latest release.

This file is a contract: "the model was trained with exactly this". In the next lesson we will use it twice: to load the serialized model with the same versions it was saved with, and as a central piece of the Dockerfile.

• Collecting frameworks instead of mastering one. Hopping from library to library produces shallow knowledge. Master scikit-learn in depth: its concepts (pipeline, CV, metrics) transfer to everything else.
• Choosing deep learning for tabular data by default. For problems like churn, tree-based boosting usually matches or beats neural networks at far lower cost (you saw this in module 7). Choose the layer of the diagram the problem calls for, not the highest one.
• Working without a virtual environment. Installing everything into the system Python ends in version conflicts between projects. One environment per project, always, from day one.
• A requirements.txt without versions (bare scikit-learn). It works today and fails six months from now, when a new version changes some behavior. Pin exact versions for everything that touches the model.
• Confusing statsmodels and scikit-learn. If you need p-values and confidence intervals, scikit-learn will not give them to you (that is not its goal); if you need pipelines and prediction at scale, statsmodels is not the way. They are complementary.
• Tip: before adopting a new dependency, look at its repository: date of the last commit, number of open unanswered issues, and whether it has numbered stable releases. Five minutes that save months.

Exercise 1. For each of these three assignments at MercaFresh, place the tools it would need in their ecosystem layer, and name the specific library you would use: (a) predicting churn from tabular RFM data; (b) automatically classifying the photos customers upload with their complaints ("damaged product" / "correct product"); (c) determining whether the effect of average delivery delay on churn is statistically significant.

Exercise 2. A colleague proposes rewriting the churn model (currently scikit-learn + LightGBM) with a deep learning framework that appeared eight months ago, "because it is the future". Write a three-point assessment using this lesson's selection criteria, and a final recommendation.

Exercise 3. On your machine, create a new virtual environment, install scikit-learn and joblib, and generate its requirements.txt with pinned versions. Then write the commands a colleague would run to reproduce your environment from that file.

Solution 1. (a) Layers 1–3: pandas for the data, scikit-learn for the pipeline and LightGBM (or XGBoost) as the model. No need to go higher: it is a classic tabular problem. (b) Layers 1, 4 and 5: OpenCV to read and prepare the images, and a CNN with Keras or PyTorch — or better, a pretrained vision model via Hugging Face, fine-tuned to the two classes. (This is developed in project 09-02.) (c) Layer 2, statsmodels: the question is one of inference (significance of a coefficient controlling for other variables), not prediction. A p-value and its confidence interval answer it; predict does not.

Solution 2. (1) Maturity: eight months of existence means an unstable API, bugs yet to be discovered and abandonment risk; the churn model must live in production for years. (2) Community: scarce by definition — no accumulated answers or tutorials, every problem gets solved alone. (3) Real need: none — the problem is tabular, and in module 7 we confirmed that boosting is the right tool; there is no gap the new framework would fill. Recommendation: keep scikit-learn + LightGBM; at most, try the new framework in an isolated experiment with no path to production, and re-evaluate it once it matures.

Solution 3.

The key point is that pip freeze writes ==version for every package (including transitive dependencies, such as NumPy), so the rebuild is exact and not "whatever the latest version happens to be that day".

You now have the map: a shared scientific base (NumPy, pandas, SciPy, matplotlib), scikit-learn as the backbone of classical ML with its fit/predict/transform API, statsmodels for inference, the boosting trio to squeeze the most out of tabular data, two large deep learning ecosystems to choose between without drama, and the domain libraries waiting in module 9. And one cross-cutting rule: every dependency is declared, pinned and isolated in its environment, because reproducibility is not an academic virtue but a deployment requirement. With the MercaFresh churn model's environment frozen in a requirements.txt, we are ready for the module's central question: how to get that model out of the notebook and have it answer real requests. That is the next lesson: serialization, APIs, containers — deploying models to production.