We have the language (07-01), the base libraries (07-02) and the tool map (07-03). What is missing is what turns all of that into a workshop in which several people can work today and a year from now: the environment. In modules 3 to 6 we wrote loose scripts, with a novamarket_ml.py file from which we imported functions, without ever saying which version of Python or scikit-learn was needed, where each thing was stored or how to check that it still worked. This lesson puts that in order: virtual environments and dependency management (venv, pip, requirements.txt, conda, uv/poetry) and why pinning versions is part of reproducibility, together with seeds; Jupyter with its virtues and its traps (hidden state); Colab and Kaggle as laboratories with free GPU; the IDEs (VS Code, PyCharm); Git applied to AI projects (what gets versioned and what does not); a recommended project structure that we will actually apply: we will refactor NovaMarket's code into a src/novamarket/ package with data.py, train.py and tests with pytest, and run it; and, to close, when a GPU or the cloud is needed and what Docker is. It matters because the project Marta and Diego will tackle in module 8 does not fit in a notebook: it needs this structure to survive the first change of person, machine or version.

1. The problem: "it works on my machine"
2. Virtual environments with venv and dependencies with pip
3. requirements.txt, conda/mamba, uv and poetry
4. Reproducibility: versions + seeds
5. Jupyter Notebook and JupyterLab: cells, kernel and hidden state
6. Google Colab and Kaggle Notebooks
7. IDEs: VS Code and PyCharm
8. Git for AI projects: what to version and what not
9. Recommended project structure
10. Code: refactoring NovaMarket into src/novamarket/ with tests
11. GPU, CPU and cloud: when they are needed
12. Docker as a reproducible environment
13. Common Mistakes and Tips
14. Exercises
15. Conclusion

Marta trains the returns predictor on her laptop, gets AUC 0.844 and passes the script to a colleague. For him it fails on import, or gives a different number, or the joblib does not load. The causes are always the same: another version of Python or of a library, another execution order of the notebook cells, another seed or none, different data in a file with the same name. The development environment is the set of decisions that removes those causes one by one: isolate dependencies, pin versions, fix seeds, organise the code, version it, test it and, when necessary, package the whole machine.

A virtual environment is a folder with its own Python interpreter (linked to the system one) and its own collection of packages, isolated from the rest. Each project has its own, so project A can use one version of pandas and project B another without stepping on each other. venv ships with Python:

pip installs packages from PyPI (Python's public index) resolving dependencies: asking for scikit-learn brings in NumPy, SciPy, joblib and threadpoolctl. The golden rule: never install libraries in the system Python; always in a project environment. The course environment is exactly this: a venv with NumPy, pandas, scikit-learn, Matplotlib, PyTorch (CPU) and, from this lesson on, pytest.

So that someone else (or you a year from now) can recreate the environment, the list of packages with versions is written in a file:

Usual convention: a requirements.txt with what you decide (direct packages, pinned with == where it matters) and a lock file generated with freeze to reproduce to the bit. Alternatives you will see in teams:

An equivalent conda environment.yml, illustrative:

conda env create -f environment.yml and conda activate novamarket. Choose one tool per project and do not mix them.

A result is reproducible if someone else gets the same number. Two things are needed that we have kept separate in the course and now bring together:

• Seeds: np.random.default_rng(42), random_state=42 in scikit-learn, torch.manual_seed(42), and seeded shuffle in train_test_split and DataLoader. Without them, the 3,000 orders, the split and the network initialisation change on every run. (On GPU, some operations are non-deterministic even if you fix the seed; PyTorch documents how to force it at the cost of speed.)
• Versions: the same seed on another version of NumPy or scikit-learn can give another sequence or another numerical result, and a joblib saved with one version may not load with the next (07-01, 07-03). Hence the requirements.txt with == and hence, in section 10, the training script will save next to the model a JSON with the Python and scikit-learn versions it was trained with.

Reproducible = versioned code + identified data + pinned versions + fixed seeds. All four; with three, the number changes.

Jupyter is the interactive environment par excellence of data science: a document (.ipynb) with code and text (Markdown) cells, executed one by one against a kernel (a live Python process that holds the variables). JupyterLab is the modern interface with tabs, file explorer and terminal. It is launched with jupyter lab from the activated environment, and opens the browser. Its virtues are obvious: you see the head() and the plot next to the code, you iterate fast, you document while exploring. Its traps come from the same thing:

• Hidden state: the variables live in the kernel, not in the document. If you run cell 5, then cell 2, then edit and run cell 5 again, the notebook you see does not correspond to the kernel's state. The typical case: you delete the cell that defined orders and everything keeps working... until you restart, and then NameError. Another: you change test_size in cell 3 but do not re-run cell 4, and the AUC you see is the old one.
• Execution order: the numbers in brackets [7] to the left of each cell say the real order; if they do not go top to bottom, be suspicious.
• Good practices that avoid almost every upset: (1) before accepting a result, "Restart Kernel and Run All" (restart and run everything top to bottom; if it fails, the notebook was lying); (2) imports and parameters in the first cells; (3) do not leave scratch cells lying around out of order: delete them or move them to an appendix; (4) functions that get repeated go to a .py module of the project (section 9) and are imported, not copied from notebook to notebook; (5) export to a script when the code matures: jupyter nbconvert --to script exploration.ipynb generates exploration.py, or better, copy by hand what is worth keeping into the module; (6) clear outputs before pushing to Git (.ipynb files with outputs store images and tables and make diffs unreadable): jupyter nbconvert --clear-output --inplace, or tools like nbstripout.
• A notebook is for exploring and telling; the code that runs every day (training, serving) goes in modules and scripts. That is the border we will draw in section 10.

When you have no GPU, or want to share a notebook without anyone installing anything, there are two free laboratories in the browser:

• Google Colab: Jupyter notebooks hosted by Google, with Python and the usual libraries preinstalled, and access to a free GPU (and TPU) with session-time, hours-per-day and memory limits that change and that in the free version are not guaranteed; paid tiers extend the quota. The session is ephemeral: when it closes, whatever is on disk is lost, so data is uploaded in each session or read from Google Drive.
• Kaggle Notebooks: similar, integrated with Kaggle's datasets and competitions, with a weekly quota of GPU hours and many public notebooks to learn from.
• How to upload a CSV (description, not runnable here): in Colab, the side file panel has an upload button, or from code from google.colab import files; files.upload() opens a dialog; for large files, mount Drive with drive.mount('/content/drive') and read with pd.read_csv('/content/drive/MyDrive/orders.csv'). In Kaggle, "Add Data" adds a dataset (your own or a public one) that appears under /kaggle/input/.
• Cautions: do not upload personal data of real customers to an external service without the 02-04 assessment; pin versions because the preinstalled environment changes; and keep the notebook and the models outside the session (Drive, Git) because they disappear.

An editor with Python support multiplies productivity compared with the notebook for module code:

• Visual Studio Code with the Python and Jupyter extensions: autocompletion, code navigation, formatting, running notebooks inside the editor and "cells" in .py files with # %% (the best of both worlds: a versionable script that runs in chunks), a debugger with breakpoints (much better than scattering print to see why the ColumnTransformer gives 22 columns instead of 21), integrated terminal, integrated Git, and selection of the virtual environment's interpreter (Python: Select Interpreter → .venv).
• PyCharm: the JetBrains IDE, more "all included" (the Community edition is free; Professional adds scientific and notebook support). Excellent refactoring and debugging.
• Others: Spyder (MATLAB/RStudio-style interface, with conda), Positron, or any editor with a language server. The choice is personal; what is not optional is using the project environment's interpreter and having a debugger and tests at hand.

Git stores the history of the code and lets several people work without stepping on each other; GitHub/GitLab host it. In an AI project there is a peculiarity: data and models are large, change for reasons different from the code and are sometimes confidential. The rule:

For data and models, DVC (Data Version Control: it stores in Git a small file with the hash of the data and the data itself in external storage, S3, Drive, a network drive) or Git LFS (Large File Storage) are used. We only mention them: for NovaMarket we will start with .gitignore and a shared folder, and DVC when it is needed.

Minimal flow:

From the loose scripts of modules 3-6 to a project that someone else understands in a minute. Structure for NovaMarket:

Principles: data/raw/ is untouchable (what arrives is stored as is; everything else is recomputed with code); notebooks are numbered and tell a story; reusable code lives in the package src/novamarket/ (the src/ folder stops Python accidentally importing an uninstalled copy); models/ stores artefacts with their metrics next to them; tests/ checks what must not break; and README.md says how to get started. There are templates (Cookiecutter Data Science is the best known) that generate this with more folders (reports/, configs/, docs/); start small and add when you need it.

Let's actually build it. We create the structure, move the functions of novamarket_ml.py (modules 4-5) to src/novamarket/data.py, write train.py as a runnable script, a minimal test with pytest and run it all.

1. Create the structure (in a terminal, with the environment activated):

2. src/novamarket/__init__.py (turns the folder into a package):

3. src/novamarket/data.py: the functions you already know, unchanged, with a header. We reproduce the first one in full; dirty_orders, prepare_orders and build_preprocessing (with their lists NUMERIC, BINARY, NOMINAL, ORDINAL) are copied as they are from 04-03:

4. src/novamarket/train.py: importable functions and a runnable script, separated by if __name__ == "__main__":. Next to the model it saves a JSON with metric, parameters and versions (section 4):

if __name__ == "__main__": is the line that makes both things possible: when you run the file, __name__ is "__main__" and the block runs; when another module (or a test) does from novamarket.train import train, __name__ is "novamarket.train" and the block does not run. argparse turns --n 3000 into args.n, with default values and --help for free.

5. pyproject.toml: the standard project metadata file; here it also tells pytest where the code is:

pip install -e . (editable install) registers the package in the environment pointing at src/, so that import novamarket works from any folder and changes to the code are seen without reinstalling. Alternative without installing: PYTHONPATH=src python -m novamarket.train.

6. tests/test_data.py and tests/test_train.py: pytest discovers test_* functions and checks asserts. We test what must not break: reproducibility, shape, the 21 columns, absence of NaN, a minimum AUC and that saving writes both files:

7. requirements.txt, .gitignore and README.md as in sections 3, 8 and 9 (the .gitignore ignores .venv/, __pycache__/, .ipynb_checkpoints/, data/raw/* and data/processed/* except the .gitkeep files, models/*.joblib, models/*.pt and .env).

8. Run. Tests, training and Git check, with the real outputs:

Four tests in under a second, the same AUC 0.844 of 04-04 obtained with one command from the terminal, a JSON that says which versions it was trained with, and a repository into which the binary model does not go but its metrics do. This is the starting point of module 8: when in 08-01 we add predict.py (which loads models/returns_model.joblib as in 07-03) or change the model, the tests will say in a second whether something has broken.

The whole course has run on CPU, and that is the answer for most of what NovaMarket does: tables of thousands or hundreds of thousands of rows with scikit-learn, the 497-parameter MLP, the rules, the optimisation. The GPU is needed when the computation is large repeated matrix multiplications: training CNNs on real images, fine-tuning a transformer, running local LLMs; there it speeds things up 10 to 100 times. Checking for it and using it in PyTorch:

(On a Mac with Apple Silicon the equivalent is torch.backends.mps.is_available() and "mps".) Options, from lowest to highest cost: the laptop's CPU; Colab/Kaggle to try with a free GPU (section 6); a virtual machine with a GPU in the cloud (AWS, Azure, Google Cloud and specialised providers) paid by the hour and switched off when done; managed ML services from those providers; or your own GPU if usage is continuous. Diego's question ("is it needed?") is answered by measuring: if training on CPU takes minutes, no; if it takes days, yes, and even then it is better first to reduce data or model to iterate fast and use the GPU only for the final run.

The last step of reproducibility is packaging the whole machine: operating system, Python, versioned libraries, code and start-up command, into an image that runs the same on Marta's laptop, on NovaMarket's server and in the cloud. That is Docker (or Podman). An illustrative Dockerfile to serve the predictor with the 07-03 API:

docker build -t novamarket-returns . builds the image and docker run -p 8000:8000 novamarket-returns starts it; NovaMarket's website calls http://server:8000/return-risk. Docker does not replace venv in day-to-day development (it is heavier), but it is the standard way to deliver a service and to guarantee that "it works on my machine" means "on all of them". Kubernetes (07-01) orchestrates many containers; outside the scope of the course.

• Installing everything in the system Python. Sooner or later two projects ask for incompatible versions and something in the system stops working. One environment per project, always.
• requirements.txt without versions, or no requirements.txt. Six months later, pip install brings new versions and the joblib does not load or the number changes. Pin with == what matters and keep a freeze.
• Notebooks that only work in the order they were run. "Restart Kernel and Run All" before accepting anything; functions to modules; outputs cleared before pushing to Git.
• Data and models in Git. The repository swells until it is unusable and you may publish personal data. .gitignore from the first commit; DVC or external storage for the large stuff.
• Seeds without versions or versions without seeds. Both, and written down next to the model (the JSON of section 10).
• Tests that check the trivial and not the fragile. Test the shape of the matrix, the absence of NaN, reproducibility, a metric threshold; those are the failures that really happen when you touch prepare_orders.
• Buying a GPU before measuring. Time it on CPU, shrink the problem, try on Colab; then decide.
• Absolute paths in the code (/home/marta/orders.csv). Paths relative to the project root or configurable by argument; pathlib.Path instead of concatenating strings.

Exercise 1. Add to the project a module src/novamarket/predict.py with a function load(path="models/returns_model.joblib") and another predict_risk(model, order: dict) -> float that builds a one-row DataFrame and returns the return probability, plus an if __name__ == "__main__": block that predicts for an example order. Write a test that trains with train(n=1500), saves to tmp_path, loads with load and checks that predict_risk returns a number between 0 and 1 equal to pipe.predict_proba on the same order.

Exercise 2. Simulate the hidden-state problem: in a notebook (or mentally, describing the cells), define in cell 1 threshold = 0.5, in cell 2 a function that uses threshold, run cell 3 which calls it; then change cell 1 to threshold = 0.3 without re-running it and run cell 3 again. Which threshold is applied? What does the notebook show? What happens when you do "Restart and Run All"? Propose two changes to the notebook's organisation that avoid the problem.

Exercise 3. Write the .gitignore, the requirements.txt (with the versions you use) and an illustrative Dockerfile for a project that serves the incident-diagnosis Bayesian network of 06-03 as an API. Justify what goes into Git and what does not, and why in this case the "model" (the conditional probability tables) probably should be versioned in Git, unlike returns_model.joblib.

Solution 1.

Run in the course environment: pytest passes 5 tests, and python -m novamarket.predict prints return risk: 0.048 (the same order as in 07-03). With this the project has the three pieces of the cycle (data, train, predict) tested.

Solution 2. 0.5 is applied: cell 1 was edited but not run, so the kernel's threshold variable is still 0.5; the notebook shows threshold = 0.3 in cell 1 and a result computed with 0.5, that is, it lies. On "Restart and Run All" the kernel is emptied, cell 1 runs with 0.3 and the result changes; if someone had copied the previous result into a report, it would be irreproducible. Two changes: (1) parameters and functions at the top, and the rule of re-running from the top after any parameter change (or simply "Run All Above" before the results cell); (2) move the function out to src/novamarket/ with threshold as an explicit argument (decide(prob, threshold=0.5)) and call it with the value visible in the cell, so that there is no global variable to depend on.

Solution 3. .gitignore: .venv/, __pycache__/, .ipynb_checkpoints/, .env, and data/raw/* (the real incident histories, confidential); src/novamarket/incidents_network.py is not ignored, nor is a configs/incidents_cpt.json with the probability tables. requirements.txt: numpy==..., pandas==..., fastapi, uvicorn, pytest (and pgmpy==... if the enumeration of 06-03 is replaced by pgmpy, 07-03). Dockerfile: the same as in section 12, copying src/, configs/ and serve_incidents.py, with CMD ["uvicorn", "serve_incidents:app", ...]. Why the "model" does go into Git: the conditional probability tables of the 06-03 network are a few dozen numbers written or reviewed by people (explicit knowledge, like the rules of 06-02), small, readable as text and with audit value (who changed the probability of "factory defect" and when?): exactly what Git does well. returns_model.joblib is a binary generated by code from data: it is regenerated with train.py, it is not read or reviewed line by line, and its place is an artefact store with the metrics-and-versions JSON in Git.

With this lesson Marta's workshop is set up. We have seen why "it works on my machine" is the enemy and how it is fought: virtual environments (venv) with one interpreter and one set of packages per project; pinned dependencies in requirements.txt (or environment.yml, uv, poetry) and the reproducibility equation, versions + seeds + code + identified data; Jupyter to explore and tell, with the discipline of restarting and running everything and of moving mature code to modules; Colab and Kaggle as laboratories with free GPU and ephemeral sessions; VS Code and PyCharm with a debugger and the environment's interpreter; Git with a clear border between what is versioned (code, dependencies, small metrics) and what is not (data, large models, secrets), and DVC on the horizon; a project structure (data/, notebooks/, src/novamarket/, models/, tests/) that we have actually built, with data.py, train.py runnable from the terminal (AUC 0.844 and a JSON with versions), a pyproject.toml and four pytest tests that pass in a second; and the infrastructure decisions: CPU unless proven otherwise, GPU and cloud when the computation demands it, Docker to deliver the service the same everywhere.

And with this we close module 7. We started in 07-01 by choosing Python and SQL among the languages of AI; in 07-02 we learned to really handle NumPy, pandas and Matplotlib with NovaMarket's orders and demand; in 07-03 we placed each library in its task and saved and reloaded the returns predictor and the MLP; and in 07-04 we have organised it all into a reproducible project with environment, structure, Git and tests. Marta has the language, the tools and the workshop; Diego has a train.py that anyone can run and a repository that does not swell with data. What they do not have yet is a method to take a use case from idea to production: define the business problem, decide the metric, assemble the data, iterate on the model, validate it, deploy it and monitor it, learning from the projects that went well and from those that did not. That is module 8, Projects and Case Studies, which begins in 08-01, Developing an AI Project, with Marta and Diego tackling from start to finish, inside the novamarket_ai/ structure we have just created, the returns predictor project.