We closed module 9 with an uncomfortable diagnosis: the Rutas Norte platform now knows how to scale on its own, survive the May bank-holiday weekend and protect itself during maintenance windows, but the k8s/ directory has grown into more than 120 YAML files duplicated across dev, pre and pro that somebody applies by hand from their laptop. Module 10 is about exactly that: the tools that surround Kubernetes and turn a pile of loose manifests into an operable system.

We start with the link closest to the development team: the local cluster. Before templating anything with Helm (10-03), overlaying anything with Kustomize (10-04) or reconciling anything with Argo CD (10-05), you need a cheap, disposable place to try changes out. In module 1 we brought up the practice cluster with minikube start --profile=rutas-norte almost as a formality, and we said we would come back to it. The moment has arrived: in this lesson we go deep into minikube, meet kind (the tool of choice in continuous integration), compare the five most common alternatives and, above all, learn what can and cannot be validated on a local cluster, which is the root of the classic "it works on my machine".

1. Why every team needs a disposable cluster
2. Minikube in depth: profiles and drivers
3. Sizing, Kubernetes version and multi-node clusters
4. The addon catalogue
5. Working with local images without a registry
6. Cluster access: tunnel, service, dashboard, mount and ssh
7. The life cycle: stop, start, delete and logs
8. kind in depth: nodes as containers
9. Comparison table: minikube, kind, k3d, Docker Desktop and Rancher Desktop
10. The differences from production that cause "it works on my machine"
11. A reproducible recipe: the complete Rutas Norte environment locally
12. Common mistakes and tips
13. Exercises
14. Conclusion

At Rutas Norte S.L. there are three real environments: rutas-norte-dev, rutas-norte-pre and rutas-norte-pro. All three live on shared clusters. That means that when a developer wants to check whether her new NetworkPolicy breaks the connection between bookings-api and bookings-postgres, she has two bad options:

• Apply it in rutas-norte-dev and risk blocking the work of the other six colleagues who share that namespace.
• Not test it at all and find out in pre, with the waiting cycle that implies.

A local cluster resolves the dilemma. It is hers, free, quick to recreate and, if she breaks it, minikube delete and back to square one in three minutes.

The practical rule: the local cluster validates the correctness of your manifests and the logic of your application; it does not validate performance, topology or cloud integration.

minikube is a tool that brings up a single-node (or multi-node) Kubernetes cluster inside a virtual machine or a container on your own equipment. We have been using it since module 1; now we are going to understand what it does under the bonnet.

A profile is an independent cluster with its own name, its own configuration and its own kubectl context. It is the reason why in module 1 we wrote --profile=rutas-norte and not simply minikube start.

Typical output:

Without --profile, minikube uses the minikube profile. You can pin the active profile so you do not have to repeat it on every command:

Tip: pinning the profile is convenient but dangerous, because it is invisible state. In the Rutas Norte team's scripts we always use an explicit --profile=rutas-norte.

Each profile also creates its kubectl context with the same name:

The driver decides where the minikube node runs. It is the most important decision you make at the start.

How to choose, in three questions:

1. Do you have Docker or Podman working? Use --driver=docker (or podman). It is the fastest and the least memory-hungry, because there is no full VM in the middle.
2. Do you need to test kernel modules, a demanding CNI or real iptables? Use kvm2 on Linux. A container shares the kernel with the host and some things simply cannot be isolated.
3. Are you on Windows without WSL2, or in a restricted corporate environment? hyperv or virtualbox are the way out.

With the docker driver, the node is literally a container. You can see it:

That kicbase is the image containing systemd, containerd, the kubelet and the tooling needed for a container to behave like a node.

By default minikube asks for 2 CPUs and 2 GB of memory. For Rutas Norte that falls short as soon as we install Prometheus (07-03) and cert-manager (04-05):

What each option does:

• --cpus=4: number of virtual CPUs assigned to the node. With fewer than 4, the kube-apiserver and Prometheus compete and the cluster feels sluggish.
• --memory=8192: memory in MiB (8 GB). You can also write 8g. Careful: this memory is reserved away from your system; if your laptop has 16 GB, do not give it 12.
• --disk-size=40g: the node's disk size. Container images pile up fast; 20 GB run out within a couple of weeks of real work.

These values are remembered per profile. If tomorrow you run minikube start --profile=rutas-norte with no flags, it honours what you already configured. To change them you have to delete and recreate the profile (or use minikube config set beforehand).

An important warning: --cpus and --memory cannot be changed on the fly with the VM driver. If you come up short, minikube delete --profile=rutas-norte and create it again with the new values. That is why it pays to keep the creation script under version control (section 11).

This is one of the most profitable and most forgotten practices. If rutas-norte-pro runs Kubernetes 1.30 and your minikube runs 1.32, you may write manifests that work on your laptop and are rejected in production, or the other way round: you may fail to spot a deprecated API.

Output:

Rutas Norte platform team rule: the local start-up script file declares the same minor version as production. When pro moves up a version, the script is updated in the same change.

A single node makes it impossible to test anything related to scheduling: affinities (06-05), topologySpreadConstraints (09-05), DaemonSets with several targets (06-02) or draining a node. minikube knows how to create several:

Note that --cpus and --memory apply per node: three nodes of 2 CPUs and 4 GB consume 6 CPUs and 12 GB of the host. It is easy to choke your laptop without noticing.

You can add or remove nodes afterwards:

And label the nodes to simulate zones and test the topology spread from 09-05:

This is a useful simulation to verify that the selector works, but remember: they are still on the same physical machine, so it does not test real tolerance to a zone going down.

Addons are pre-packaged components that minikube knows how to install and maintain for you. They are what saves you from applying the Ingress controller or metrics-server manifests by hand.

The ones this course uses, and why:

Internally each addon is a set of manifests that minikube applies in the kube-system namespace (or one of its own). You can look at them:

Important notice: minikube's ingress addon installs ingress-nginx with a configuration designed for a single node. In production, your platform team probably installs it with Helm and with very different values (replicas, PDB, resources). The behaviour of the rules is the same; the behaviour of availability is not.

In day-to-day practice, this is where most time gets lost. You have built bookings-api:dev-local on your laptop and the pod sits in ImagePullBackOff, because the minikube node has its own image store, separate from the Docker on your machine.

There are three ways to solve it.

In the manifest, the key is to stop Kubernetes from trying to pull it from the registry:

Classic trap: if the image tag is latest, Kubernetes applies imagePullPolicy: Always by default and will always try to pull it, ignoring the local copy. Never use latest (we already saw this in 08-05); here it additionally breaks your local workflow.

If there are several nodes, image load copies it to all of them automatically.

This option repoints your Docker client at the daemon running inside the minikube node. Whatever you build is immediately available to the kubelet, with no copy step.

Advantage: it is faster in the edit-build-test loop, because there is no copy. Disadvantage: it only works with the docker runtime; if the profile uses containerd, it does not apply. And it is terminal state that is easy to forget (you have built inside the node and then cannot find the image on your machine).

This brings up a registry inside the cluster. It is the option closest to production, but also the one demanding the most steps (you have to push the image, resolve the name from the node). For day-to-day work, option A wins nearly every time.

When you declare a Service of type LoadBalancer (04-02), Kubernetes asks a cloud provider controller to provision a balancer. On your laptop there is no cloud provider, so the Service stays in <pending> for ever:

minikube tunnel simulates that balancer: it creates routes on your machine and assigns an external IP.

Now, in another terminal:

Points that confuse everybody the first time:

• The process must stay alive. If you close the terminal, the IP goes back to <pending>.
• It is not a real balancer: there are no provider health checks, no distribution across zones, no cloud firewall rules. It only routes.
• On macOS and Windows with the docker driver, tunnel is also needed to reach the Ingress.

For NodePort Services (or even ClusterIP ones with --url), this is the quick route:

It enables the dashboard addon if it was not already on, brings up a proxy and opens the browser. Useful for exploring the cluster visually while you are learning; in professional day-to-day work kubectl and k9s are faster.

Sometimes you want the pod to see a directory from your laptop: an SQL dump of test data for bookings-postgres, or the static files for web-store.

In the pod you use a hostPath (05-01) pointing at /test-data, which now exists inside the node:

Warning: hostPath is forbidden by the restricted level of the Pod Security Standards (08-03). Locally it is acceptable for test data; in rutas-norte-pro it would be rejected, and rightly so.

It drops you inside the node. Useful for:

On a multi-node cluster, pick the node:

--purge additionally removes the ~/.minikube directory with the configuration and the Kubernetes image cache. Use it when minikube is in a strange state you cannot explain; it is the equivalent of turning it off and on again.

When minikube start fails or the cluster comes up only halfway:

That kind of message almost always means the node ran out of resources or the container's network state was corrupted after an abrupt shutdown. minikube delete && minikube start fixes it, and that is exactly why a local cluster must be disposable.

kind (Kubernetes IN Docker) takes a different approach and a narrower philosophy: every cluster node is a Docker container running containerd and the kubelet. There are no addons, no tunnel, no dashboard. Just Kubernetes, very fast and very reproducible.

Precisely because of that austerity, it is the tool the Kubernetes project itself uses for its tests and the usual choice in continuous integration.

It takes about 25 seconds. The context is called kind-rutas-norte (with the kind- prefix).

Here is where kind's real power lies: the cluster is declared in a file you can version.

Let us go over the key fields:

• extraPortMappings: publishes a port of the node-container on your machine. It is kind's answer to the external access problem. With hostPort: 80, once ingress-nginx is installed, curl -H "Host: www.rutasnorte.example" http://localhost reaches web-store. It can only be defined when creating the cluster: if you forget it, you have to recreate it.
• extraMounts: the equivalent of minikube mount, but declarative and with no foreground process. Far more convenient.
• kubeadmConfigPatches: kind uses kubeadm internally (we will see it in 10-02). This field lets you modify its configuration; here we use it to set the ingress-ready=true label that the ingress-nginx manifest for kind expects in its nodeSelector.
• image: pins the Kubernetes version. Just as with minikube, match it to production.
• labels: node labels applied from the outset, with no subsequent kubectl label.

Now it works:

Just as in minikube, the image is copied to all the nodes. And here too you need imagePullPolicy: IfNotPresent and to avoid the latest tag.

You can also load an exported image file, handy in pipelines where the build and the load happen in separate steps:

A typical step in the ci-rutasnorte pipeline boils down to: kind create cluster --config kind-ci.yaml --wait 120s, build and load the image, kubectl apply -k k8s/environments/dev (that is Kustomize, we will see it in 10-04), wait for the rollout, run the smoke tests and kind delete cluster at the end whatever happens.

Notes to help you choose:

• k3s (the engine behind k3d and Rancher Desktop) is a lightweight, conformant distribution, but it swaps out some pieces: it uses Traefik as its Ingress, SQLite instead of etcd by default and disables cloud components. It is perfect for developing; it is misleading if you want to check fine-grained control plane behaviour.
• Docker Desktop with Kubernetes enabled is the most convenient thing if you already have it, but it is a single node, not configurable, and its licence is paid for companies above a certain size. Rutas Norte S.L. is small, so it is unaffected, but it is a real factor in team decisions.
• This course's recommendation: minikube for daily work (because of the addons, which save a lot of manual installation) and kind in the ci-rutasnorte pipeline (for speed and reproducibility). Having two different tools is not a problem as long as the manifest you deploy is the same.

This section is the most important of the lesson. Every point is a real incident waiting to happen.

Locally, type: LoadBalancer does nothing without minikube tunnel. In rutas-norte-pro, that same Service provisions a real balancer, with its public IP, its monthly cost and its provisioning time of several minutes.

Consequences you only see in the cloud: provider-specific annotations (balancer type, certificates, access lists), the provider's own health checks, and the fact that deleting the Service deletes the balancer. We will see this in 10-06.

Mitigation: locally, use Ingress (04-04) and not LoadBalancer in the application manifests. Leave LoadBalancer for the Ingress controller alone, which locally is published via NodePort or extraPortMappings.

In minikube, the standard StorageClass is a node directory served by storage-provisioner. In rutas-norte-pro, bookings-postgres uses a network-disk StorageClass with provisioned IOPS (05-04).

That last point is treacherous: in production the PV is not created until the pod is scheduled, and that changes the order of events. A StatefulSet that starts fine locally can end up Pending in the cloud because of an incompatible zone affinity.

This is the most dangerous one because it fails silently. You write a NetworkPolicy isolating bookings-postgres so that only bookings-api can talk to it (04-06), you apply it on your minikube, the object is created without error... and it does absolutely nothing, because the default CNI does not implement it.

It looks like it works. But:

It connects. The policy was ignored.

How to detect it: kubectl get pods -n kube-system -o wide | grep -Ei 'calico|cilium|flannel|kindnet'. kindnet (kind's default) and the basic bridge do not implement NetworkPolicies; calico and cilium do.

Mitigation: on minikube, --cni=calico. On kind, disableDefaultCNI: true with podSubnet: "192.168.0.0/16" and applying the Calico manifest afterwards.

And the golden rule: a NetworkPolicy is not tested until you have verified that something which used to connect no longer connects. Testing only that what is allowed still works proves nothing.

topologySpreadConstraints (09-05) with topologyKey: topology.kubernetes.io/zone on a single-node cluster is satisfied trivially. Labelling the nodes of a multi-node cluster simulates zones and verifies that the selector and the labels are correct, but it does not test fault tolerance.

The same goes for the Cluster Autoscaler (09-03): locally there is no elastic node group, so a pod that is Pending for lack of resources stays Pending for ever. In production a new node would appear within two minutes. The local symptom is identical to a serious production failure, which is misleading.

The worst thing about a local environment is that everyone on the team has a different one. The solution is a script versioned in the repository, alongside the manifests.

The reasoning behind the script's decisions:

• set -euo pipefail: aborts on the first command that fails, instead of carrying on and leaving a half-built cluster.
• The Kubernetes version in a variable at the top: when pro moves up a version, you change one line.
• --cni=calico: the difference between genuinely testing NetworkPolicies and fooling yourself.
• --dry-run=client -o yaml | kubectl apply -f -: the idempotent pattern from 01-06; it creates the namespace if it does not exist and does not fail if it does.
• kubectl wait: the script does not finish until the cluster is genuinely usable. Without this, whoever runs it tries to deploy and fails because the Ingress is not up yet.
• The closing note about /etc/hosts: the domains www.rutasnorte.example and api.rutasnorte.example do not exist in any DNS; they have to be resolved by hand against the node's IP.

The kind equivalent is even shorter, because all the cluster configuration is already in kind-rutas-norte.yaml: kind create cluster --config ... --wait 180s is enough, then apply Calico and ingress-nginx, wait for the controller and load the images with kind load docker-image. And there is no need to touch /etc/hosts: thanks to extraPortMappings, the Ingress responds at http://localhost with the right Host header.

1. Giving minikube nearly all the laptop's memory. With 16 GB physical, --memory=12288 leaves the operating system no headroom and everything stutters. Rule: no more than half the physical RAM, and remember that in multi-node the value is multiplied by the number of nodes.

2. Forgetting that --cpus and --memory are recorded in the profile. You run minikube start --profile=rutas-norte --memory=8192 one day, and the next day minikube start --profile=rutas-norte with no flags: it still uses 8192. If you think you changed the value and nothing changed, this is why. To change it you have to recreate the profile.

3. ImagePullBackOff with an image you "definitely built". It is almost always one of two causes: you did not run minikube image load / kind load docker-image, or the tag is latest and the implicit imagePullPolicy is Always. Use explicit tags (dev-local) and imagePullPolicy: IfNotPresent.

4. Believing an applied NetworkPolicy is working. Always check the CNI and run the negative test: attempt the connection that is meant to be blocked and verify that it times out. Without that test, you have tested nothing.

5. Closing the minikube tunnel or minikube mount terminal. Both are foreground processes. If the external IP or the mounted directory disappears, check whether the process is still alive before hunting for exotic causes.

6. Discovering too late that an extraPortMapping was missing in kind. They cannot be added to an already-created cluster. Have the configuration file with ports 80, 443 and perhaps a couple more from the very beginning.

7. Using latest for the node image. Both kindest/node and the minikube version must be pinned. A colleague on v1.31 and you on v1.29 produce different results from the same manifests.

8. Accumulating forgotten profiles. Every stopped profile still takes up disk. Run minikube profile list now and then and minikube delete --profile=X for whatever you are not using.

9. Treating the local cluster as if it were precious. If you suspect something is in a strange state, delete it and recreate it. The script in section 11 makes that cost three minutes; investigating a corrupted state costs an afternoon.

10. Testing performance locally. A benchmark of bookings-postgres on your SSD says nothing about the network disk in production. That is what rutas-norte-pre and k6 (09-06) are for.

Create a minikube profile called rutas-norte-topology with 3 nodes, Kubernetes v1.30.4 and Calico as the CNI. Label the two worker nodes as zone-a and zone-b. Deploy bookings-api with 4 replicas and a topologySpreadConstraint that spreads across zones with maxSkew: 1, and check with kubectl get pods -o wide that the split is 2 and 2.

Create two kind clusters: one with the default CNI (kindnet) and another with disableDefaultCNI: true and Calico installed. In both, apply a deny-all ingress NetworkPolicy in the rutas-norte-dev namespace and check, using a busybox pod trying to connect to a Service, which of the two genuinely honours it.

With the rutas-norte profile running, build an image registry.rutasnorte.example/web-store:dev-local, load it into minikube and deploy it with an Ingress for www.rutasnorte.example. Then change the page content, rebuild with the same tag, reload the image and get the pod to serve the new content. Explain why a plain kubectl apply is not enough.

Solution 1

You should get 2 pods on -m02 and 2 on -m03. The control plane node receives no pods because it carries the corresponding taint (06-05).

Solution 2

On no-netpol it returns nginx's HTML: the policy was ignored. On with-netpol it times out: Calico enforces it.

Solution 3

Why kubectl apply is not enough: the manifest has not changed (same image tag), so the object in the API is identical and the Deployment does not generate a new ReplicaSet. rollout restart adds an annotation with a timestamp to the pod template, which does trigger the rollout. This is exactly why production uses immutable tags with a digest (08-05): the new image has a new identifier and the change propagates on its own.

You now know how to build and destroy local clusters with judgement. The points worth taking away:

• minikube shines thanks to its addons: ingress, metrics-server, storage-provisioner and csi-hostpath-driver save you installing half a platform by hand. Profiles let you keep several clusters at once, --kubernetes-version aligns your environment with production, and --nodes opens the door to testing scheduling and topology.
• kind wins on speed and reproducibility: the nodes are containers, the whole cluster is declared in a versioned file with extraPortMappings and extraMounts, and that is why it is the natural choice for the ci-rutasnorte pipeline.
• The local image workflow is solved with minikube image load or kind load docker-image, always with imagePullPolicy: IfNotPresent and tags that are not latest.
• And most importantly: a local cluster validates correctness, not realism. There is no cloud balancer, the StorageClass is a different one, the default CNI may silently ignore your NetworkPolicies, and there are no zones and no node scaling. Knowing exactly what you are not testing is what prevents "it works on my machine".

One question remains open: local clusters are controlled toys, but how do you build a real cluster, on your own machines, without a cloud provider doing it for you? In the next lesson, Kubeadm, we will look at the official tool for bootstrapping a conformant cluster: preparing the machines, installing containerd, running kubeadm init with a configuration file, joining nodes, building a highly available control plane, renewing certificates and upgrading versions. And we will discover why kind, on the inside, is nothing more than kubeadm inside a container.