So far we have decided what runs and how each pod is composed, but never where. The kube-scheduler has been placing our pods on whichever nodes it fancied and we have not complained, because in a homogeneous test cluster any spot will do.

In production not any spot will do. bookings-postgres needs the node with the SSD, because on a spinning disk the occupancy queries take five times as long. The six replicas of bookings-api should not all be on the same node: if that node reboots, the whole API disappears just as somebody is paying for a ticket. And heavy analytics workloads should not compete for CPU with ticket sales on a bank-holiday weekend.

All of that is expressed with three mechanisms we will study here: affinity (the pod picks a node), anti-affinity (the pod avoids company) and taints with tolerations (the node turns pods away). By the end we will have the complete Rutas Norte scheduling plan and we will know how to diagnose the pod that stays Pending forever.

1. How the kube-scheduler decides: filtering and scoring
2. nodeName and nodeSelector: the basic mechanisms
3. Node affinity: required versus preferred
4. Affinity and anti-affinity between pods: topologyKey
5. Taints and tolerations: the node turns pods away
6. Kubernetes' automatic taints
7. PriorityClass and preemption
8. topologySpreadConstraints: a mention and a forward reference
9. The Rutas Norte scheduling plan
10. Diagnosing the eternally Pending pod

In lesson 01-02 we said the scheduler "assigns pods to nodes". Now we see how.

When a pod is created, its spec.nodeName field is empty. The scheduler watches those unassigned pods and, for each one, runs a two-phase cycle.

Nodes that cannot host the pod are discarded. Each check is an absolute veto: one is enough to eliminate the node.

A detail that connects with module 5: the VolumeBinding filter is what makes volumeBindingMode: WaitForFirstConsumer work. A PVC already bound to a volume in zone eu-west-1a removes every node in the other zones from the filtering.

And another that connects with module 3: NodeResourcesFit looks at requests, not actual usage. A node with 8 CPUs where the pods have reserved 7.8 CPUs but only use 0.5 counts as full. That is why inflated requests waste cluster capacity.

Among the viable nodes, each scoring plugin awards between 0 and 100 points, and a weighted average is taken.

The highest score wins; on a tie, one of the tied nodes is picked at random. The scheduler then writes the nodeName and that node's kubelet picks the pod up.

With this the essential difference between the two kinds of rule we will see is already clear: required rules act in the filtering (if they are not met, the pod does not go), and preferred rules act in the scoring (if they are not met, the pod goes anyway, but to a lower-scoring node).

It skips the scheduler entirely: that node's kubelet sees a pod with its name on it and starts it, no questions asked.

Never use it in production. If the node does not exist, is full or is down, the pod hangs with no useful event; if the node disappears, there is no rescheduling. Its only legitimate use is one-off debugging: forcing a pod onto a particular node to investigate a local problem.

The pod is only scheduled on nodes carrying all of those labels with those exact values. It is a conjunction of equalities, nothing more.

Labels are put on nodes just as on any other object:

Kubernetes also adds a number of standard labels that are worth knowing:

nodeSelector has three limits, and they motivate everything that follows:

1. Exact equality only. You cannot say "SSD or NVMe", or "any node that is not an analytics node".
2. It is mandatory. If no node matches, the pod stays Pending forever. There is no way to express a preference.
3. It says nothing about other pods. It cannot express "do not put me with my siblings".

Node affinity solves the first two limits: an expressive syntax and the ability to prefer rather than demand.

They read in two halves:

• requiredDuringScheduling / preferredDuringScheduling: whether the rule is mandatory (filtering) or desirable (scoring).
• IgnoredDuringExecution: what happens if the node's label changes while the pod is already running. The answer is: nothing. The pod stays where it is.

That IgnoredDuringExecution matters more than it looks. If you schedule bookings-postgres on a node with disk=ssd and somebody changes that label to disk=hdd, the pod does not move. Affinity is evaluated at scheduling time and never again. A RequiredDuringExecution was planned that would evict pods once the rule stopped holding, but it was never implemented. For eviction based on node conditions, the mechanism is the NoExecute effect of taints (section 5).

• nodeSelectorTerms is a list with OR semantics: one term being satisfied is enough.
• matchExpressions inside a term is a list with AND semantics: all of them must hold.

Available operators:

NotIn and DoesNotExist give you node anti-affinity: there is no separate object for it, it is expressed by negation.

A detail about Gt and Lt: they compare as integers, the values list must have exactly one element, and the node's label value must parse as an integer. cores Gt "15" means strictly greater than 15, that is, 16 or more.

Every preferred rule that is satisfied adds its weight to the node's score. With two rules of weight 80 and 20, a node satisfying both scores 100 in this plugin; one satisfying only the first scores 80. The weights are relative to each other: what matters is the ratio, not the absolute value.

A typical and highly recommended combination: required for what is a genuine requirement (CPU architecture, the presence of a local disk) and preferred for what is an optimisation (a particular zone, a machine tier). Putting into required what was really a preference is the leading cause of needless Pending pods.

Node affinity looks at node labels. Affinity between pods looks at which other pods are already on that node or in that zone. It is what lets you say "I want to be near the cache" or "I do not want to be with my sisters".

Read it as: "do not place me on a node (kubernetes.io/hostname) where there is already a pod with app=bookings-api and environment=pro".

topologyKey is a node label that defines the scope in which the rule applies. The scheduler groups nodes by the value of that label, and each group is a "topology domain".

Choosing the topologyKey well is choosing which failure you protect yourself against:

• hostname protects against a node rebooting or going down.
• zone protects against a whole data centre going down.
• region protects against a regional disaster, at the cost of latency between replicas.

Careful with required and hostname: if you demand at most one replica per node and you have 3 nodes, you will not be able to go beyond 3 replicas. The fourth will stay Pending indefinitely. This mistake often shows up combined with autoscaling that tries to go up to 10.

The usual solution is to use preferred, which spreads but does not block:

With weight: 100, the scheduler heavily penalises nodes that already host a replica, so it spreads them out while it can; when there is no alternative left, it places two together instead of leaving the pod Pending. For bookings-api, whose availability matters more than perfect spreading, it is the right choice.

Note the change of syntax between the two variants: in required the list contains the terms directly; in preferred each element is an object with weight and podAffinityTerm.

The symmetric case: notifications-worker queries redis-cache a lot and benefits from being on the same node, where the traffic does not cross the physical network.

Use podAffinity sparingly. Concentrating related pods on the same node improves latency and worsens availability: that node becomes a single point of failure for two components at once. In Rutas Norte we keep it as preferred with a medium weight and do not apply it to anything critical.

This is a practical warning that the official documentation underlines. To evaluate affinity between pods, the scheduler must, for every candidate node, examine the pods of every node in its topology domain. The complexity grows with the product of nodes and pods, not linearly.

On a 10-node cluster it is imperceptible. On one with several hundred, with inter-pod affinity rules on many workloads, the decision time per pod can go from milliseconds to seconds, and in a mass deployment that shows. The documentation expressly advises against these rules on clusters of more than a few hundred nodes.

Mitigations:

• Use topologyKey: kubernetes.io/hostname (small domains) rather than zone (large domains).
• Narrow the scope with namespaceSelector so that pods from other namespaces are not evaluated.
• For the specific case of "spread replicas evenly", topologySpreadConstraints is more efficient; we look at it in section 8.

Affinity and anti-affinity are mechanisms of the pod: the pod expresses where it wants to go. Taints are the inverse mechanism, belonging to the node: the node declares that it turns pods away unless they are expressly authorised.

The difference matters because it solves a problem affinity cannot: reserving a group of nodes. With affinity, to make sure only analytics workloads go to the analytics nodes, you would have to add a "do not go there" rule to every other pod in the cluster, including those somebody deploys tomorrow. With a taint, the node protects itself.

A taint has the form key=value:effect.

NoExecute is the only one of the three that acts on running pods. It is the mechanism that does what RequiredDuringExecution would have done in affinity.

Syntax rules:

• operator: Equal (the default value) requires key, value and effect to match.
• operator: Exists requires key and effect to match; it must not carry a value.
• Omitting effect means "any effect for that key".
• Omitting key with operator: Exists means "all taints". It is what the CNI DaemonSets that must run no matter what use (06-02).

It only makes sense with NoExecute, and it expresses how long the pod stays on the node before being evicted:

"If the node becomes unreachable, stay for 5 minutes in case it comes back; if not, evict me." It is exactly what Kubernetes adds by default to every pod, as we will see now.

A toleration does not attract the pod: it merely stops it from being turned away.

A pod with the dedicated=analytics toleration may go to the analytics nodes, but it may also go to any other node in the cluster, and it will probably end up on another one. To reserve nodes properly you need both pieces:

Taint + toleration = exclusivity. Taint + toleration + affinity = dedication.

Kubernetes applies taints of its own accord when a node is in trouble. Knowing them explains a lot of behaviour that otherwise looks like magic.

And the best known one, which is not automatic but applied by kubeadm when the cluster is created:

It is the one that forced the log DaemonSet of 06-02 to carry an explicit toleration.

Here is the real mechanism behind "if a node goes down, its pods are recreated somewhere else":

1. The kubelet stops sending heartbeats.
2. After 40 seconds, the node controller marks the node NotReady and adds the taint node.kubernetes.io/unreachable:NoExecute to it.
3. The node's pods are not evicted immediately, because Kubernetes automatically added this toleration when it created them:

4. After 300 seconds (5 minutes) the tolerance runs out and the pods are marked for deletion. Their controllers create replacements on other nodes.

That explains the roughly five-minute delay between a node going down and its pods reappearing somewhere else. It can be tuned per pod:

Lowering it too far has its risks: a transient 40-second network glitch would cause a mass eviction and a needless recreation. For bookings-postgres the opposite applies: a high value, because recreating the database on another node means unmounting and remounting a volume, and it is better to wait for the node to come back.

A warning: DaemonSet pods do not get these automatic tolerations with tolerationSeconds; instead they tolerate those taints indefinitely, so that the agents keep working on a node in trouble.

When the cluster is full, what happens if an important pod arrives? Without priorities, it stays Pending like any other. With priorities, it can evict less important pods to make room for itself.

Use in a pod:

PriorityClasses are cluster-scoped objects, not namespaced. Kubernetes ships two predefined ones for system components, system-cluster-critical (2000000000) and system-node-critical (2000001000); the latter is the one we used in the log DaemonSet of 06-02.

It acts at two moments:

1. In the scheduler's queue: pending pods are ordered by descending priority. A critical pod is evaluated before a batch one, even if it has been waiting less time.
2. In preemption: if a high-priority pod does not fit on any node, the scheduler looks for a node where, by evicting lower-priority pods, it would fit. If it finds one, it marks those pods for deletion (with their SIGTERM and their grace period from 02-01) and schedules the important pod in their place.

preemptionPolicy: Never makes a pod benefit from priority in the queue but never evict anyone. It is the right thing for batch jobs: let them be scheduled early if there is room, but let them knock nothing down.

It is a mechanism that degrades on its own if used badly:

• Priority inflation. Every team marks its workload as "critical". In the end everything is critical and priority stops discriminating. Define few classes (three are enough) and document what justifies each one.
• Cascading preemption. Pod A evicts B; B's controller recreates it on another node; there it evicts C… A cluster at its limit with badly distributed priorities can spend minutes shuffling pods without settling.
• Evictions do not respect PodDisruptionBudget absolutely. The scheduler tries to respect them, but if it finds no other option, it evicts anyway. PDBs are the subject of 09-05.
• Stateful workloads suffer a lot. Evicting bookings-postgres means closing connections, unmounting the volume and mounting it again on another node. Give it a high priority and do not give it preemptionPolicy: PreemptLowerPriority expecting it to reschedule itself neatly: what you want is for nobody to touch it.

Practical rule: use priorities to protect what is important from being evicted, not to make what is important evict aggressively.

There is a third mechanism, more modern and more efficient than anti-affinity for the specific case of spreading replicas evenly:

In one sentence: "the difference between the zone with the most bookings-api replicas and the one with the fewest must not exceed 1; if that cannot be met, schedule it anyway".

Advantages over podAntiAffinity:

Its full application to high availability — how to combine it with PodDisruptionBudgets to survive the loss of a zone — is content for lesson 09-05. Here it is enough to know that it exists and that, for the "spread my replicas" case, it is usually a better tool than anti-affinity.

Let us put it all together into a coherent plan for rutas-norte-pro.

Here required in the anti-affinity is the correct choice: two PostgreSQL instances on the same node add nothing and do introduce disk contention. And since the StatefulSet has one replica, there is no risk of blocking.

Two rules with different weights: spreading across nodes matters twice as much as spreading across zones. The result in a typical deployment:

Six replicas over four nodes, as spread out as possible. With required the sixth would have stayed Pending. web-store carries an equivalent configuration.

The CronJob from 06-03 runs heavy aggregate queries. Having it run on the dedicated node stops it from competing with ticket sales:

The two pieces together, as we explained in section 5: the toleration lets it in where nobody else can, and the affinity forces it to go there. The rutasnorte-batch priority with preemptionPolicy: Never guarantees that a nightly report will never evict anything.

We already wrote it in 06-02, and now it makes complete sense:

Without the second toleration, node m06 would have no collector and the logs of the nightly reports would go nowhere: exactly the logs you would want to consult when a report fails at three in the morning.

A Pending pod has found no node. The scheduler tells you exactly why, and learning to read that message is the most profitable skill in this lesson.

The first line gives the count: out of 6 nodes, 0 available. Then the breakdown by reason, and the numbers must add up to 6:

• 1 node(s) had untolerated taint {dedicated: analytics} → node m06 turned the pod away because it does not carry the toleration. If the pod was occupancy-reports, the toleration is missing.
• 2 node(s) didn't match Pod's node affinity/selector → two nodes do not satisfy the required affinity.
• 3 Insufficient cpu → three nodes do not have enough free CPU for the requests.

The preemption: block explains why priority did not help either: No preemption victims found means there are no lower-priority pods that, if evicted, would make room.

Methodological tip: always start with the scheduler's event. It is the only place where the system tells you, with a count per reason, what ruled each node out. Eyeballing nodes before reading it is a waste of time.

Confusing a toleration with affinity. It is the most widespread misunderstanding on the topic. A toleration only prevents rejection; it takes the pod nowhere. To dedicate nodes you need both things.

required where preferred would have done. Every required rule is an absolute filter that can leave a pod Pending forever. Before writing it, ask yourself: "if this is not met, would I rather the pod did not run?". If the answer is no, it belongs in preferred.

required anti-affinity with topologyKey: hostname and many replicas. It caps the replicas at the number of nodes. Combined with an HPA trying to go up to 15 during a bank-holiday peak, it produces Pending pods exactly when they are needed most.

Expecting a label change to move pods. IgnoredDuringExecution means exactly what it says. To move a pod you have to delete it, or use a NoExecute taint.

Using nodeName in production manifests. It skips the scheduler and with it all the resilience. If the node goes down, the pod is not rescheduled.

Setting operator: Exists with a value. The API rejects it. With Exists no value is specified.

Forgetting the hyphen when removing a taint. kubectl taint node m06 dedicated=analytics:NoSchedule adds it; with the trailing hyphen it removes it. Without the hyphen you end up with duplicate taints.

PriorityClass inflation. If everything is critical, nothing is. Three well-defined and documented classes are enough for almost any platform.

Preemption over stateful workloads. Evicting bookings-postgres means unmounting and remounting a volume on another node. Protect it with a high priority, but do not count on preemption rescheduling it cleanly.

Tip: kubectl get pods -o wide --sort-by=.spec.nodeName is the command for checking at a glance that the spread is what you expected.

Tip: test a rule before applying it in production. Apply the manifest in rutas-norte-dev, scale to the figure you would use in production and check that no replica ends up Pending. A badly calibrated required anti-affinity only shows up when the number of replicas exceeds the number of nodes.

Tip: document why each rule exists. A podAntiAffinity with no comment is indistinguishable from a copy-and-paste, and nobody will dare touch it a year from now. One line of comment with the reason saves a great deal.

On your minikube (profile rutas-norte, several nodes), label one node with disk=ssd and another with disk=hdd. In rutas-norte-dev create a Deployment db-demo with 2 replicas of nginx:1.27.2-alpine that:

• Requires (required) running on nodes with disk in [ssd, nvme].
• Prefers (preferred, weight 70) nodes with tier=high.

Check where they are placed. Then scale to 4 replicas and see what happens.

Create a Deployment api-demo with 2 replicas with required anti-affinity by kubernetes.io/hostname over its own app label. Verify that the two replicas land on different nodes. Scale to a number larger than the available nodes and diagnose the Pending pod by reading the scheduler's event. Fix the problem by switching to preferred.

Pick a node, label it dedicated=analytics and give it the taint dedicated=analytics:NoSchedule. Then:

1. Check that a normal pod is not placed there.
2. Create a Job analytics-demo that does run there, with a toleration and affinity.
3. Show that the toleration alone is not enough: create a pod with the toleration but no affinity and see where it ends up.
4. Clean up the taint and the labels.

Solution 1

Both replicas go to m02, the only node with disk=ssd. The fact that it also has tier=high gives it 70 extra points, but that was irrelevant: the filtering had already left a single candidate.

All four on the same node. Node affinity says where a pod may go, but says nothing about spreading. For that you need anti-affinity, which is exercise 2. If m02 reboots, the whole of db-demo goes down.

Solution 2

One per node, as requested.

The message is explicit: two nodes already have a replica and the required rule forbids a second one; the third is the control plane, with its taint. With required anti-affinity by hostname, the maximum number of replicas is the number of eligible nodes.

The fix:

Four replicas spread two and two: the preference spread them as best it could and, once the alternatives ran out, allowed doubling up instead of blocking.

Solution 3

1. A normal pod does not go there:

It ended up on m02, the only one with no taint to turn it away.

2. A Job with a toleration and affinity:

3. The toleration alone is not enough:

It could have gone to m03 and it did not: the toleration permits but does not attract. The scheduler picked m02 on its score. This confirms the warning from section 5: to dedicate nodes you need taint + toleration + affinity.

4. Clean-up:

The kube-scheduler decides in two phases: it filters out the unviable nodes and scores the ones that remain. That structure explains the whole lesson: required rules act in the filtering and can leave a pod Pending; preferred rules act in the scoring and only influence the choice.

We have worked through the mechanisms from least to most expressive: nodeName (brute force, for debugging only), nodeSelector (simple equality), node affinity with its nodeSelectorTerms in OR, its matchExpressions in AND and its In, NotIn, Exists, DoesNotExist, Gt and Lt operators. The IgnoredDuringExecution in their long names means that a change to a node's labels will never move an already-scheduled pod.

Affinity and anti-affinity between pods look at who else is in the domain defined by topologyKey: node, zone or region, depending on which failure you want to protect yourself against. Their computational cost is real on large clusters, and for the specific case of spreading replicas it is worth looking at topologySpreadConstraints, whose use for high availability is content for 09-05.

Taints invert the relationship: it is the node that turns pods away, with three effects (NoSchedule, PreferNoSchedule and NoExecute, the only one that evicts running pods). Kubernetes uses them of its own accord when a node fails, and the automatic tolerations with tolerationSeconds: 300 explain the five-minute delay before the pods of a downed node reappear elsewhere. The rule to burn in: a toleration permits, it does not attract; to dedicate nodes you need a taint, a toleration and affinity.

PriorityClasses order the queue and enable preemption, with the risk of priority inflation and cascading evictions. And the FailedScheduling event, with its count of nodes ruled out for each reason, is the first and almost always the only stop in diagnosing a Pending pod.

With that, the Rutas Norte scheduling plan is complete: bookings-postgres on the SSD nodes, bookings-api and web-store spread across nodes, an analytics node reserved for occupancy-reports and a log DaemonSet that reaches everywhere.

Up to here we have used the objects Kubernetes ships with: Pods, Deployments, StatefulSets, Services, PVCs, Jobs. But in module 4 we installed cert-manager and started creating objects of type Certificate and Issuer, and in module 5 we created VolumeSnapshot. None of those types is part of Kubernetes: somebody added them to the API, and the cluster treats them exactly like the native ones — kubectl get, describe, explain, validation, RBAC. How that is done, and how we would define our own RutaProgramada type for Rutas Norte, is the subject of the next lesson: Custom Resource Definitions.