In the previous lesson we solved the problem of stateful workloads: the StatefulSet gives each replica its name, its disk and its turn. But there is a family of processes for which the question "how many replicas do I want?" makes no sense. A log collector does not need three copies or ten: it needs exactly one on every node, because the log files it has to read live on each node's disk. If tomorrow the cluster grows to twelve nodes, twelve copies are needed, without anyone having to remember to edit a replicas field.

That is the job of the DaemonSet: to guarantee that there is one pod — one, exactly one — for every node that meets certain criteria, and to maintain that invariant as nodes are added or removed.

It is an object you have, curiously enough, been using since the first lesson without ever looking at it: the kube-proxy we studied in 04-01 and the CNI network plugin are DaemonSets. In this lesson we will understand them from the inside and deploy the first infrastructure agent of our own for Rutas Norte: a log collector that reads the container files from every node.

1. One workload per node: what it is and what sets it apart
2. The four canonical use cases
3. Anatomy of the manifest: why there is no replicas
4. Where it runs: nodeSelector, affinity and tolerations
5. Privileged access: hostPath, hostNetwork, hostPID and their cost
6. Update strategies
7. Practice: a log collector for Rutas Norte
8. Verification: what happens when you add a node
9. Resources: why a badly sized DaemonSet is expensive

A DaemonSet looks like a Deployment in which someone set replicas to the number of nodes. The difference is substantial, and it lies in who decides where each pod goes.

With a Deployment, the kube-scheduler distributes the replicas according to available resources, affinity and balance. Nothing guarantees there is one per node: three could land on the same node if it has room. With a DaemonSet, the controller creates a pod with the node already assigned, one for each eligible node, and permanently watches over that correspondence.

The most important practical consequence is automatic elasticity with respect to the cluster. If you enable node autoscaling (which we will see in 09-03) and the cluster goes from 3 to 20 nodes during a bank-holiday weekend peak, the 17 new nodes get their log collector and their metrics agent with no human intervention. That property is the whole reason the object exists.

Practically everything deployed as a DaemonSet falls into one of these four categories. The rule that unites them: the pod needs something that only exists on the node where it runs.

Fluent Bit, Fluentd, Vector, Promtail. They read the files in the node's /var/log/containers/ — where the container runtime writes the standard output of every pod on that node —, enrich them with Kubernetes metadata and send them to a central backend.

They could not be a Deployment: a pod can only read the files of the node it is on. To read the logs of every node you need a pod on every node.

Prometheus Node Exporter, Datadog or New Relic agents. They read the node's /proc and /sys to expose CPU, memory, disk, network and hardware temperature. Here too, the information is strictly local to the node.

The metrics-server we enabled as a minikube addon in 01-04 is a different case: it is not a DaemonSet but a Deployment that queries each kubelet's API. We will look at it in 07-02.

Calico, Cilium, Flannel, Weave. In 04-01 we said that "the CNI programs the node's routes". The one doing that is a pod from a DaemonSet that writes configuration into /etc/cni/net.d, installs binaries in /opt/cni/bin and manipulates the node's routing tables and its iptables/eBPF rules.

There it is: kube-proxy, the component we studied in 04-01 that translates Services into network rules, is a DaemonSet. It makes perfect sense: every node needs its own set of rules.

The CSI drivers we saw in 05-05 are deployed in two pieces: a controller (Deployment or StatefulSet, one per cluster, which talks to the provider's API to create volumes) and a node plugin (DaemonSet, which mounts and unmounts the volumes on the node's file system). Only someone on that node can do the mounting.

A minimal DaemonSet:

Compared with a Deployment, one single thing is missing: spec.replicas. And its absence is not an API oversight, it is the definition of the object. The number of pods is not a value you declare, it is a consequence of the state of the cluster: as many as there are eligible nodes. If you try to add it, the API rejects it:

Apart from that, selector and template work just as they do in Deployments and StatefulSets, and spec.selector is immutable just as it is there. The template follows the same Rutas Norte conventions: only app and environment in the selector, app.kubernetes.io/part-of as an informational label.

The status is read in a characteristic way:

If DESIRED is lower than you expect, the problem is in the next section: there are nodes the DaemonSet does not consider eligible.

By default, "one pod per node" means all nodes. There are three mechanisms for restricting or widening that set.

Pods are only created on nodes carrying that label. It is what kube-proxy does in the output above: in a mixed cluster with Windows nodes, the Linux binary would be of no use.

A case of our own from Rutas Norte: if production has a group of SSD nodes labelled disk: ssd, an agent that is only of interest there is restricted like this:

When the filter is not a simple equality, node affinity is used. Only just enough here to make it work; the full mechanism, with its operators and its preferred variants, is lesson 06-05.

This example keeps the collector off the nodes dedicated to analytics.

This is the key mechanism for DaemonSets and it is worth pausing on.

Some nodes carry a taint, a mark saying "do not place pods here unless they are expressly authorised". The most common case is the control plane, which in a standard cluster comes with:

That taint stops ordinary applications from ending up competing for CPU with the apiserver and etcd. But a log collector does have to run there: the apiserver logs are precisely the ones you need most during an incident.

The way to say "I am allowed" is a toleration:

Read it as: "I tolerate the taint whose key exists with that effect". It is not a preference for going to the control plane; it is permission not to be turned away if the DaemonSet takes me there.

Infrastructure agents that must run absolutely everywhere, no matter what use a universal toleration:

That is what the CNI does: if the network plugin does not run on a node, that node has no network, so no condition should keep it out. Use it with judgement: for your log collector it is probably excessive, because it would also place it on nodes marked as not ready or under memory pressure.

There is also a courtesy Kubernetes grants DaemonSets automatically: the controller adds tolerations of its own accord for certain system taints (node.kubernetes.io/not-ready, unreachable, disk-pressure, memory-pressure, pid-pressure, unschedulable) so that the agents are not evicted just when the node is in trouble and the telemetry is needed most. You can see it by inspecting a DaemonSet pod:

The full mechanics of taints, effects and tolerationSeconds is the content of 06-05.

Node agents need to see things a normal pod does not. Kubernetes allows it, and each permission widens the attack surface in a way it is worth understanding before writing it into a manifest.

It mounts a directory of the node's file system inside the container. We saw it in 05-01 as a dangerous ephemeral volume; here it is indispensable.

A technical detail that surprises a lot of people: the files in /var/log/containers are symbolic links to /var/log/pods, which in turn usually point to the runtime's data directory (/var/lib/docker/containers or the containerd equivalent). If you only mount the first one, the container sees broken links. That is why collectors mount two or three directories.

Risk: a hostPath with write permission over a sensitive node directory (/etc, /var/lib/kubelet, the runtime socket) is equivalent to control of the node. Always mount read-only (readOnly: true) whatever you only need to read.

The pod shares the node's network stack: its ports are node ports and it sees the real interfaces. It is mandatory for CNI plugins and for agents that capture traffic. Note the dnsPolicy: without it, a pod with hostNetwork would use the node's DNS and would lose the Kubernetes name resolution we studied in 04-03.

Risk: it bypasses network isolation, and with it the NetworkPolicies of 04-06, which operate on pod IPs. A pod with hostNetwork in rutas-norte-pro is not subject to the deny-all.

The container sees every process on the node. Needed for profiling or security agents that inspect processes.

Risk: high. Seeing the host's process tree makes it possible to read command lines (which sometimes contain credentials) and, combined with privileges, to enter other containers' namespaces.

The golden rule: each one of these fields makes your DaemonSet part of the cluster's trust plane. An attacker who compromises your log collector's image with a writable hostPath has the same powers as if they had compromised the kubelet. And since it runs on every node, the compromise is total, not partial. In 08-02 we will harden these containers with securityContext, capabilities and read-only file systems.

Just like the other controllers, the DaemonSet defines how it propagates a template change.

It updates node by node. maxUnavailable limits how many pods may be unavailable at the same time; with 1 on a three-node cluster, the update takes three steps but never leaves more than one node without an agent.

maxSurge (stable since 1.25) allows the opposite: creating the new pod before removing the old one, so that no coverage gap is left. Since both fit temporarily on a node, it requires that they can coexist: if the agent uses hostNetwork with a fixed port, maxSurge: 1 causes a port conflict. The rules:

They cannot both be zero, and they cannot both be non-zero.

Pods are only updated when you delete them by hand or when the node is recreated. It is the norm for CNI plugins: updating a node's network in production is an operation you want to carry out under supervision, node by node and with a rollback plan.

History and rollback work as they do in the Deployments of 02-04:

We are going to deploy an agent that reads the container logs of every node and — for now — prints them on its own standard output. The complete centralised logging stack (Elasticsearch, Fluentd, Kibana) is built in 07-05; here the goal is the DaemonSet, not the destination of the data.

We use Fluent Bit, lightweight and an industry standard.

What each block does:

• [INPUT] tail follows the files in /var/log/containers/ whose name contains rutas-norte-pro. The name of those files has the form <pod>_<namespace>_<container>-<id>.log, so that pattern filters by namespace. DB stores the read position of each file so that everything is not resent after an agent restart: it is a state file, and that is why we will put it on a hostPath.
• [FILTER] kubernetes queries the API to add the pod, the namespace, the labels and the node to each line. This is what will let us filter by app: bookings-api in 07-05, and it requires read permissions on pods.
• [OUTPUT] stdout dumps to the agent's own standard output. In 07-05 it will be replaced by an output towards Elasticsearch.

The Kubernetes filter needs to read pods and namespaces across the whole cluster. Following the practice of 03-06, a dedicated ServiceAccount:

Read verbs only and only over two resources: the bare minimum. The detail of RBAC — what a ClusterRole is, how it is scoped, how it is audited — is lesson 08-01; here we give it ready-made so the example works.

Note that this pod does need to mount its ServiceAccount token, unlike almost every other Rutas Norte component.

Decisions worth explaining:

• priorityClassName: system-node-critical: if the node runs out of memory, this pod should not be the first to go. The mechanics of priorities and eviction are content for 06-05.
• The tolerations: only the control-plane one. We do not use operator: Exists without a key because we do not want the collector to be placed on the analytics nodes we will taint in 06-05.
• readOnly: true on both log hostPaths: the agent reads, it does not write. The only writable one is the state directory, and it points to a path of our own under /var/lib/rutasnorte, not to a system directory.
• readOnlyRootFilesystem: true with capabilities dropped to nothing: basic hardening that we will extend in 08-02.
• The NODE variable comes from the Downward API of 03-03 and lets the agent itself know where it is without querying anything.

One pod per node, and the NODE column confirms it: there are never two in the same place.

There is the result: a log line from bookings-api enriched with the pod, the namespace, the labels and the node. That enrichment is what will make it possible in 07-05 to search for "all the errors from bookings-api in production in the last two hours".

The DaemonSet's central property is demonstrated in thirty seconds with minikube:

Nobody has touched the manifest. The DaemonSet controller watches the nodes, has seen a new eligible one and has created its pod. It is exactly the same reconciliation loop we described in 01-02, applied to the "one pod per node" relationship.

It works the same way in reverse:

DESIRED has gone back to 3 on its own. Note the difference with a Deployment: when a node disappears, the Deployment reschedules its replicas onto the remaining nodes to keep the count; the DaemonSet reschedules nothing, because the pod only made sense on that node.

Here is the economic consequence that is forgotten most often.

When you ask for 200 MiB in a three-replica Deployment, you are asking for 600 MiB. When you ask for 200 MiB in a DaemonSet, you are asking for 200 MiB × the number of nodes, and that number grows with the cluster.

An example with realistic figures from a 50-node cluster and four common DaemonSets:

Ten CPUs and seventeen gigabytes reserved before deploying a single line of Rutas Norte code. And these are requests: capacity set aside by the scheduler whether it is being used or not, which reduces what is left for bookings-api and web-store.

Worse still if you err on the generous side. Setting requests.memory: 512Mi on a collector that uses 60 MiB "just in case" wastes 22 GiB on that cluster: more than a whole node paid for nothing.

Concrete advice:

• Measure before you decide. Deploy with low requests in rutas-norte-dev, watch with kubectl top pod (07-02) over a few days of real traffic and adjust.
• Always set memory limits. A collector with a leak or a spike in logs can consume all the node's memory and trigger evictions of the applications. Mem_Buf_Limit in the Fluent Bit configuration is the first line of defence; the container's limits.memory is the second.
• Be careful with CPU limits. A collector throttled on CPU falls behind and loses lines when the node is most loaded, which is exactly when the logs are needed most. A generous limit or none at all is defensible here.
• Review your quotas. The per-namespace ResourceQuota from 03-04 also applies to DaemonSet pods. If the quota for rutas-norte-pro runs out, DaemonSet pods on new nodes will not be created, and you will find out when logs go missing.
• Ask yourself whether you really need one per node. It is the most profitable piece of advice. An agent that queries the Kubernetes API does not need to be on every node: a single-replica Deployment is enough.

Trying to put replicas in the manifest. The API rejects it. If you are coming from Deployments it is the first stumble, and it is a healthy one: it forces you to internalise that the number of pods is dictated by the cluster.

Not setting tolerations and losing the control-plane nodes. The symptom is subtle: the DaemonSet shows 3/3 and everything looks fine, but the apiserver logs are missing. Always compare DESIRED with kubectl get nodes | wc -l.

Mounting only /var/log/containers. Since they are symbolic links to /var/log/pods, the agent sees names but no content, and fails with file-not-found errors that are very misleading. Mount both.

A hostPath with the wrong type. With type: Directory, if the path does not exist on the node the pod stays in ContainerCreating with a MountVolume.SetUp failed event. For your own state directories use DirectoryOrCreate.

Forgetting dnsPolicy: ClusterFirstWithHostNet with hostNetwork: true. The pod inherits the node's /etc/resolv.conf, stops resolving Service names and fails to talk to any internal component. It is a mistake that costs hours of diagnosis.

maxSurge: 1 with hostNetwork and a fixed port. The new pod cannot start because the old one is holding the port, and the update gets stuck. With hostNetwork always use maxSurge: 0.

Believing that a DaemonSet guarantees coverage during the update. With maxUnavailable: 1 and maxSurge: 0 there is always a node without an agent for a few seconds. For audit logs that may be unacceptable: use maxUnavailable: 0 and maxSurge: 1 if the agent allows it.

Tip: kubectl get pods -o wide is your verification. The NODE column is the only direct way of confirming the "one per node" invariant. Add --sort-by=.spec.nodeName to read it comfortably.

Tip: a DaemonSet is an attack vector with total reach. It runs on every node, usually has hostPath and sometimes hostNetwork. Pin the image by an immutable tag — better still, by digest —, review what RBAC it asks for and apply the hardening of 08-02 and the image scanning of 08-05.

In rutas-norte-dev on your minikube (profile rutas-norte, at least two nodes), deploy a DaemonSet called node-probe with busybox:1.36 that every 30 seconds writes to its standard output the name of the node (obtained through the Downward API) and the free space on the node's /, reading /host-root mounted read-only from hostPath: /.

Verify that there is one pod per node and check the output.

Check whether the node-probe pod has been created on the control-plane node. If not, work out why and add the necessary toleration. Verify that DESIRED goes up.

Change the node-probe image to busybox:1.37 with a strategy that leaves no node without an agent at any moment. Check the behaviour by watching the pods during the update and explain what difference you would have seen with the default strategy.

Solution 1

The resources are minimal on purpose: it is a DaemonSet and everything it asks for is multiplied by the number of nodes.

Solution 2

Three nodes but DESIRED is 2: the control plane is missing. The cause:

(On a single-node minikube this taint is not set, precisely so that applications can be deployed; on a multi-node or real cluster it does appear.)

The solution is to tolerate that taint:

Note that the toleration does not attract the pod towards the control plane: it merely stops excluding it. What places it there is the DaemonSet's "one per node" logic.

Solution 3

To leave no gaps you have to create before destroying, that is, maxUnavailable: 0 and maxSurge: 1. It is viable because node-probe uses neither hostNetwork nor host ports, so two pods can coexist for a moment on the same node.

During the update you see this pattern on each node:

The new pod reaches Running before the old one starts terminating: at no point is node m02 left without an agent.

With the default strategy (maxUnavailable: 1, maxSurge: 0) the order would be the reverse: first the old one Terminating, then the download of the new image and the start-up. Between those two moments — which with an uncached image can be tens of seconds — that node has no collector and its logs are lost.

The DaemonSet is the controller for workloads that belong to the node, not to the application. It differs from a Deployment in having no replicas: the number of pods is a consequence of how many eligible nodes there are, and it adjusts itself when the cluster grows or shrinks. The four canonical uses — logs, metrics, CNI plugins and CSI node plugins — share one trait: they need something that only exists on the node where they run, and that is why kube-proxy and your cluster's own CNI are DaemonSets.

We have seen how its reach is narrowed with nodeSelector and affinity, and above all how it is widened with tolerations to reach the control-plane nodes. We have reviewed the price of the privileged permissions these agents usually ask for (hostPath, hostNetwork, hostPID), which make the DaemonSet part of the cluster's trust plane. We have deployed the Rutas Norte log collector — whose final destination we will build in 07-05 —, checked that a new node gets its pod with no intervention, and worked out why badly tuned requests get multiplied by the number of nodes until they cost whole servers.

A whole family of workloads is still left to cover. Deployments, StatefulSets and DaemonSets have something in common: their pods are meant to run indefinitely, and if a container terminates, the system treats it as a failure and restarts it. But there is work that consists precisely of finishing: generating a report, migrating a schema, processing a batch. For that Kubernetes has two other objects, and with them we will at last deploy the missing Rutas Norte component, occupancy-reports. That is the subject of the next lesson: Jobs and CronJobs.