The previous lesson ended with an honest list of Helm's shortcomings: the templates stop being valid YAML, nindent and whitespace are a constant source of errors, and you have to learn the Go template engine before you can touch anything. Kustomize was born out of exactly that discontent and starts from the opposite premise: no templating at all.

In Kustomize, your manifests remain perfectly valid Kubernetes YAML. You can open them in your editor and have the schema validated, apply them with kubectl apply -f, and read them without decoding anything. What changes between environments is not expressed with variables but with overlays: small files declaring the differences from a common base.

In this lesson we will migrate Rutas Norte's k8s/ from 120 duplicated files to one base plus three overlays, we will cleanly solve the restart-on-configuration-change problem that in 03-03 we patched with a hand-written annotation, and we will finish by comparing Helm and Kustomize without favouritism.

1. The template-free philosophy
2. Kustomize inside kubectl and as a standalone binary
3. The base + overlays model
4. kustomization.yaml field by field
5. Patches: strategic merge and JSON Patch
6. ConfigMap and Secret generators
7. Components: reusable optional pieces
8. Workflow: kustomize, diff and apply
9. The complete Rutas Norte migration
10. Helm versus Kustomize
11. Combining Helm and Kustomize
12. Common mistakes and tips
13. Exercises
14. Conclusion

Compare the two approaches to the same goal: giving bookings-api 1 replica in development and 4 in production.

With Helm, the manifest stops being Kubernetes YAML: replicas: {{ .Values.bookingsApi.replicaCount }}, image: {{ .Values.imageRegistry }}/bookings-api:{{ .Values.bookingsApi.image.tag }}. With Kustomize, the base is plain ordinary YAML (replicas: 1, image: registry.rutasnorte.example/bookings-api:2.4.0), applicable with kubectl apply -f, and the overlay declares the differences:

Kustomize works as a pipeline of transformations: it reads the base manifests, applies a series of declared operations (change the namespace, add labels, substitute the image, apply patches) and emits the resulting YAML.

That "transform existing YAML" nature has an elegant consequence: Kustomize understands Kubernetes types. When it merges two container lists it knows the correlation key is name; when it changes an image it knows where the image fields are in a Deployment, a StatefulSet, a CronJob or a DaemonSet. Helm, which only concatenates text, knows none of that.

An enormous advantage: there is nothing to install. Anybody with kubectl can deploy. Disadvantage: the built-in version lags behind the standalone one and cannot be updated without updating kubectl.

Recommendation for Rutas Norte: kubectl -k for everyday interactive use; the binary with a pinned version in the ci-rutasnorte pipeline, so that everybody generates exactly the same YAML.

Three concepts:

• Base: the common manifests. It must be deployable on its own and it holds the most conservative values.
• Overlay: a directory that references a base and declares the differences. One per environment.
• Component: a reusable piece that overlays optionally enable.

Note that this file is directly applicable with kubectl apply -f. You cannot do that with a Helm template: it is the main advantage of the approach.

And the development overlay is eleven lines:

Files, directories with their own kustomization.yaml, or remote URLs:

About remote resources: always pin ?ref= to a tag or a commit. Without it you are pointing at the main branch and your deployment changes whenever somebody outside your team commits. It is the same principle as --version in Helm and immutable image tags (08-05).

namespace: rutas-norte-pro sets metadata.namespace on every generated object and updates the cross-references (the namespace in a RoleBinding's subjects, for example). A single field eliminates the main duplication between environments. Cluster-scoped objects (ClusterRole, StorageClass, CRD) are unaffected: Kustomize knows.

namePrefix: rn- and nameSuffix: -pro rename the objects, and — very importantly — Kustomize updates the references: the HPA's scaleTargetRef, the StatefulSet's serviceName, the ConfigMap name in envFrom, the Ingress's backend.service.name. Rutas Norte does not use them, because it already separates environments by namespace and appending -pro to everything's name complicates the diagnostic commands; they are useful when you deploy two instances into the same namespace.

This section deserves special attention because it is a real trap that breaks deployments.

commonLabels (the old form) adds the labels to metadata.labels and also to the Deployment's spec.selector.matchLabels, to the pod's spec.template.metadata.labels and to the Service's spec.selector. And there is the problem: a Deployment's spec.selector is immutable (02-03, 02-07). If the base is already deployed and you add a label, the next apply fails:

The only way out is to delete the Deployment and recreate it, with an outage. In rutas-norte-pro, at eleven in the morning.

There is a second, subtler problem: if it adds environment: pro to the Service's selector, that Service will stop finding the pods that already existed without that label. Traffic to nowhere, with no visible error.

Rutas Norte's rule: the selector labels are defined once in the base (app: bookings-api) and are never touched again. Everything else is added with includeSelectors: false.

Kustomize looks for the image field in any type that has one, including initContainers. You do not have to tell it where to look. This is what the ci-rutasnorte pipeline modifies on every deployment:

kustomize edit modifies the kustomization.yaml on disk. Combined with an automatic commit, it is the piece that connects the image build with GitOps (10-05).

It saves writing a patch for something so common. But careful, picking up from 09-01: if bookings-api has an HPA, declaring replicas here is counterproductive. Every kubectl apply -k will return the Deployment to 4 replicas even though the HPA has it at 15 because of the May bank-holiday weekend load, with a capacity dip until the HPA reacts.

For components with an HPA, the correct solution is not to declare replicas anywhere (neither in the base nor in the overlay) and to let the HPA's minReplicas govern. Kubernetes does not require the field: it defaults to 1 and the HPA raises it immediately. The missing piece — making sure Argo CD does not report drift when the HPA changes that field — we will solve in 10-05 with ignoreDifferences.

The fields above cover the usual cases. For everything else, there are patches.

You write a partial YAML with the same structure as the object. Kustomize merges it while understanding the semantics of Kubernetes types.

What makes strategic merge special is that Kustomize knows the schema: it knows that containers correlates by name, ports by containerPort, volumeMounts by mountPath. A patch that only mentions the api container leaves the sidecars untouched.

It can be written inline with patch: |- inside the kustomization.yaml, handy for small changes. And there are two special directives: $patch: replace replaces an entire list instead of merging it, and $patch: delete removes the matching element.

When strategic merge is not enough — because you have to operate on list indexes, or on a CRD that Kustomize does not know — you use JSON Patch: a list of explicit operations.

Details you need to know: the paths start at the root of the object and list indexes are numeric (/spec/template/spec/containers/0/); replace fails if the path does not exist, whereas add creates or replaces it, so when in doubt use add.

Here is the system's real power. A patch can be aimed at many objects at once.

The selectors available in target are kind, name (exact or regex), namespace, group, version, labelSelector and annotationSelector.

You will see plenty of old code using those two fields. The unified patches field replaces both: it automatically detects whether the content is a strategic merge or a JSON Patch, and it accepts target with selectors in both cases, which the old ones did not allow.

kustomize edit fix migrates the deprecated fields automatically.

This is Kustomize's most elegant feature, and it cleanly solves a problem we fixed by hand in 03-03.

Kustomize appends a hash of the contents to the name. And — this is the important bit — it automatically updates every reference, so the Deployment ends up with configMapRef: { name: bookings-api-config-9t2hmf6b4d }.

Think about what that implies. When you change LOG_LEVEL from info to warn: the contents change → the hash changes → the reference in the Deployment changes → the pod template changes → the Deployment rolls out automatically.

This solves, natively and without tricks, the problem from 03-03. There we explained that an updated ConfigMap does not restart the pods and that the workaround was adding an annotation with the hash by hand. With Kustomize there is no workaround: it is the default behaviour. And compared with Helm (10-03), where you had to write checksum/config: {{ include ... | sha256sum }} in every template, here you write nothing.

An additional advantage: rollback genuinely works. Since every version of the configuration is a distinct object with a distinct name, a kubectl rollout undo returns the pod to the previous reference, and the old ConfigMap still exists as long as some ReplicaSet references it. With a fixed-name ConfigMap, the rollback would restore the old code but with the new configuration: the worst of both worlds.

Also per generator, with options: { disableNameSuffixHash: true } inside a specific entry.

When should you disable it? Only when something external references the ConfigMap by a fixed name that Kustomize cannot update: a CRD from an operator that Kustomize cannot interpret, a pod that deliberately re-reads the ConfigMap at runtime, or one shared between applications managed by different systems. Firm advice: do not disable it without a concrete reason. The hash is Kustomize's best feature.

If the base defines bookings-api-config with LOG_LEVEL=info, BOOKING_TTL_MINUTES=15 and CACHE_HOST=redis-cache, and the production overlay declares:

The result combines the three inherited keys with LOG_LEVEL overridden and GATEWAY_URL added.

The same mechanics, with automatic base64 encoding and support for type: kubernetes.io/tls for certificates from files.

Critical warning: secretGenerator encrypts nothing. Base64 is not encryption (03-02). If you put the bookings-postgres password in a literals entry or in a file in the repository, that password is in the clear in Git for ever, even if you delete it later.

The correct approach at Rutas Norte, picking up what we announced in 03-02, is to reference a Secret encrypted with SOPS or Sealed Secrets, or better still an External Secrets Operator resource that fetches the value from Vault:

That file is harmless: it only says where the secret is, not what it is. We will come back to it in 10-05.

A component is like an overlay, but designed to be included by several of them. It solves the "I want this in pre and in pro, but not in dev" case. Rutas Norte has three clear candidates: the NetworkPolicies (04-06), high availability (09-05) and observability (07-03). In development they get in the way; in the other two environments they are mandatory.

A component can also carry patches, not just resources:

And the overlays enable them: dev carries none, pre carries network-policies and observability, and pro adds high-availability on top.

Components are applied in the order they appear, after the resources. If two of them patch the same field, the last one wins.

That output tells you exactly what is going to happen: the configuration hash changes (there will be a rollout) and the memory limit goes up. It is the equivalent of helm diff from the previous lesson, but built into kubectl, with no plugins.

A real problem: if you delete a manifest from the repository, kubectl apply -k does not delete the object from the cluster. It is left orphaned. There is --prune with --applyset and --prune-allowlist, but it is cumbersome and you have to enumerate the kinds. It is another of the problems GitOps (10-05) solves natively: Argo CD and Flux know which objects belong to them and prune them on their own.

And what matters is not the number of files, but that every line of configuration exists exactly once.

Step 1: choose the base. The simplest environment, normally dev. Copy its manifests to k8s/base/, stripping the namespace (the overlay will set it) with yq -i 'del(.metadata.namespace)' k8s/base/**/*.yaml.

Step 2: write the base's kustomization.yaml files (section 3).

Step 3: work out each environment's real delta.

Four differences. Three are solved with kustomization.yaml fields (namespace, replicas, images) and only one needs a patch (resources).

Step 4: write the overlays.

Step 5: verify that the migration changes nothing. This is the step that builds confidence.

An empty diff means you can run kubectl apply -k and absolutely nothing will happen. The migration is a risk-free operation.

Step 6: delete the old directories with git rm -r k8s/dev k8s/pre k8s/pro and update the pipeline.

Helm to consume, Kustomize to produce. It is Rutas Norte's decision and that of most mature teams: nobody wants to maintain kube-prometheus-stack's 60 objects by hand, and nobody wants to turn their own Deployment into an unreadable template. The exception: if you distribute your software to customers who install it in their own clusters, you need a versioned, configurable package, and there Helm has no rival.

There are two ways to use the two tools together, and they serve the same purpose: modifying a third-party chart in a field its author did not parameterise.

A single file declares everything and updating means changing the version number, but it needs --enable-helm and a helm installation, the result is not in Git, and support is uneven (kubectl -k does not handle it well; Argo CD and Flux need extra configuration).

Recommendation for Rutas Norte: approach A. Having the generated YAML in Git is exactly what makes GitOps work well: anybody can read the repository and know what is deployed.

1. Using commonLabels (or labels with includeSelectors: true) on something already deployed. It modifies spec.selector, which is immutable, and the apply fails. The only way out is delete and recreate, with an outage. Always use includeSelectors: false.

2. Declaring replicas on a component with an HPA. Every apply undoes the autoscaling until the HPA reacts. For anything with an HPA, do not declare replicas anywhere.

3. Secrets in secretGenerator with literals. Base64 is not encryption. If you write the bookings-postgres password into a file in the repository, it is in the clear in the Git history for ever. Use SOPS, Sealed Secrets or External Secrets Operator.

4. Setting disableNameSuffixHash without needing to. You lose the automatic rollout on configuration changes, which is Kustomize's best feature.

5. Remote resources without ?ref=. Your deployment changes when a stranger makes a commit. Always pin a tag or a commit.

6. Putting too much in the base. If the base contains things half the environments have to patch out, it is badly designed. The base is the common minimum; the optional goes in components.

7. Very deep overlay chains. Base → common → region → environment → customer is technically possible and humanly impractical: nobody knows where a value comes from. Two levels, three at most.

8. A patch that does not apply and nobody notices. If the target matches nothing, Kustomize does not always warn you. Always verify with kubectl kustomize that the change appears in the output.

9. replace in a JSON Patch on a path that does not exist. It fails with "missing value". Use add, which creates or replaces.

10. Forgetting to escape / in JSON Patch keys. rutasnorte.example/revision is written rutasnorte.example~1revision.

11. Applying without kubectl diff -k first. It is free, it takes two seconds and it shows you exactly what is going to change. In production it should be mandatory.

12. Assuming that lists always merge. In strategic merge, lists with a known correlation key (containers by name) merge; those without one (args, command) are replaced wholesale.

13. Editing objects by hand with kubectl edit and forgetting about it. Kustomize does not reconcile: it only acts when somebody runs apply. The next deployment will overwrite the manual change without warning. This is exactly the problem GitOps solves.

Starting from the base described in the lesson, write k8s/environments/pre/kustomization.yaml so that it: uses the rutas-norte-pre namespace, labels everything with environment: pre without touching the selectors, pins the images to rc-2.4.0, enables the network-policies and observability components, merges a ConfigMap with LOG_LEVEL=info and a test gateway URL, and applies a patch raising the resources to half of production's. Verify the result without applying anything.

Write a patch that adds to all Deployments labelled app.kubernetes.io/part-of: rutas-norte an initContainer waiting for bookings-postgres to be reachable before starting (06-04), without modifying any file in the base. Explain why you need JSON Patch and not strategic merge.

With the dev overlay deployed, demonstrate in three steps that changing one line of config.env triggers an automatic rollout: capture the ConfigMap name and the Deployment's generation beforehand, change the value, apply, and compare. Explain why this would not happen with a fixed-name ConfigMap.

Solution 1

Solution 2

Note: the first operation would empty an already-existing initContainers list. If some Deployments already have init containers that must be preserved, the robust solution is to split it into two patches with different targets.

Why JSON Patch: strategic merge demands one file per object with its exact metadata.name, so it cannot be aimed at "everything matching a label". JSON Patch combined with labelSelector in target reaches N objects with a single declaration, and /- lets you append without knowing how many elements there were.

Solution 3

Explanation: the hash is part of the ConfigMap's name, and that name appears inside the Deployment's spec.template. Changing the pod template makes the controller create a new ReplicaSet and run the rolling update (02-03).

Why it would not happen with a fixed name: spec.template would remain byte-for-byte identical, metadata.generation would not change, and there would be no new ReplicaSet. The pods would carry on with the old variables until somebody ran kubectl rollout restart by hand. That is precisely the checksum/config annotation workaround from 03-03, which is not needed here.

Kustomize has reduced Rutas Norte's k8s/ from 114 duplicated files to 43, with every value defined exactly once. The essentials:

• No templating: the manifests in k8s/base/ remain valid Kubernetes YAML, applicable with kubectl apply -f and checkable by your editor. What changes between environments is declared as transformations in the overlay.
• It ships with kubectl (apply -k, diff -k, kubectl kustomize), although the standalone binary is more up to date and is the one worth pinning in the pipeline.
• The base + overlays + components model covers the three real axes: the common part, what is specific to each environment, and the optional capabilities only some environments want.
• namespace, images, replicas and labels resolve most differences without writing a patch. And labels must carry includeSelectors: false: commonLabels touches the immutable selectors and breaks running deployments.
• For everything else there are patches, with strategic merge or JSON Patch, and a target that can be aimed by kind, name, label or annotation. patchesStrategicMerge and patchesJson6902 are deprecated.
• The generators with their hash suffix are the jewel: changing one line of configuration changes the ConfigMap's name, changes the pod template and triggers an automatic rollout. It natively solves what we fixed by hand in 03-03, and it also makes the rollback restore code and configuration together.
• And in the honest comparison with Helm, the rule is clear: Helm for consuming third-party software, Kustomize for your own applications, with two ways of combining them when you need to patch somebody else's chart.

But one problem remains that neither Helm nor Kustomize solves, and it has come up in both lessons. Both tools only act when somebody runs a command. If somebody runs kubectl edit in production, neither finds out. If the laptop of whoever deploys breaks, nobody knows how to deploy. If the ci-rutasnorte pipeline needs cluster administrator credentials to run kubectl apply, we have a serious security problem that contradicts the least-privilege RBAC we defined in 08-01.

In the next lesson, GitOps with Argo CD and Flux, we take the final step: an agent that lives inside the cluster, pulls from Git continuously and reconciles reality with what is declared. We will look at Argo CD's Application resource, ApplicationSets for deploying the three environments from a single definition, the Flux equivalent, the three solutions to the problem of secrets in a repository, and we will finally close the conflict between the HPA and the replicas field that we flagged back in 09-01.