The Project | About Us | Contribute | Donations | License

HOME

We arrive at the problem that opened this module. Rutas Norte's k8s/ directory holds more than 120 YAML files. web-store-dev.yaml, web-store-pre.yaml and web-store-pro.yaml are identical except for the namespace, the replica count, the image tag and two resource limits. Multiply that by six components, each with its Deployment, Service, HPA, ConfigMap, Ingress and NetworkPolicy, and you have the current mess: every change has to be replicated three times, and nobody is sure which version is running in production.

Helm is the Kubernetes package manager, and it is the first of the two answers to that problem (the other, Kustomize, is the next lesson). We have already used it twice in this course without explaining it: to install cert-manager in 04-05 and kube-prometheus-stack in 07-03. In this lesson we are going to understand what was really happening in those commands, and then we will build the Rutas Norte chart from scratch.

Rutas Norte's problem and how Helm tackles it
Concepts: chart, release, repository and values
Consuming third-party charts with judgement
The structure of a chart
Templates: syntax, functions and control flow
Building the Rutas Norte chart
Values per environment
The life cycle of a release
Debugging: template, lint, dry-run and diff
Dependencies and subcharts
Hooks
Packaging and publishing
What Helm does badly
Common mistakes and tips
Exercises
Conclusion

Rutas Norte's problem and how Helm tackles it

The real difference between dev and pro for bookings-api is this:

Field	dev	pre	pro
`metadata.namespace`	`rutas-norte-dev`	`rutas-norte-pre`	`rutas-norte-pro`
`spec.replicas`	1	2	4
`image`	`:dev-abc123`	`:rc-2.4.0`	`:2.4.0@sha256:...`
`resources.limits.memory`	`256Mi`	`512Mi`	`1Gi`
HPA `maxReplicas`	3	5	20

Five values. And for five values we maintain 114 duplicated lines per component.

Helm solves this with a simple idea: it turns the manifests into templates and pulls the variable values out into a separate file.

flowchart LR
    T["templates/<br/>deployment.yaml"] --> R{{"helm<br/>render"}}
    V1["values-dev.yaml"] --> R
    V2["values-pre.yaml"] --> R
    V3["values-pro.yaml"] --> R
    R --> M1["dev manifests"]
    R --> M2["pre manifests"]
    R --> M3["pro manifests"]
    M1 --> K[(Cluster)]
    M2 --> K
    M3 --> K

But Helm does more than substitute variables, and this is what sets it apart from a simple sed: it manages the complete life cycle of the installation. It knows which objects it installed, in which version, and it can uninstall them all or go back with a single command.

Concepts: chart, release, repository and values

Chart: the package. A directory (or a .tgz) with the templates, the default values and the metadata. It is the equivalent of a .deb. It is not installed: it is inert material.
Release: a specific, named installation of a chart. The same chart can be installed several times, and each release has its own independent history.
Repository: an HTTP server (or an OCI registry) holding packaged charts and an index.yaml cataloguing them.
Values: the configuration. They are resolved in layers, and the lowest one wins: the chart's values.yaml → each -f file.yaml in order (the last one wins) → --set on the command line.

Helm is a stateful package manager

This is its most misunderstood characteristic. When you run helm install, Helm renders the templates and applies them, but it also stores a copy of everything in a Secret inside the release's namespace.

kubectl get secrets -n rutas-norte-dev -l owner=helm

NAME                                TYPE                 DATA   AGE
sh.helm.release.v1.rutas-norte.v1   helm.sh/release.v1   1      3d
sh.helm.release.v1.rutas-norte.v2   helm.sh/release.v1   1      2d
sh.helm.release.v1.rutas-norte.v3   helm.sh/release.v1   1      4h

Each revision is a Secret containing the complete chart, the values used and the rendered manifests, compressed and base64-encoded.

Consequence	Explanation
`helm rollback` is possible	Helm knows exactly what was there before
`helm uninstall` deletes everything it installed	It knows which objects are its own
If somebody runs `kubectl edit`, Helm never finds out	The stored state and the real one diverge
If you delete those Secrets, Helm "loses" the release	Even though the objects are still in the cluster
The state lives in the cluster, not in Git	The main criticism of Helm, which GitOps resolves (10-05)

Helm 2 had a server-side component called Tiller, with administrator permissions, which was a notorious security problem. Helm 3 removed it: today helm is just a binary on your machine that talks to the API using your kubeconfig and your RBAC (08-01).

Consuming third-party charts with judgement

Let us revisit what we did in 04-05 and 07-03, now understanding every step.

helm repo add jetstack https://charts.jetstack.io
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Download the indexes. WITHOUT THIS, helm works from a stale cache.
helm repo update

# See ALL the available versions
helm search repo jetstack/cert-manager --versions | head -4

NAME                    CHART VERSION   APP VERSION   DESCRIPTION
jetstack/cert-manager   v1.16.1         v1.16.1       A Helm chart for cert-manager
jetstack/cert-manager   v1.15.3         v1.15.3       A Helm chart for cert-manager

Note the two version columns. CHART VERSION is the packaging version (it changes when the author modifies the templates); APP VERSION is the version of the software being installed. They advance independently, and --version pins the chart's.

The step almost nobody takes, and the one that separates a professional from somebody copying commands off a blog: reading the available values before installing.

helm show values jetstack/cert-manager --version v1.16.1 > /tmp/defaults.yaml
wc -l /tmp/defaults.yaml        # 1204 lines of documented options

Everything configurable is in there: replicas, resources, tolerations, PDB, Prometheus integration, crds.enabled. If you install without looking at it, you are blindly accepting the author's defaults. helm show chart, helm show readme and helm show all complement it.

cert-manager, now with judgement

In 04-05 we ran something like helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.16.1 --set crds.enabled=true. Now we can read it: cert-manager is the release name (many objects inherit it); --create-namespace is needed because Helm does not create namespaces unless you ask it to; and --version is essential, because without it you install whatever was latest at that moment, and reinstalling six months later installs something different. It is the same principle as immutable image tags (08-05).

The professional version uses a values file kept under version control in the repository:

# platform/cert-manager/values-pro.yaml
crds:
  enabled: true
  keep: true          # do not delete the CRDs on uninstall: they would take
                      # every Certificate in the cluster down with them
replicaCount: 2       # high availability (09-05)
podDisruptionBudget:
  enabled: true
  minAvailable: 1
resources:
  requests: { cpu: 10m, memory: 32Mi }
  limits:   { memory: 128Mi }
prometheus:
  servicemonitor:
    enabled: true     # so kube-prometheus-stack picks it up (07-03)
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels: { app.kubernetes.io/instance: cert-manager }

helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.16.1 -f platform/cert-manager/values-pro.yaml \
  --atomic --timeout 5m

And the same for Prometheus, where one value deserves special attention:

# platform/kube-prometheus-stack/values-pro.yaml
prometheus:
  prometheusSpec:
    retention: 30d
    # Make it discover ServiceMonitors in ALL namespaces. Without this,
    # the ServiceMonitors in rutas-norte-pro are ignored SILENTLY.
    serviceMonitorSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: rutas-norte-ssd
          resources: { requests: { storage: 100Gi } }
grafana:
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts: ["grafana.rutasnorte.example"]

A checklist before installing somebody else's chart

Check	How
Who maintains it?	`helm show chart` and the source repository
Which defaults am I accepting?	`helm show values > file.yaml` and read it
What exactly is it going to create?	`helm template ... \| less`
Does it ask for cluster-admin permissions?	Look for `ClusterRole` in the `helm template` output
Does it install CRDs? What happens on uninstall?	Look for the `crds/` directory in the chart
Which images does it pull, and from where?	`helm template ... \| grep image:`
Have I pinned the version and stored the values in Git?	`--version` and `-f`, always

The structure of a chart

helm create rutas-norte-chart

rutas-norte-chart/
├── Chart.yaml           # the chart's metadata
├── values.yaml          # default values
├── charts/              # downloaded subcharts
├── crds/                # CRDs, installed before everything else
├── .helmignore          # what to exclude when packaging
└── templates/
    ├── NOTES.txt        # message printed after installing
    ├── _helpers.tpl     # reusable fragments (generates no manifests)
    ├── deployment.yaml  # ... and the rest of the templates
    └── tests/

# Chart.yaml
apiVersion: v2                    # v2 = Helm 3. Do not use v1.
name: rutas-norte-chart
description: Ticket-sales platform for Rutas Norte S.L.
version: 1.4.0                    # the CHART's version, semantic
appVersion: "2.4.0"               # the version of the APPLICATION it deploys
type: application
kubeVersion: ">=1.28.0-0"         # if the cluster does not comply, install fails clearly
maintainers:
  - { name: Platform team, email: [email protected] }
dependencies:
  - name: redis
    version: "20.1.0"
    repository: "https://charts.bitnami.com/bitnami"
    condition: redis.enabled

The distinction between version and appVersion confuses everybody. Rule: if you change a template, bump version. If you change the version of the deployed software, bump appVersion (and version as well, because the chart has changed).

About the special directories:

templates/: everything goes through the template engine. Files starting with _ (such as _helpers.tpl) are rendered but generate no manifests: they exist to define fragments. NOTES.txt is printed on screen after installing.
charts/: this is where subcharts land with helm dependency update.
crds/: plain YAML, no templating. Helm installs them before everything else and never updates or deletes them. It is a deliberate limitation and a constant source of friction (section 13).
.helmignore: just like .dockerignore, what not to include when packaging.

Templates: syntax, functions and control flow

Helm uses the Go template engine with the functions from the Sprig library. Everything goes inside {{ }}.

The built-in objects

Object	Contents	Example
`.Values`	The resolved values	`.Values.replicaCount`
`.Release`	Data about the release	`.Release.Name`, `.Release.Namespace`, `.Release.Revision`, `.Release.IsUpgrade`
`.Chart`	The contents of `Chart.yaml`	`.Chart.Name`, `.Chart.Version`, `.Chart.AppVersion`
`.Capabilities`	What the cluster can do	`.Capabilities.APIVersions.Has "..."`
`.Files`	Chart files outside `templates/`	`.Files.Get "config/app.conf"`

Functions and pipelines

The pipe | passes the result on the left as the last argument of the function on the right.

# 'quote' adds quotation marks. Essential with numbers that must be text.
version: {{ .Chart.AppVersion | quote }}        # -> "2.4.0"

# 'default' supplies a value if the one on the left is empty
replicas: {{ .Values.replicaCount | default 1 }}

# 'trunc 63' because Kubernetes labels have that limit
name: {{ .Release.Name | trunc 63 | trimSuffix "-" }}

# 'required' ABORTS with your message if the value is missing. Far better
# than deploying something broken.
image: {{ required "You must set image.repository" .Values.image.repository }}

# 'toYaml' turns a structure from values.yaml into YAML;
# 'nindent N' adds a newline + N spaces to every line
resources:
  {{- toYaml .Values.resources | nindent 2 }}

# 'sha256sum' computes a hash: we will use it for the automatic restart
checksum/config: {{ .Values.config | toYaml | sha256sum }}

indent versus nindent causes more YAML syntax errors than anything else: indent N adds N spaces to every line; nindent N does the same plus a newline at the start. Practical rule: always use {{- ... | nindent N }}, that is the pattern that works.

The hyphens {{- and -}} strip the whitespace before and after. Without them, every {{ if }} leaves a blank line in the output and the resulting YAML is unreadable even when it is valid.

Control flow

{{- if .Values.ingress.enabled }}
# ...
{{- else if eq .Values.environment "pre" }}
# ...
{{- end }}

The following count as false: false, 0, the empty string, nil, and empty lists or maps. The operators (eq, ne, lt, gt, and, or, not, empty) go first, in prefix notation: {{- if and .Values.hpa.enabled (gt (int .Values.hpa.maxReplicas) 1) }}.

# 'with' checks that the value is not empty AND switches '.' to that value
{{- with .Values.nodeSelector }}
nodeSelector:
  {{- toYaml . | nindent 2 }}
{{- end }}

It is the idiomatic pattern for optional fields. Trap: inside a with, . is no longer the root; for the global context you use $ (for example, {{ $.Release.Name }}).

# 'range' over a list, or over a map capturing key and value
env:
{{- range $key, $value := .Values.env }}
  - name: {{ $key }}
    value: {{ $value | quote }}
{{- end }}

`define` and `include`: reusable fragments

Here is the key to not repeating Rutas Norte's labels across twenty templates.

{{/* templates/_helpers.tpl */}}

{{- define "rutas-norte.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{- define "rutas-norte.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}

{{/* Labels common to ALL objects, following the conventions from 02-07 */}}
{{- define "rutas-norte.labels" -}}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 }}
{{ include "rutas-norte.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/part-of: rutas-norte
environment: {{ .Values.environment | quote }}
{{- end }}

{{/* SELECTOR labels. They are IMMUTABLE in a Deployment (02-03):
     if they change, the upgrade fails. That is why they live apart and hold
     the BARE minimum, never to be touched again. */}}
{{- define "rutas-norte.selectorLabels" -}}
app.kubernetes.io/name: {{ include "rutas-norte.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

They are used like this: {{- include "rutas-norte.labels" . | nindent 4 }}.

include versus template: template inserts the result but cannot be chained with pipes; include can. Since you almost always need | nindent, always use include.

Building the Rutas Norte chart

`values.yaml` (the defaults)

environment: dev                            # dev | pre | pro
imageRegistry: registry.rutasnorte.example

bookingsApi:
  enabled: true
  replicaCount: 1
  image:
    repository: bookings-api
    tag: ""            # empty -> .Chart.AppVersion is used
    digest: ""         # if set, it takes precedence (08-05)
    pullPolicy: IfNotPresent
  service: { type: ClusterIP, port: 80, targetPort: 8080 }
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits:   { memory: 256Mi }

  # Non-sensitive configuration -> ends up in a ConfigMap (03-01)
  config:
    LOG_LEVEL: info
    BOOKING_TTL_MINUTES: "15"
    GATEWAY_URL: https://pagos.proveedorexterno.example/v2

  # Secrets: ONLY the name of a Secret that already exists.
  # NEVER put sensitive values in values.yaml (03-02).
  existingSecret: bookings-api-credentials

  hpa: { enabled: false, minReplicas: 1, maxReplicas: 3, targetCPUUtilizationPercentage: 70 }
  probes:
    liveness:  { path: /health/live,  initialDelaySeconds: 15, periodSeconds: 20 }
    readiness: { path: /health/ready, initialDelaySeconds: 5,  periodSeconds: 5 }
  nodeSelector: {}
  tolerations: []
  topologySpreadConstraints: []

# Shared security context (08-02)
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001
  fsGroup: 10001
  seccompProfile: { type: RuntimeDefault }
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: { drop: ["ALL"] }

The ConfigMap

{{- if .Values.bookingsApi.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "rutas-norte.fullname" . }}-api-config
  labels:
    {{- include "rutas-norte.labels" . | nindent 4 }}
data:
  {{- range $key, $value := .Values.bookingsApi.config }}
  {{ $key }}: {{ $value | quote }}
  {{- end }}
{{- end }}

Simple but powerful: adding a new variable in values.yaml does not touch the template.

The Deployment

{{- if .Values.bookingsApi.enabled }}
{{- $c := .Values.bookingsApi }}            {{/* local variable: saves the long path */}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "rutas-norte.fullname" . }}-api
  labels:
    {{- include "rutas-norte.labels" . | nindent 4 }}
    app: bookings-api
spec:
  {{- if not $c.hpa.enabled }}
  # IMPORTANT: if the HPA is active (09-01) we do NOT declare replicas.
  # If we did, every 'helm upgrade' would return the Deployment to the
  # chart's value, undoing the autoscaling.
  replicas: {{ $c.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "rutas-norte.selectorLabels" . | nindent 6 }}
      app: bookings-api
  strategy:
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  template:
    metadata:
      labels:
        {{- include "rutas-norte.labels" . | nindent 8 }}
        app: bookings-api
      annotations:
        # Hash of the configuration: if the ConfigMap changes, this hash
        # changes, the pod template changes and the Deployment rolls out
        # by itself. It is the mechanism from 03-03, now computed automatically.
        checksum/config: {{ include (print $.Template.BasePath "/bookings-api/configmap.yaml") . | sha256sum }}
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: {{ include "rutas-norte.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: api
          image: {{ printf "%s/%s" .Values.imageRegistry (include "rutas-norte.apiImage" .) }}
          imagePullPolicy: {{ $c.image.pullPolicy }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          ports:
            - { name: http, containerPort: {{ $c.service.targetPort }} }
          envFrom:
            - configMapRef: { name: {{ include "rutas-norte.fullname" . }}-api-config }
            {{- if $c.existingSecret }}
            - secretRef: { name: {{ $c.existingSecret }} }
            {{- end }}
          livenessProbe:
            httpGet: { path: {{ $c.probes.liveness.path }}, port: http }
            initialDelaySeconds: {{ $c.probes.liveness.initialDelaySeconds }}
          readinessProbe:
            httpGet: { path: {{ $c.probes.readiness.path }}, port: http }
            initialDelaySeconds: {{ $c.probes.readiness.initialDelaySeconds }}
          resources:
            {{- toYaml $c.resources | nindent 12 }}
          volumeMounts:
            # readOnlyRootFilesystem demands volumes for anything writable
            - { name: tmp, mountPath: /tmp }
      volumes:
        - { name: tmp, emptyDir: {} }
      {{- with $c.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with $c.topologySpreadConstraints }}
      topologySpreadConstraints:
        {{- toYaml . | nindent 8 }}
      {{- end }}
{{- end }}

Four points deserve attention:

replicas conditioned on the HPA solves, within Helm, half of the conflict flagged in 09-01. The other half (making sure Argo CD does not report drift on that field) we will see in 10-05.
checksum/config renders the ConfigMap template and hashes it: it automates exactly what we did by hand in 03-03.
with on the optional fields prevents nodeSelector: {} or, worse, nodeSelector: null from appearing.
The Service and the HPA follow the same pattern: {{- if and .Values.bookingsApi.enabled .Values.bookingsApi.hpa.enabled }} wraps the HPA, whose scaleTargetRef.name uses the same include "rutas-norte.fullname" so it can never drift out of sync with the Deployment.

And NOTES.txt closes the installation with contextual instructions:

The Rutas Norte platform has been deployed to the {{ .Values.environment }} environment.
Release: {{ .Release.Name }} | Namespace: {{ .Release.Namespace }} | Revision: {{ .Release.Revision }}

  kubectl get pods -n {{ .Release.Namespace }} -l app.kubernetes.io/part-of=rutas-norte
{{- if eq .Values.environment "pro" }}

*** PRODUCTION ENVIRONMENT *** Check the rollout before closing the window:
  kubectl rollout status deploy/{{ include "rutas-norte.fullname" . }}-api -n {{ .Release.Namespace }}
{{- end }}

Values per environment

This is where the 120 files disappear.

# environments/values-dev.yaml
environment: dev
bookingsApi:
  replicaCount: 1
  image: { tag: dev-abc123f, pullPolicy: Always }
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { memory: 256Mi }
  config:
    LOG_LEVEL: debug
    GATEWAY_URL: https://pagos-sandbox.proveedorexterno.example/v2
  hpa: { enabled: false }

# environments/values-pro.yaml
environment: pro
bookingsApi:
  replicaCount: 4
  image:
    tag: "2.4.0"
    # Immutable digest: this is what actually gets deployed (08-05)
    digest: "sha256:9c1e4a7b3d2f8e6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d4e3f2a"
  resources:
    requests: { cpu: 250m, memory: 512Mi }
    limits:   { memory: 1Gi }
  config:
    LOG_LEVEL: warn
    GATEWAY_URL: https://pagos.proveedorexterno.example/v2
  hpa: { enabled: true, minReplicas: 4, maxReplicas: 20, targetCPUUtilizationPercentage: 60 }
  podDisruptionBudget: { enabled: true, minAvailable: 2 }
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels: { app: bookings-api }

values-pre.yaml sits in the middle: 2 replicas, the rc-2.4.0 image, a 512Mi limit, an HPA from 2 to 5 and the test gateway. Three 30-line values files have replaced 120 duplicated manifests.

`--set` versus `-f`

Aspect	`-f file.yaml`	`--set key=value`
Versionable in Git and reviewable in a PR	Yes	No
Reproducible	Yes	Only if somebody remembers the exact command
Complex structures	Natural	Bracket syntax, painful
Types	Explicit	Inferred, with surprises (`--set-string` to force text)

The Rutas Norte team's rule: -f for everything that defines an environment; --set only for the image tag injected by the ci-rutasnorte pipeline, which is the one thing that legitimately changes on every deployment. There is also --set-file key=./file to dump a file's contents in as a value.

The life cycle of a release

# THE COMMAND YOU SHOULD ALWAYS USE: installs if absent, upgrades if present.
# Idempotent, and therefore fit for automation.
helm upgrade --install rutas-norte ./rutas-norte-chart \
  -n rutas-norte-pro --create-namespace \
  -f environments/values-pro.yaml \
  --atomic --timeout 10m

A plain helm install fails if the release exists; a plain helm upgrade fails if it does not. upgrade --install covers both cases.

Flag	What it does	Why it matters
`--wait`	Waits for the pods to be `Ready` and the Services to have endpoints	Without this, `upgrade` reports success as soon as the API accepts the objects, even if the pods are in `CrashLoopBackOff`
`--timeout 10m`	How long it waits (5m by default)	A large StatefulSet or a slow migration needs more
`--atomic`	Implies `--wait`, and if it fails or times out it rolls back automatically	The production safety net: either it works, or the cluster is left as it was
`--cleanup-on-fail`	Deletes new objects from a failed upgrade	Avoids leaving rubbish behind
`--force`	Deletes and recreates the objects instead of patching them	Dangerous: it causes an outage
`--reuse-values`	Reuses the previous values and only applies the new ones	Convenient and treacherous: it hides what is actually applied

In production, --atomic --timeout are not optional. Without them, a deployment with a mistyped image leaves rutas-norte-pro with pods in ImagePullBackOff and Helm saying "deployed".

Inspecting the state

helm list -n rutas-norte-pro        # releases in the namespace
helm list -A --all                  # all of them, including the failed ones
helm history rutas-norte -n rutas-norte-pro

REVISION  UPDATED                    STATUS      CHART                    APP VERSION  DESCRIPTION
5         2026-08-01 16:45 CEST      superseded  rutas-norte-chart-1.4.0  2.4.0        Upgrade complete
6         2026-08-04 10:11 CEST      failed      rutas-norte-chart-1.4.0  2.4.1        Upgrade failed
7         2026-08-05 09:14 CEST      deployed    rutas-norte-chart-1.4.0  2.4.0        Rollback to 5

That table tells a complete story: revision 6 failed and revision 7 was a rollback to 5.

helm get values rutas-norte -n rutas-norte-pro         # the values YOU supplied
helm get values rutas-norte -n rutas-norte-pro --all   # including the defaults
helm get manifest rutas-norte -n rutas-norte-pro       # what is applied RIGHT NOW
helm get manifest rutas-norte -n rutas-norte-pro --revision 5

helm get manifest is the command that answers "what is actually deployed in production?", the question nobody could answer at the start of this module.

Rollback and uninstall

helm rollback rutas-norte -n rutas-norte-pro           # to the previous one
helm rollback rutas-norte 5 -n rutas-norte-pro --wait  # to a specific one

A rollback creates a new revision (revision 8 would be a copy of 5): the history never goes backwards, only forwards.

A critical limitation: the rollback affects the Kubernetes objects, not the data. If revision 6 ran a migration that added a column to bookings-postgres, the rollback restores the code but does not undo the migration. Schema migrations have to be backwards-compatible.

helm uninstall rutas-norte -n rutas-norte-dev                  # deletes objects and history
helm uninstall rutas-norte -n rutas-norte-dev --keep-history   # keeps the history
helm list -n rutas-norte-dev --uninstalled                     # list them
helm rollback rutas-norte 7 -n rutas-norte-dev                 # and resurrect it

--keep-history is the prudent option when you uninstall something in an important environment and you are not entirely sure. Watch out for PVCs: Helm does not delete the ones created by a StatefulSet's volumeClaimTemplates (06-01), so bookings-postgres keeps its data. That is good (it protects against accidental deletion) and confusing (reinstalling reuses the old data).

Debugging: template, lint, dry-run and diff

# Render locally, without touching the cluster or needing a connection
helm template rutas-norte ./rutas-norte-chart -f environments/values-pro.yaml

# A single template
helm template rutas-norte ./rutas-norte-chart -f environments/values-pro.yaml \
  -s templates/bookings-api/deployment.yaml

# Compare two environments: answers "how exactly do they differ?"
diff <(helm template rn ./rutas-norte-chart -f environments/values-pre.yaml) \
     <(helm template rn ./rutas-norte-chart -f environments/values-pro.yaml)

# Chain it with real server-side validation (01-06): schema + admission
# webhooks (Kyverno, from 08-03), without applying anything
helm template rutas-norte ./rutas-norte-chart -f environments/values-pro.yaml | \
  kubectl apply --dry-run=server -f - -n rutas-norte-pro

Unlike helm template, the --dry-run --debug variant does contact the cluster, so .Capabilities is real, it validates against the API and it shows the computed values. It is the mandatory step before any production deployment.

helm lint checks the structure, that Chart.yaml is complete and that the templates render. Put it in ci-rutasnorte as a mandatory check on any change to the chart.

`helm diff`: the safety net

By some distance the most valuable tool in this section. It is a plugin, not included by default:

helm plugin install https://github.com/databus23/helm-diff
helm diff upgrade rutas-norte ./rutas-norte-chart -n rutas-norte-pro -f environments/values-pro.yaml

rutas-norte-pro, rutas-norte-api, Deployment (apps) has changed:
        metadata:
          annotations:
-           checksum/config: 8f2a91bc4d7e...
+           checksum/config: c3e7d92a1b8f...
          containers:
          - name: api
-           image: registry.rutasnorte.example/bookings-api:2.4.0
+           image: registry.rutasnorte.example/bookings-api:2.4.1
            resources:
              limits:
-               memory: 1Gi
+               memory: 2Gi

It compares what is in the cluster with what you are about to apply, line by line. It is what stops you deploying a resource change you did not want, or lets you notice that upgrading a third-party chart changes thirty unexpected things.

Production rule: nobody runs helm upgrade against rutas-norte-pro without first pasting the helm diff output into the change request. It is the Helm equivalent of kubectl diff (01-06).

When a template will not compile, the most frequent causes are, in order: indent where nindent belonged, a nil value that does not exist in values.yaml, a string with a : and no quotes, and a {{- if }} without its {{- end }}. Isolate the suspect template with -s and use --debug.

Dependencies and subcharts

Rutas Norte needs redis-cache. Instead of maintaining its StatefulSet, we can use an existing chart, declared in Chart.yaml (section 4) with condition: redis.enabled, tags to enable groups and alias to rename it.

helm dependency update ./rutas-norte-chart   # resolves, downloads and REWRITES Chart.lock
helm dependency build  ./rutas-norte-chart   # downloads EXACTLY what Chart.lock says

Chart.lock goes into Git; charts/*.tgz does not. It is exactly the logic of package-lock.json. In the pipeline, always build; on your machine when you want to move a version up, update.

Passing values to a subchart

A subchart's values are nested under its name (or its alias), and global is visible from the parent chart and from every subchart:

global:
  imageRegistry: registry.rutasnorte.example
  storageClass: rutas-norte-ssd

redis:                     # <- the subchart's name
  enabled: true
  architecture: replication
  auth: { enabled: true, existingSecret: redis-cache-credentials }
  master:
    persistence: { storageClass: rutas-norte-ssd, size: 8Gi }
  replica: { replicaCount: 2 }

Most serious charts (Bitnami among them) honour global.imageRegistry and global.storageClass.

The trap of overriding a subchart's values

Three rules you need to internalise:

Rule 1: the parent chart wins, but only if it gets the path right. If the subchart expects master.persistence.size and you write redis.persistence.size, Helm says nothing: you can write redis.diskSize: 500Gi and absolutely nothing happens. Always verify with helm template -s charts/redis/templates/... that the change took effect.

Rule 2: -f overrides, it does not merge lists. If values.yaml has extraFlags: ["--appendonly yes", "--maxmemory 512mb"] and values-pro.yaml sets extraFlags: ["--maxmemory 2gb"], the result is just ["--maxmemory 2gb"]. --appendonly yes was lost. Maps merge key by key; lists are replaced wholesale. An inexhaustible source of surprises.

Rule 3: a subchart cannot read the parent's values except through global.

Hooks

A hook is a Kubernetes object Helm creates at a specific moment in the life cycle, outside the normal flow: pre-install, post-install, pre-upgrade, post-upgrade, pre-rollback, post-rollback, pre-delete, post-delete and test.

The real case: a database schema migration

bookings-api 2.4.0 needs a new column in bookings-postgres. If we deploy the code before migrating, the pods fail. The solution is a pre-upgrade hook that blocks the deployment until the migration finishes.

apiVersion: batch/v1
kind: Job
metadata:
  # The name includes the revision: every upgrade creates a distinct Job
  name: {{ include "rutas-norte.fullname" . }}-migration-{{ .Release.Revision }}
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    # Lower weights go first: this lets you chain hooks
    "helm.sh/hook-weight": "-5"
    # before-hook-creation: deletes the previous one before creating the new one
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrator
          image: {{ .Values.imageRegistry }}/bookings-api-migrations:{{ .Chart.AppVersion }}
          command: ["/app/migrate", "--up-to", "{{ .Chart.AppVersion }}"]
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef: { name: {{ .Values.bookingsApi.existingSecret }}, key: db-password }

sequenceDiagram
    participant U as Operator
    participant H as Helm
    participant K as Cluster
    U->>H: helm upgrade --install --atomic
    H->>K: creates the pre-upgrade Job (migration)
    alt Migration succeeds
        K-->>H: Job Complete
        H->>K: applies Deployment, Service, HPA, ConfigMap
        H->>K: waits for the pods to be Ready (--wait)
        H-->>U: release deployed (revision N+1)
    else Migration fails
        K-->>H: Job Failed
        H->>H: aborts: does NOT touch the application objects
        H-->>U: error; the cluster stays on revision N
    end

The important part: if the migration fails, the bookings-api pods never notice. They carry on serving the previous version. Without the hook, we would have deployed new code against an old schema.

Deletion policies

before-hook-creation (the default) deletes the previous resource just before creating the new one; hook-succeeded deletes it if it ends well; hook-failed deletes it if it fails. They can be combined with commas.

Firm advice: do not use hook-failed. When a migration fails at three in the morning, the only thing you will want is kubectl logs from the Job that failed. If the policy deleted it, you are left with nothing.

Test hooks

apiVersion: v1
kind: Pod
metadata:
  name: "{{ include "rutas-norte.fullname" . }}-api-test"
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  restartPolicy: Never
  containers:
    - name: curl
      image: curlimages/curl:8.10.1
      args: ["-sS", "--fail", "--max-time", "10",
             "http://{{ include "rutas-norte.fullname" . }}-api/health/ready"]

helm test rutas-norte -n rutas-norte-pre --logs

It is an excellent smoke test to run automatically after deploying to rutas-norte-pre.

Packaging and publishing

helm package ./rutas-norte-chart              # -> rutas-norte-chart-1.4.0.tgz

# Sign it with GPG, complementing the image signing from 08-05
helm package ./rutas-norte-chart --sign --key '[email protected]'
helm verify rutas-norte-chart-1.4.0.tgz

Since Helm 3.8, a container registry can host charts directly. Rutas Norte already has registry.rutasnorte.example; it needs no other server.

helm registry login registry.rutasnorte.example -u ci-rutasnorte
helm push rutas-norte-chart-1.4.0.tgz oci://registry.rutasnorte.example/charts

# Consuming it: note that NO 'helm repo add' is needed
helm upgrade --install rutas-norte \
  oci://registry.rutasnorte.example/charts/rutas-norte-chart \
  --version 1.4.0 -n rutas-norte-pro -f environments/values-pro.yaml --atomic

Advantages of OCI over the classic HTTP repository (helm repo index over a statically served directory): the same credentials and access control as the images, no index.yaml to maintain, versioning by digest, and the same security scanning you already have in place (08-06).

What Helm does badly

Helm solves a real problem, but it has honest shortcomings worth knowing about.

1. It is a text engine, it does not understand YAML. It does not manipulate structures: it concatenates strings and at the end tries to parse the result. That is where nindent, the {{- and the fact that one extra space breaks the deployment all come from. On top of that, the templates are not valid YAML: you cannot open them and have your editor validate the Kubernetes schema, nor apply them with kubectl apply -f. You lose the whole toolchain that exists around YAML.

2. Complex templates become unreadable. Large third-party charts have 300-line templates with six levels of conditionals. Debugging them is painful.

3. CRDs are a problem. Helm installs whatever is in crds/ but never updates or deletes it (the stated reason: deleting a CRD removes all of its resources, which would be catastrophic). Consequence: upgrading cert-manager from 1.15 to 1.16 may require a manual kubectl apply -f cert-manager.crds.yaml before the helm upgrade.

4. The state lives in the cluster, not in Git. If somebody runs kubectl scale by hand, Helm does not detect it until the next upgrade, and then it overwrites it without warning. There is no continuous reconciliation: it only acts when somebody runs a command. That is exactly the gap GitOps fills (10-05).

5. Depending on somebody else's chart means depending on somebody else's decisions. If you need to change something the author did not parameterise, your options are opening a request with the project, forking the chart or post-processing the output. None of them is comfortable.

Why many people prefer Kustomize

Criticism of Helm	Kustomize's answer
The templates are not valid YAML	The base manifests are ordinary YAML, applicable with `kubectl`
`nindent`, `{{-`, whitespace errors	There is no template engine
State in the cluster	Stateless: it only generates YAML
You have to learn Go templates and Sprig	Just one file, `kustomization.yaml`
An external tool	It ships inside kubectl

And where Helm still clearly wins: the ecosystem (thousands of ready-made charts), packaging and distribution (a versioned, publishable .tgz), life cycle management (rollback, history, an uninstall that knows what to delete), the hooks and real conditional logic with loops.

The industry's practical conclusion, and the one we will adopt at Rutas Norte: Helm for consuming third-party software, Kustomize for our own applications. cert-manager and Prometheus via Helm; web-store, bookings-api and company via Kustomize.

Common Mistakes and Tips

1. Not pinning --version. You install cert-manager today and get 1.16.1; you reinstall in March and get 1.19, with breaking changes. Always pin the version and record it in the repository.

2. Putting secrets in values.yaml. The file goes into Git. Use existingSecret pointing at a Secret managed by another system (Sealed Secrets, SOPS, External Secrets Operator: 03-02 and 10-05).

3. Confusing indent with nindent. The most frequent syntax error. Use {{- toYaml . | nindent N }} as your default pattern.

4. Declaring fixed replicas with an active HPA. Every helm upgrade returns the Deployment to the chart's replica count, cancelling the autoscaling until the HPA reacts. Make the field conditional, as in section 6.

5. helm upgrade without --atomic in production. A failed deployment leaves the cluster half-done and the release in failed. With --atomic, it returns to the previous state on its own.

6. Overusing --reuse-values. It is convenient until nobody knows which values are actually applied. Always use a complete, version-controlled -f file.yaml.

7. Writing subchart values with the wrong path. Helm ignores them without saying a word. Check with helm template -s that the change landed.

8. Believing that lists merge. They do not: they are replaced wholesale. If you redefine extraFlags in values-pro.yaml, you lose the ones from the base values.yaml.

9. Changing the selector labels between chart versions. A Deployment's spec.selector is immutable (02-03). If your selectorLabels helper changes, the upgrade fails with "field is immutable" and you have to delete and recreate the object, with an outage. Keep selectorLabels minimal and stable for ever.

10. Upgrading a third-party chart without helm diff. A single minor version jump can change thirty fields. Look at them first.

11. hook-delete-policy: hook-failed on migrations. It deletes precisely the Job whose logs you need.

12. Forgetting helm dependency update after editing Chart.yaml. The deployment uses the old charts/ and you cannot work out why nothing changes.

13. Uninstalling without --keep-history in an important environment. No history means no rollback. And remember that the PVCs survive, which can surprise you on reinstall.

Exercises

Exercise 1: templating `notifications-worker`

Add the notifications-worker component to the chart: a Deployment with no Service (it receives no traffic), with a conditional enabled, parameterised resources and replicas, and the common labels via include. It must have a KEDA ScaledObject (09-04) only if worker.keda.enabled is true, and in that case the Deployment must not declare replicas. Write the three per-environment value blocks as well.

Exercise 2: subchart, global values and the list trap

Add redis as a dependency with a condition, configure it to use the rutas-norte-ssd StorageClass via a global value, and demonstrate with helm template that the value arrived. Then define master.extraFlags in values.yaml with two flags and override it in values-pro.yaml with a single one; check what comes out and explain the result.

Exercise 3: a migration hook with a prior check

Write two hooks chained by weight: a pre-upgrade one with weight -10 verifying that bookings-postgres accepts connections, and another with weight -5 that runs the migration. If the first fails, the second must not run and the deployment must abort without touching the application. Explain how you would verify all this without breaking rutas-norte-pro.

Solutions

Solution 1

# templates/worker/deployment.yaml
{{- if .Values.worker.enabled }}
{{- $w := .Values.worker }}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "rutas-norte.fullname" . }}-worker
  labels: {{- include "rutas-norte.labels" . | nindent 4 }}
spec:
  {{- if not $w.keda.enabled }}
  replicas: {{ $w.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "rutas-norte.selectorLabels" . | nindent 6 }}
      app: notifications-worker
  template:
    metadata:
      labels:
        {{- include "rutas-norte.labels" . | nindent 8 }}
        app: notifications-worker
    spec:
      securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: worker
          image: {{ .Values.imageRegistry }}/notifications-worker:{{ $w.image.tag | default .Chart.AppVersion }}
          envFrom: [{ secretRef: { name: {{ $w.existingSecret }} } }]
          resources: {{- toYaml $w.resources | nindent 12 }}
{{- end }}

# templates/worker/scaledobject.yaml
{{- if and .Values.worker.enabled .Values.worker.keda.enabled }}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ include "rutas-norte.fullname" . }}-worker
spec:
  scaleTargetRef: { name: {{ include "rutas-norte.fullname" . }}-worker }
  minReplicaCount: {{ .Values.worker.keda.minReplicas }}
  maxReplicaCount: {{ .Values.worker.keda.maxReplicas }}
  triggers:
    - type: redis
      metadata:
        address: {{ .Values.worker.keda.redisAddress }}
        listName: email-queue
        listLength: {{ .Values.worker.keda.listLength | quote }}
{{- end }}

# values.yaml (defaults)
worker:
  enabled: true
  replicaCount: 1
  existingSecret: worker-credentials
  image: { tag: "" }
  resources: { requests: {cpu: 50m, memory: 64Mi}, limits: {memory: 256Mi} }
  keda: { enabled: false, minReplicas: 0, maxReplicas: 10,
          listLength: 20, redisAddress: "redis-cache:6379" }
# dev: worker.keda.enabled: false
# pre: worker.keda: { enabled: true, minReplicas: 0, maxReplicas: 5 }
# pro: worker.keda: { enabled: true, minReplicas: 1, maxReplicas: 30 }
#      worker.resources.requests: { cpu: 200m, memory: 256Mi }

Solution 2

# Chart.yaml
dependencies:
  - { name: redis, version: "20.1.0",
      repository: "https://charts.bitnami.com/bitnami", condition: redis.enabled }

# values.yaml
global: { storageClass: rutas-norte-ssd }
redis:
  enabled: true
  master: { extraFlags: ["--appendonly yes", "--maxmemory 512mb"] }
---
# values-pro.yaml
redis:
  master: { extraFlags: ["--maxmemory 4gb"] }

helm dependency update ./rutas-norte-chart
helm template rn ./rutas-norte-chart -f environments/values-pro.yaml \
  -s charts/redis/templates/master/statefulset.yaml | grep -E 'storageClassName|maxmemory|appendonly'

        storageClassName: rutas-norte-ssd
        - --maxmemory 4gb

Explanation: global.storageClass arrived because the Bitnami chart explicitly honours it. --appendonly yes vanished: lists do not merge, they are replaced wholesale. To keep both you have to repeat the two of them in values-pro.yaml.

Solution 3

# templates/hooks/00-verify-db.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "rutas-norte.fullname" . }}-verify-db-{{ .Release.Revision }}
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "-10"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 120
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: verify
          image: postgres:16.4
          command: ["sh","-c","pg_isready -h bookings-postgres -p 5432 -t 10"]

The second hook is the migration Job from section 11, with "helm.sh/hook-weight": "-5".

How it works: Helm orders the hooks of the same event by ascending weight and waits for each one to finish before launching the next. If the one weighted -10 fails, the -5 one is never created and the upgrade aborts without touching Deployments or Services.

Verifying it without breaking production:

# 1. Check that they are generated and in what order
helm template rn ./rutas-norte-chart -f environments/values-pro.yaml | grep -B2 hook-weight

# 2. Try the complete flow in pre
helm upgrade --install rutas-norte ./rutas-norte-chart \
  -n rutas-norte-pre -f environments/values-pre.yaml --atomic --timeout 10m

# 3. Trigger the failure on purpose: point at a non-existent host
helm upgrade rutas-norte ./rutas-norte-chart -n rutas-norte-pre \
  -f environments/values-pre.yaml --set db.host=does-not-exist --atomic
kubectl get jobs -n rutas-norte-pre     # only verify-db should exist, NOT migration
helm history rutas-norte -n rutas-norte-pre   # the revision shows up as failed
kubectl get deploy -n rutas-norte-pre -o wide # the image has NOT changed

Conclusion

Helm has turned Rutas Norte's 120 files into a chart with twenty templates and three values files. The essentials:

A chart is the package, a release is its named installation, and Helm stores state in the cluster as release Secrets: that is what makes history, rollback and uninstall possible.
Consuming third-party charts with judgement means pinning --version, reading helm show values before installing, and keeping your values in a version-controlled file. That is what we now know we were doing with cert-manager and kube-prometheus-stack.
Templates combine .Values, .Release, .Chart and .Capabilities with functions (default, quote, toYaml, nindent, required, sha256sum) and control flow (if, with, range); define/include in _helpers.tpl centralise the project's labels.
The professional life cycle is helm upgrade --install ... --atomic --timeout, with -f instead of --set, and with helm diff as a mandatory step before touching rutas-norte-pro.
Hooks order database migrations relative to the deployment, with the enormous benefit that a failed migration never touches the application.
And Helm has real shortcomings: it is a text engine, the templates stop being valid YAML, CRDs are not updated, and the state lives in the cluster rather than in Git.

That last point opens the two lessons that follow. The next one, Kustomize, attacks the templating criticism from the opposite end: no template engine, no state, with manifests that remain valid YAML and are customised through overlays. We will look at its base-and-overlay model, its ConfigMap generators with automatic hashing — which cleanly solves the restart-on-configuration-change problem from 03-03 — and we will migrate Rutas Norte's entire k8s/. At the end we will compare the two tools honestly and see how to combine them.

Helm

Contents