We closed module 2 with a list of debts, and the first of them was this one: the API URL is baked into the web-store image. Every time we want to deploy the store to rutas-norte-pre instead of rutas-norte-pro we have to build a different image, with the same code and a single character changed in a configuration file. That breaks the most basic promise of the container: build once, deploy anywhere. The object that fixes it is the ConfigMap, a store of non-confidential configuration that lives in the cluster, is versioned in Git alongside the rest of the manifests and is injected into the container at start-up. In this lesson you will build the web-store ConfigMap with its real nginx.conf and mount it as a file inside the container, which is the most powerful way to consume it and the one that produces the fewest surprises. Injection as environment variables is covered in depth in Environment Variables; here we mention it only to place it on the map.

1. Why configuration is separated from code
2. Anatomy of the ConfigMap object
3. The four ways to create a ConfigMap
4. The three ways to consume it (and which one we cover here)
5. Mounting a ConfigMap as a volume
6. Selecting specific keys with items and defaultMode
7. subPath: what it is for and its two dangers
8. The central case: the nginx.conf of web-store
9. Updating a ConfigMap and propagation to volumes
10. Immutable ConfigMaps
11. The 1 MiB limit and what to do if you exceed it
12. Per-environment ConfigMaps in Rutas Norte

The twelve-factor methodology (The Twelve-Factor App), written in 2011 for cloud applications, devotes its third factor to exactly this problem, and its statement is a single sentence:

Store configuration in the environment. Configuration is everything that can vary between deployments; code is what does not vary.

The practical test it proposes is blunt, and it is worth applying it to Rutas Norte: could you make the code repository public right now without leaking a single credential? Today the answer is no, because the bookings-postgres password is written in a manifest in the repository. That part is settled by the Secrets lesson. But there is a second, quieter test: is it exactly the same image that runs in rutas-norte-dev, in rutas-norte-pre and in rutas-norte-pro? Today it is not, either.

This is not purism. It has very concrete consequences:

The two concrete cases Rutas Norte is dragging along:

1. The API URL in web-store. The SPA needs to know which address to call: http://bookings-api inside the cluster, https://api.rutasnorte.example from the customer's browser. Today it is baked into the compiled JavaScript inside the image.
2. The nginx.conf of web-store. The server configuration — compression, security headers, the fallback rule to index.html that every SPA needs, the proxy towards bookings-api — is baked into the image. Tweaking the cache of the static assets requires rebuilding it.

Both are solved today, in this lesson.

A ConfigMap is one of the simplest objects in Kubernetes. It has neither spec nor status: only metadata and data.

The points that define the object:

• apiVersion: v1: it belongs to the core group, like Pod, Service or Namespace. It carries no group prefix.
• It has no spec. A ConfigMap does not describe a desired state that a controller has to reconcile; it is a passive data container. That is also why it has no status.
• data contains valid UTF-8 text. The keys are potential file names, so they must be limited to alphanumeric characters, -, _ and ..
• binaryData contains data that is not text, base64-encoded when written into the manifest. Kubernetes decodes it when mounting. It is useful for a certificate in DER format, an .ico, a font, a small .tar.gz of templates.
• The same key cannot appear in both data and binaryData: the apply fails.
• It is namespaced. A pod can only mount ConfigMaps from its own namespace. There is no native way to reference one from another namespace, and that restriction is what makes the "one ConfigMap per environment" pattern of section 12 feel natural.

One important detail about multi-line keys. YAML's | is a literal block: it preserves line breaks as they are. That is what you want for a configuration file. Its cousin > (folded block) turns single line breaks into spaces, and would ruin an nginx.conf. For files, always use |.

This is the canonical form and the only one that versions well in Git. You have already seen it above. It is applied like any other object:

For short values, above all when you are exploring or generating the YAML with --dry-run.

Let us check what it created exactly:

Notice that it carries no labels at all. Objects created with kubectl create do not follow the Rutas Norte labelling scheme, and that leaves them out of every query with -l app.kubernetes.io/part-of=rutas-norte. That is why the recommended use of the imperative command is as a YAML generator, not as a way of creating objects:

You then edit the labels by hand and apply the file. It is the same --dry-run=client -o yaml flow from module 1, now applied to configuration.

Here is the interesting case. You have the real nginx.conf in your repository and you want to put it into a ConfigMap without copying it by hand into a YAML (with the risk of ruining the indentation).

The key is the base name of the file (nginx.conf), not the full path. If you want the key to have a different name, you prepend it with an =:

The key will be default.conf even though the file on disk is called nginx-production.conf. This trick is indispensable when the name the application expects does not match the one that is convenient in the repository.

If you point at a directory, each file inside it becomes a key. It is not recursive: subdirectories are ignored.

Three files, three keys. When we mount this ConfigMap in /etc/nginx/conf.d the three files will appear with those same names, which is exactly what nginx expects.

This is the one that confuses people most. It takes a file in KEY=value format and creates one key per line, not one key holding the whole file.

Compare that with what --from-file would have done to the same file: a single key called web-store.env whose value would be the entire text. The difference is enormous and it is the source of a classic mistake.

Rules of the --from-env-file format: blank lines and lines starting with # are ignored; quotes are not accepted around the value (they would be stored as part of the value), nor are line continuations; and the keys must be valid environment variable names.

This lesson focuses on the second one because it is the one that solves the nginx.conf case — a 40-line file does not fit comfortably into an environment variable — and because it is the only one that lets you change configuration without recreating pods.

The mechanism always has two halves that must line up: a volume at pod level that declares where the data comes from, and a volumeMount at container level that declares where it appears. The bridge between the two is the name field.

What happens exactly when that pod starts:

1. The kubelet reads the pod and sees a volume of type configMap.
2. It asks the API for the web-store-nginx ConfigMap in the rutas-norte-dev namespace. If it does not exist, the pod stays in ContainerCreating until it appears, with a FailedMount event. It does not fail for ever: as soon as you create the ConfigMap, the pod starts.
3. It creates a directory on the node and writes one file per key of the ConfigMap inside it.
4. It mounts that directory at /etc/nginx/conf.d inside the container.

And here is the point that surprises everyone the first time: the mount replaces the contents of the container's directory. Whatever the nginx image brought in /etc/nginx/conf.d (the default default.conf file) stops being visible. It is not deleted from the image; it is simply hidden under the mount point, just as in Linux when you mount a disk over a directory that already had files.

Let us see it:

This output contains all the magic of the mechanism, so it is worth reading slowly:

• The files the application sees (compression.conf, etc.) are symbolic links to ..data/.
• ..data is itself a symbolic link to a timestamped directory.
• The real directory contains the actual files.

Why so much indirection? Because it allows an atomic update. When the ConfigMap changes, the kubelet writes a new directory with a new timestamp and then moves the ..data link in one go. The application never sees a half-finished state with half the files old and half new. It is an implementation detail, but it explains two behaviours you will see later on: why the update works, and why it does not work with subPath.

By default all keys are projected. Often that is not what you want: the same ConfigMap can hold keys destined for environment variables (API_URL) and keys that are files (nginx.conf), and mounting them all together would fill /etc/nginx/conf.d with rubbish that nginx would try to interpret.

With items you choose which keys are projected and under which file name:

Details that matter:

• path is relative to mountPath. With mountPath: /etc/nginx/conf.d and path: default.conf, the file ends up at /etc/nginx/conf.d/default.conf.
• path accepts subdirectories: path: extra/headers.conf creates the subdirectory. What it does not accept is starting with / or containing ...
• If you use items, only what you list is projected. Keys not mentioned disappear from the volume.
• If a key listed in items does not exist in the ConfigMap, the pod does not start. This is desirable behaviour: it fails early and with a clear message instead of starting nginx with no configuration.
• defaultMode is written in octal and must carry the leading 0 (0444). If you write 444 in decimal, YAML reads it as the number 444, which in octal is 0674: absurd permissions. It is a real and frequent mistake.
• The mode of a specific item takes precedence over defaultMode.

A warning about permissions: if the container runs as a non-root user (the usual and recommended thing, as you will see in Security Contexts), a defaultMode: 0400 means "read only for the owner", and the owner of the file is root. The process would not be able to read it. For that case, 0444 is the safe choice.

Go back to the problem from section 5: mounting a volume over a directory hides what was already there. What if you only want to add or replace a single file inside a directory that must keep the rest of its contents?

The canonical case in Rutas Norte is /etc/nginx/nginx.conf. That file lives in /etc/nginx/, a directory that also contains mime.types, fastcgi_params, the conf.d directory... If we mount the volume at /etc/nginx/, nginx loses everything else and does not start.

The solution is subPath:

With this, only the file /etc/nginx/nginx.conf is replaced; the rest of /etc/nginx/ stays intact.

Everything in its place. Now the two dangers, which you need to know before using it:

Danger 1: with subPath the update does NOT propagate. When you mount with subPath, the kubelet copies the file directly instead of creating the chain of symbolic links from section 5. With no ..data link to move, the file is frozen at the value it had when the pod started. You can edit the ConfigMap a hundred times: the container will keep seeing the original until you recreate it. It is by far the number one cause of "I changed the configuration and nothing happens".

Danger 2: it is easy to get the direction wrong. mountPath must point at the destination file and subPath at the source key. If you swap them, or if mountPath points at an existing directory, you will get a directory where you expected a file and the application will fail with a confusing message.

The practical recommendation: avoid subPath if you can. In the nginx case you have a better alternative, and it is the one we will use: mount at /etc/nginx/conf.d/, the directory the image itself reserves for added configuration, without touching /etc/nginx/nginx.conf. That directory is designed to be replaced wholesale.

Let us settle the module 2 debt with everything above. We create the file k8s/base/web-store/configmap-nginx.yaml:

And we modify the web-store Deployment we already had:

We apply and check:

We verify that nginx is reading our configuration and not the one from the image:

nginx -T dumps the effective configuration after resolving every include. Our try_files showing up confirms that the SPA rule is active. And the definitive proof, from an ephemeral pod:

The three security headers are there. The nginx configuration no longer lives in the image: it lives in a YAML file reviewable in a pull request.

Now for the payoff. We add one more header without rebuilding the image and without restarting anything.

We wait and look inside the container:

The file has changed inside a container that has not restarted. The three pods still show RESTARTS 0.

About timings and conditions:

• Propagation is not instantaneous. The kubelet refreshes volumes on its sync cycle, governed by --sync-frequency (1 minute by default), plus the cache time of its API client (configMapAndSecretChangeDetectionStrategy, Watch by default). In practice count on up to a couple of minutes, not seconds.
• It propagates only if the volume does not use subPath (section 7).
• It propagates only if the ConfigMap is not immutable (section 10).
• And, above all: the file changing does not mean the application notices. nginx read its configuration at start-up and holds it in memory.

This last point is fundamental. There are three kinds of application:

For nginx we can send it the reload signal, which does not cut any connection in flight:

For the general case, the clean solution is to force a new rollout:

This replaces the pods with the RollingUpdate strategy from module 2, with no service interruption. There is a more elegant and automatic technique — storing a hash of the ConfigMap in an annotation on the template, so that changing the configuration triggers the rollout by itself — which you will see in Environment Variables, where it is even more necessary.

Since Kubernetes 1.21, a ConfigMap can be declared immutable:

What you gain:

1. Performance and scale. This is the main reason. Each kubelet keeps a watch open against the apiserver for every ConfigMap it mounts. In a large cluster that means thousands of connections looking out for changes. If the object is immutable, the kubelet closes the watch: there is nothing to watch. The load on the apiserver drops noticeably.
2. Protection against accidental change. Nobody can modify the production configuration by mistake in one stroke.
3. Determinism. A pod that starts today and another that starts a month from now with the same ConfigMap will have exactly the same configuration. With a mutable ConfigMap, two replicas of the same Deployment could hold different contents if the ConfigMap changed between their start-ups.

What you lose: the live update from section 9. And there are two hard rules:

• Immutability cannot be reverted. Once immutable: true, you cannot go back to false. The only option is to delete the object and recreate it.
• metadata can still be changed (labels, annotations). What is frozen is data and binaryData.

Trying to modify it produces a very clear error:

The pattern that goes with immutability is versioning the name: web-store-nginx-v3, -v4, and so on. The Deployment points at the specific version, and changing configuration means creating a new ConfigMap and updating the reference in the Deployment, which triggers a rollout with history and rollback. It is exactly what the Kustomize generators do by adding a suffix with the hash of the contents.

Recommendation for Rutas Norte: mutable in dev (iterate fast with nginx -s reload), immutable with a versioned name in pro (determinism and rollback).

A ConfigMap cannot exceed 1 MiB, counting the sum of data and binaryData plus the metadata. The limit is not arbitrary: ConfigMaps are stored in etcd, which imposes a default maximum size of 1.5 MiB per value, and etcd is the heart of the cluster.

When you go over, the error is immediate:

Or else, when the validation is done by the object itself:

Bear in mind as well that base64 inflates by 33%: an 800 KiB .ico takes up about 1.07 MiB in binaryData and does not fit.

What to do if you get close to the limit:

The rule of thumb: if you are approaching a megabyte, you are almost certainly putting something into a ConfigMap that is not configuration. In Rutas Norte this would apply to the full catalogue of routes and stops, which is business data and lives in bookings-postgres, not in a ConfigMap.

Remember the restriction from section 2: a ConfigMap is namespaced and a pod can only mount those from its own namespace. Since the Rutas Norte environments are namespaces, this fits naturally: one ConfigMap per environment, with the same name in each one.

The name of the ConfigMap is identical in all three namespaces. That lets the Deployment be literally the same file in k8s/base/, and the only thing that changes between environments is the ConfigMap in k8s/environments/<environment>/. The repository structure ends up like this:

And this is the concrete content of the Rutas Norte configuration per environment. Take this table as a reference; in 03-03 we will extend it to the full catalogue for every component.

Note two design decisions:

• REDIS_TTL_SECONDS grows as you get closer to production. In development you want a short cache so you can see your changes; in production, with the traffic peaks of bank-holiday weekends and school holidays, you want a long cache that takes load off bookings-postgres.
• MAINTENANCE_MODE is an operational switch. Setting it to true and reloading nginx shows the "we are updating" page without deploying anything. It is exactly the kind of lever that justifies separating configuration from code.

And one thing that is not in this table and never should be: the bookings-postgres password. A ConfigMap offers no protection whatsoever: anyone who can list ConfigMaps in the namespace reads its contents in the clear. That debt is settled by the next lesson.

Five tips that save hours:

1. Label your ConfigMaps with the same scheme as everything else. kubectl get cm -l app=web-store -A must work.
2. Always check from inside the container, not from the YAML: kubectl exec ... -- cat <path>. It is the only truth.
3. Document every key with a comment in the YAML. The manifest is the documentation of the configuration.
4. One ConfigMap per purpose, not one giant one per application. Keep the one mounted as a file separate from the one feeding environment variables.
5. In production, immutable and versioned. Determinism is worth more than convenience.

bookings-api needs a /app/config/fare-rules.json file with the discount rules, which today lives inside its image.

1. Create the file fare-rules.json on your machine with this content:

2. Generate with kubectl (without writing YAML by hand) the manifest of a ConfigMap called bookings-api-fares in rutas-norte-dev from that file, and save it under k8s/environments/dev/.
3. Add the Rutas Norte labelling scheme to it and apply it.
4. Write the volumes and volumeMounts fragment that mounts it at /app/config in read-only mode.
5. Verify from inside the pod that the file exists and has the right content.

1. Deploy a web-store pod that mounts its configuration with subPath over /etc/nginx/nginx.conf.
2. Check that /etc/nginx/ still holds mime.types and the conf.d directory.
3. Modify the ConfigMap and wait three minutes. Check whether the file has changed inside the container.
4. Repeat the experiment mounting the same ConfigMap without subPath at /etc/nginx/conf.d/. Does it propagate now?
5. Explain in two lines the difference and what you see with ls -la in each case.

1. Create the web-store-config ConfigMap in the three namespaces with the values from the table in section 12.
2. Show with a single command that three objects exist with that name and that their LOG_LEVEL values differ.
3. Make only the one in rutas-norte-pro immutable and try to modify it. Copy the error.
4. Update the production REDIS_TTL_SECONDS knowing that the ConfigMap is immutable. Describe the complete sequence of steps.
5. Answer: if tomorrow you have to switch on maintenance mode in production at 3 in the morning, which exact command would you run?

Solution 1

The generated YAML, with the labels already added by hand:

The Deployment fragment:

And the verification:

One note: the ConfigMap key is fare-rules.json because --from-file takes the base name of the file, and the mountPath is the /app/config directory. The result is /app/config/fare-rules.json, the requested path.

Solution 2

With subPath:

The directory keeps everything. After modifying the ConfigMap and waiting:

It has not changed. The file still holds the value it had at start-up.

Without subPath, mounting at /etc/nginx/conf.d/:

This time it does. The difference is visible with ls -la:

Without subPath, the kubelet updates by writing a new directory and moving the ..data link. With subPath there is no link: the file was copied once and there it stays until the pod is recreated.

Solution 3

The check in a single command:

Three objects with the same name and different contents, exactly what allows the Deployment to be identical.

Now for production immutability:

To update REDIS_TTL_SECONDS in production while it is immutable, the correct sequence is:

1. Create a new ConfigMap with a versioned name, web-store-config-v2, with the new value and immutable: true.
2. Apply it.
3. Edit the rutas-norte-pro Deployment so that it points at web-store-config-v2, with kubernetes.io/change-cause documenting the reason.
4. kubectl rollout status to follow the rolling update from module 2.
5. Verify and keep the -v1 for a few days: if you have to run kubectl rollout undo, the Deployment will reference it again and it must exist.
6. Delete the -v1 once the rollback window has passed.

Note the advantage: production configuration now has history and rollback, just like the code.

For maintenance mode at 3 in the morning, with the mutable nginx ConfigMap:

No image rebuild, no deployment, no dropped connections. That is exactly the value of having configuration outside the code.

You have settled the first of the module 2 debts. You know why the third of the twelve factors demands that configuration live outside the code, and you have seen the concrete cost of not doing so: one image per environment, what was tested in pre differing from what was deployed to pro, and a timeout change turned into a full pipeline run. You know the anatomy of the ConfigMap — no spec, no status, data for text and binaryData for everything else, namespaced and with keys as potential file names — and its four creation methods, and you can tell apart the trap between --from-file and --from-env-file.

You have mastered consumption as a volume, which is the most versatile one: the volumes/volumeMounts pair, the fact that the mount hides the previous contents of the directory, the chain of symbolic links towards ..data that makes the update atomic, key selection with items, permissions with defaultMode in octal and how they interact with non-root users, and the two dangers of subPath, starting with the most expensive one: with subPath configuration never propagates. You know that propagation takes up to a couple of minutes and that the file changing is not the same as the application noticing, with the three strategies according to the kind of application. And you know about immutable ConfigMaps, what they gain in apiserver load and determinism, the versioned-name pattern that accompanies them, and the 1 MiB limit and the signal it sends: if you are getting close, it is not configuration.

On the ground, web-store now serves its nginx.conf from a ConfigMap mounted at /etc/nginx/conf.d, with security headers, compression, static-asset caching, the SPA try_files rule and a /healthz endpoint that the probes lesson will make use of. The three environments have their own ConfigMap with the same name, which has allowed the Deployment to be a single file in k8s/base/.

The big debt remains. Everything we have stored today is public: anyone with read permission in the namespace sees the full contents of a ConfigMap. The bookings-postgres password — the database that holds the name, ID number, phone number and email of every Rutas Norte customer — is still in the clear in a Git manifest. The next lesson, Secrets, takes it out of there: you will see how a Secret really differs from a ConfigMap, why base64 is not encryption and you will prove it by decoding one in three seconds, the Secret types and their uses, encryption at rest in etcd which by default is not enabled, how private images are pulled from registry.rutasnorte.example and what real solutions the ecosystem uses when the native Secret is not enough.