The two previous lessons built the stores: a ConfigMap with the web-store configuration and a Secret with the bookings-postgres credentials. In both you saw a single way of getting them into the container, the mounted file, and in both the other way was deferred: the environment variable, which is what appears in the vast majority of real manifests and the only thing third-party images such as postgres:16 or redis:7.2-alpine understand. This lesson covers it in full and unifies the previous two: how variables are declared, the four sources they can come from, how a pod can query data about itself with the Downward API, what happens when the same key arrives by two routes, why the shell does not expand what you think it does, and above all the limitation that conditions everything: an environment variable does not change while the process is alive. By the end you will have the complete variable catalogue for the six Rutas Norte components across the three environments.

1. What an environment variable really is inside a container
2. env with a literal value
3. envFrom: dumping a whole ConfigMap or Secret
4. valueFrom: taking a specific key
5. The Downward API: fieldRef
6. The Downward API: resourceFieldRef
7. Variable expansion with $(VAR)
8. Interaction with command and args
9. Precedence and collisions
10. The fundamental limitation: they do not refresh
11. The configuration hash in an annotation
12. Environment variable versus mounted file
13. The Rutas Norte variable catalogue

Before the syntax it is worth understanding the mechanism, because it explains almost every surprise.

When the kubelet starts a container, it builds a list of KEY=value pairs and hands it to the runtime (containerd), which in turn passes it to the kernel in the execve() call that launches the main process. From that point on:

• The process stores that list in its own memory (environ).
• Nobody from outside can modify it. There is no system call to change another process's environment. Not Kubernetes, not the kubelet, not kubectl.
• Child processes inherit it at the moment they are created.

Everything that follows can be deduced from this: why you have to restart the pod when a ConfigMap changes, why variables are visible in /proc/<pid>/environ, and why the content of a variable is always a text string (there are no numbers or booleans in a process's environment).

Let us check it:

There are three origins mixed into that list:

1. The ones coming from the image (PATH, HOME): defined by the Dockerfile or the base image.
2. The ones we declared ourselves (DB_HOST, LOG_LEVEL).
3. The ones Kubernetes injects automatically: HOSTNAME (the pod name) and the Service discovery variables (KUBERNETES_SERVICE_HOST, REDIS_CACHE_SERVICE_HOST...).

Those last ones are a relic of the early days of Kubernetes, inherited from Docker links. They have two serious limitations: only the Services that already existed when the pod started appear, and only those in the same namespace. That is why nobody uses them today: discovery is done via DNS, as you will see in Internal DNS. If you have many Services and want a clean environment listing, they can be switched off with enableServiceLinks: false in the pod's spec.

The simplest form. It is a list, not a map, and that is the first detail that confuses people:

Quotes on numeric and boolean values are not optional. The value field is of type string in the API schema. If you write value: 5432, the YAML parser produces an integer and validation fails:

The same problem with value: true, with value: no (YAML 1.1 reads it as a boolean) and with value: 08:00 (it reads it as sexagesimal). Practical rule: quote every env value that is not clearly text.

When to use a literal value instead of a ConfigMap:

When the ConfigMap has ten keys and you want them all, writing ten valueFrom blocks is absurd. envFrom dumps them in one go:

With this ConfigMap:

The container receives LOG_LEVEL, TIMEOUT_MS and MAX_SEATS_PER_BOOKING with those values. The key name becomes literally the variable name, which imposes a condition: the keys must be valid environment variable names (letters, digits and _, not starting with a digit).

What happens if a key is not valid? Kubernetes ignores it silently and records it as an event:

This is exactly why in the ConfigMaps lesson I insisted on keeping the file ConfigMap separate from the variables ConfigMap. An nginx.conf cannot be an environment variable, but it lives perfectly happily in the same object and ruins the envFrom.

envFrom accepts a prefix, which solves the problem of collisions between sources:

The result inside the container:

Notice DB_password in lower case: the prefix is prepended as it is, without changing the rest. If you want DB_PASSWORD, the Secret key must be called PASSWORD. That is a reasonable motive for naming Secret keys in upper case when you know they will be consumed with envFrom.

By default, if the referenced ConfigMap or Secret does not exist, the pod does not start: it sits in CreateContainerConfigError.

With optional: true the pod starts without those variables:

Use it carefully. It is appropriate for genuinely optional configuration (a feature-flag ConfigMap that only exists in dev), but it is dangerous for anything essential: a silent failure where the pod starts with half its configuration is far harder to diagnose than a pod that does not start at all. In Rutas Norte, optional: true is used only for the experimental flags ConfigMap.

When you want a single key, or when the variable name must differ from the key name, you use valueFrom:

This is the form recommended by default over envFrom, for three solid reasons:

1. It is explicit. Reading the Deployment tells you exactly which variables the container receives. With envFrom you have to go and look at the ConfigMap.
2. It decouples names. The application expects LOGGING_LEVEL while our key is called LOG_LEVEL, because that is the house convention shared by every component. valueFrom translates it without renaming anything.
3. Least privilege. Only the keys the pod needs reach it, just as we did with items in volumes.

And a warning about secretKeyRef: even though the reference is clean, the value ends up in the process environment. It is visible in /proc/<pid>/environ to any process in the container, and many frameworks dump the whole environment onto debug error pages. For high-value credentials, the mounted file from the previous lesson is still the better option.

So far configuration has come from outside. The Downward API allows the opposite: letting the container know things about itself and about the pod that holds it, without talking to the Kubernetes API or needing any permissions.

The Rutas Norte case: when a customer reports that a booking failed at 22:47, we want to be able to search the traces for which pod and which node served that request. With 6 replicas of bookings-api spread across 3 nodes, without that information the investigation is impossible.

Fields available in fieldRef:

Two restrictions you need to know:

• Only those fields are accepted. You cannot ask for spec.containers[0].image or an arbitrary pod field. For that you have to talk to the API, which is what we will see in ServiceAccounts.
• metadata.labels and metadata.annotations without a specific key only work in a downwardAPI volume, not in env. In env you must give the specific key in square brackets.

Let us look at the result and its real value:

And this is what a bookings-api trace using those variables looks like:

With that line, the diagnosis is immediate: every error comes from the same pod and the same node, so the problem is not the application but that particular node (or its network path to bookings-postgres). Without the Downward API, that correlation does not exist.

An important detail about ENVIRONMENT: we could have set it as a literal value: "pro", but then we would have the data duplicated (in the label and in the variable) with the risk of the two drifting apart. Reading it from metadata.labels['environment'] guarantees that it always matches the pod's real label. It is a direct application of the module 2 labelling scheme.

There is also a file variant, useful when you want all the labels or annotations:

And with one advantage: unlike variables, this file does update if the pod's labels change, using the same mechanism as in the ConfigMaps lesson.

The second half of the Downward API exposes the container's declared resources, which we will study in depth in Resource Quotas and Limits:

Resources available: limits.cpu, requests.cpu, limits.memory, requests.memory, limits.ephemeral-storage, requests.ephemeral-storage.

The divisor decides the unit: 1m gives millicores, 1 gives whole cores (rounding up), 1Mi gives mebibytes, 1Gi gibibytes. If you omit it, the default is 1, which for memory means bytes and produces awkward numbers like 536870912.

What is this for in practice? So that the process can size itself. It is a real and very common problem: many runtimes do not see cgroup limits and believe they have all the memory and all the CPUs of the node, so they size their thread pools and their memory heaps generously and end up OOMKilled.

Applied to bookings-api (Node.js):

That 384 is 75% of 512Mi. Ideally it would be calculated, but the $(VAR) expansion of the next section does no arithmetic, so in practice it is computed in the container's entrypoint by reading MEMORY_LIMIT_MB:

For the JVM there is an automatic equivalent (-XX:MaxRAMPercentage=75), which is the preferable option when the language offers it.

Kubernetes allows you to reference some variables from others using the $(NAME) syntax:

The rules, which you need to know precisely:

Rule 1: the syntax is $(VAR), with parentheses. Not ${VAR} and not $VAR. Those two are shell syntax and Kubernetes leaves them untouched.

Rule 2: only variables defined EARLIER in the same env list are expanded. Order matters:

When a reference cannot be resolved, it is left literal. There is no error and no warning, and the value $(DEFINED_LATER) reaches the application exactly like that. It is a source of silent bugs.

Rule 3: variables coming from envFrom are not expanded. Those from envFrom are processed as a block and take no part in the expansion of the env ones. If you need to compose something with a ConfigMap key, bring it in first with valueFrom:

Rule 4: variables from the image are not expanded. $(PATH) or $(HOME) are not resolved: Kubernetes only knows the ones it defines itself.

Rule 5: $$ escapes the dollar sign. If your password contains a literal $(, write it $$(.

Here is the most frequent misunderstanding on the topic, and it deserves a section of its own.

This works: Kubernetes expands $(MAX_BATCH) in args with the same rules as the previous section, and the process receives --batch=50.

But this does not work:

And neither does this:

The cause of the confusion: there is no shell. When you write command: ["/app/worker"], containerd runs that binary directly with execve(). /bin/sh is never invoked, so nobody expands $VAR or ${VAR}: that expansion is the shell's job, and the shell is not in the chain.

The three options and when to use each one:

If you use the second form, two important warnings:

Warning 1: the shell becomes PID 1 and many shells do not forward signals to their children. That breaks the graceful termination with SIGTERM you studied in the Pods lesson: when the pod is deleted, the real process never receives the signal and dies abruptly once the grace period runs out. The solution is exec:

With exec, the shell is replaced by the program, which inherits PID 1 and receives the signals.

Warning 2: it is a doorway to command injection. If a variable comes from an untrusted source and you interpret it inside an sh -c, a value like ; rm -rf / gets executed. With Kubernetes' $(VAR) that cannot happen: the value is passed as an argument, it is not interpreted.

And a reminder from the previous lesson: never put a credential in args, not even expanded from a Secret. kubectl describe pod shows Args with the value already resolved.

When SAME_KEY arrives by several routes, which one wins? The resolution order is strict:

The rule in one sentence: env always beats envFrom, and within each block the last entry wins.

A full example showing the four levels:

debug wins, the one from env. And if we removed that env block, warn would win, the one from the last configMapRef.

Tips to save yourself pain here:

1. Avoid collisions rather than managing them. One ConfigMap per purpose and no repeated keys.
2. Use prefix when you dump several sources with envFrom.
3. When in doubt, check the effective value, which is the only truth:

4. Watch out for image variables. A base image can define NODE_ENV=production in its Dockerfile, and unless you explicitly override it, it will be there without appearing in any manifest of yours. kubectl exec ... -- env reveals it.

We come back to the mechanism from section 1. Environment variables are fixed at execve() and nobody can change them afterwards. The consequence is blunt:

If you change a ConfigMap or a Secret, the pods already running will never see the change, no matter how many hours you wait.

Demonstration:

The object says trace and the process still sees debug. And it will stay that way until the end of time.

A comparison with what does refresh:

The three possible ways out:

1. Restart the pods by hand. It works and it is what we did when rotating credentials:

The problem is that you have to remember. If someone changes the ConfigMap in a pull request and forgets the restart, the platform is left in a state where the manifest says one thing and the pods do another. That silent gap is dangerous: it is discovered weeks later, when a pod restarts for some other reason and suddenly changes behaviour without anybody having touched anything.

2. Use a mounted file for whatever has to change live.

3. Automate the restart with a hash, which is the elegant solution and the subject of the next section.

The technique is simple and has become a standard pattern: store a digest (hash) of the ConfigMap and Secret contents in an annotation on the pod template.

The key is that Kubernetes triggers a new rollout when anything inside spec.template changes, annotations included. Since the hash depends on the configuration contents, changing the configuration changes the hash, changing the hash changes the template, and changing the template triggers the rolling update.

In the manifest:

How the hash is computed in the deployment pipeline:

It is important to hash only .data: if you hashed the whole object, resourceVersion would change on every update even when the contents were identical, and you would get spurious rollouts.

Four advantages of this pattern:

1. Changing the configuration deploys by itself. There is nothing to remember.
2. You get history. It shows up in kubectl rollout history and can be undone with rollout undo, just like a code change.
3. There is no gap. The running pod always corresponds to the current ConfigMap.
4. It is auditable. The annotation says which exact version of the configuration each revision corresponds to.

It is worth knowing that the ecosystem tools do this for you:

For Rutas Norte, while we work with plain YAML, the script above in the pipeline is enough. In module 10 we will replace it with Kustomize.

You now have all the elements to decide. This is the decision table:

Decision rules for Rutas Norte:

1. Non-sensitive configuration that is stable for the pod's lifetime → environment variable. LOG_LEVEL, TIMEOUT_MS, REDIS_HOST. It is the simplest thing and everyone understands it.
2. Configuration files → volume. The nginx.conf of web-store, the fare-rules.json of bookings-api.
3. High-value credentials → volume, with *_FILE if the image accepts it. The bookings-postgres password for bookings-api.
4. Credentials demanded by third-party images → variable with secretKeyRef, no alternative. POSTGRES_PASSWORD in the official PostgreSQL image.
5. Configuration that must change live → volume. The nginx MAINTENANCE_MODE.
6. Pod identity → Downward API in variables. It is information that is immutable for the pod's lifetime, so the limitation does not get in the way.

We close with the complete inventory, which is also the state of the platform at the end of this lesson. In each case I mark the origin: L literal, CM ConfigMap, S Secret, DA Downward API.

web-store barely uses variables: its configuration is the nginx.conf from the mounted ConfigMap, because nginx reads a file, not the environment. It is the perfect example of rule 2.

Look at DB_POOL_MAX: it grows with the environment because production has more replicas and more traffic on bank-holiday weekends and school holidays, but it cannot grow without limit, because PostgreSQL's max_connections is finite and replicas × DB_POOL_MAX must not exceed it. With 4 replicas and DB_POOL_MAX: 25 that is 100 connections. It is exactly the kind of calculation that should be documented in the YAML with a comment.

Here there is no choice: the official PostgreSQL image reads those variables. This is rule 4. Putting PGDATA in a subdirectory is an essential precaution for when the persistent volume of module 5 arrives: the mount point usually contains a lost+found that prevents the database from initialising.

allkeys-lru is a business decision: redis-cache is the seat-availability cache and it is expendable, so we prefer it to discard the least used keys rather than reject writes.

In dev the SMTP setting points at mailhog, a local mail catcher: no real emails are sent to customers in development. It is a data-protection decision, not a convenience one, and it is exactly the kind of thing that separating configuration makes possible.

We bring everything we have learnt together into a single manifest, which is the real state of the component at the end of this lesson:

This manifest can be published without leaking anything. It contains references, not values. It is exactly the bar we set ourselves at the start of the module.

Tips:

1. valueFrom by default, envFrom only when there are many and they are already named correctly. The traceability of the manifest is worth more than saving lines.
2. Add the Downward API to every component from day one. It costs six lines and on the day of an incident it is worth its weight in gold.
3. Document every variable with a comment in the manifest: what it does, what range is valid, who consumes it.
4. Order env in blocks — literals, ConfigMap, Secret, Downward API, composed — as in the final manifest in section 13. It reads far better.
5. Whenever in doubt, kubectl exec -- env. The manifest is the intention; the process environment is the reality.

Create a diagnostic pod called inspector-config in rutas-norte-dev that receives variables by all four routes and prints them:

1. A literal COMPONENT=inspector.
2. Every key of the bookings-api-config ConfigMap with the CFG_ prefix.
3. Only the password key of the bookings-postgres-credentials Secret, under the name DB_KEY.
4. The pod name, the node and the environment label via the Downward API.
5. A SUMMARY variable composed with $(VAR) containing <component>@<node>:<environment>.

Apply the manifest, show the output and explain why point 5 only works if the variables are in the right order.

1. Deploy bookings-api with LOGGING_LEVEL coming from the ConfigMap.
2. Check the effective value inside the pod.
3. Change the ConfigMap to trace and wait three minutes. Check the value in the pod and the value of the object again.
4. Also mount the same ConfigMap as a volume at /etc/config and repeat the experiment. What difference do you observe?
5. Implement the hash technique: compute the sha256 of the ConfigMap's .data, put it in an annotation on the template and prove that changing the ConfigMap and recomputing the hash triggers a rollout.

A colleague has deployed notifications-worker with this fragment and complains that the worker processes batches of size ${MAX_BATCH} instead of 200:

1. Explain what is wrong in args and what is right in command, and why they are different cases.
2. Fix the manifest using the Kubernetes syntax.
3. Explain why this command with sh -c is problematic for the graceful termination from module 2 and fix it.
4. Rewrite the fragment without using a shell at all.
5. If MAX_BATCH came from an envFrom instead of a valueFrom, would the $(MAX_BATCH) expansion work in args? Justify your answer.

Solution 1

Point 5 works because COMPONENT, NODE_NAME and ENVIRONMENT are defined before SUMMARY in the env list. Kubernetes resolves the list in order and can only substitute what it has already processed. If SUMMARY were in first position, the value would literally be $(COMPONENT)@$(NODE_NAME):$(ENVIRONMENT), with no error and no warning.

And notice the side effect that serves as a warning: DB_KEY with the password appears in the pod's log, because the command runs env. Anyone with kubectl logs can see it, and in production that log would go to the centralised system of module 7. Never dump the environment in a container that consumes secrets.

Solution 2

The object says trace, the process still says debug. With the volume added:

The same ConfigMap, the same pod, the same instant, two different values. The file refreshed and the variable did not. That is the difference on a single screen.

The hash:

This time it works. And with history:

Solution 3

1. args is wrong, command is right. In args someone wrote $MAX_BATCH with shell syntax; Kubernetes only understands $(VAR), so it leaves the text as it is and the program receives --queue=$MAX_BATCH. In command, by contrast, the ${MAX_BATCH} expansion does work, but Kubernetes is not the one doing it: it is done by the sh -c that is running. They are two different expansions performed by two different actors, and confusing them is the origin of the bug.

There is also a second, hidden problem: when you define command with sh -c "...", the contents of args are added as positional arguments of the shell ($0, $1...), not of the program. That --queue=... never reaches anywhere useful.

2. Fixed with the Kubernetes syntax:

3. The problem with sh -c without exec: the shell is the container's PID 1 and /app/worker is its child. When Kubernetes deletes the pod it sends SIGTERM to PID 1, that is, to the shell. Most shells do not forward signals to their children, so the worker never finds out and keeps processing emails until the terminationGracePeriodSeconds expires (30 s by default) and the SIGKILL arrives, killing it outright. If it was sending a confirmation email, that email is lost.

With exec, the shell replaces itself with the worker, which becomes PID 1 and receives the SIGTERM directly.

4. Without a shell at all, which is the right way:

With no shell there is no signal problem, no risk of command injection, and the expansion is done by Kubernetes before anything starts. The cleanest thing of all would be for /app/worker to read MAX_BATCH straight from its environment and drop args altogether.

5. No, it would not work. The $(VAR) expansion only reaches variables defined in the env list before their use. The ones coming through envFrom are resolved as a separate block and take no part in the expansion of env or of args. --queue=$(MAX_BATCH) would arrive literal. The fix is to bring it in explicitly with valueFrom before using it, even if it also arrives through the envFrom.

With this lesson you close the configuration block of the module. You understand the real mechanism — the KEY=value list the kubelet hands to execve() and that stays frozen in the process — and everything else follows from it. You have mastered the four sources: the literal with its mandatory quotes, envFrom with prefix and optional and its trap of invalid keys being silently ignored, valueFrom with configMapKeyRef and secretKeyRef as the recommended option for being explicit and for allowing renaming, and the Downward API in its two forms, fieldRef for pod identity and resourceFieldRef so the process can size itself and not end up OOMKilled.

You know that bookings-api now records in every trace which pod and which node served it, which turns an intractable incident into a two-minute diagnosis. You know the five rules of $(VAR) expansion — parentheses, order, nothing from envFrom, nothing from the image, $$ to escape — and why in command/args there is no shell unless you ask for one, with the consequences that brings: ${VAR} is not expanded, and if you use sh -c without exec you break the graceful termination from module 2. You are clear about precedence (env beats envFrom, and within each block the last one wins) and about how to check the truth, which is always kubectl exec -- printenv.

And above all you have internalised the limitation that governs the design decisions: environment variables never refresh, with the solution of the configuration hash in a template annotation that turns a ConfigMap change into a rolling update with history and rollback. With the "variable versus file" decision table you no longer hesitate over where to put each thing, and the complete catalogue of the six Rutas Norte components per environment is the map of the platform as it stands now: one image per component, zero credentials in Git and all the variability concentrated in k8s/environments/.

Two module 2 debts remain, and both are about resources. No namespace has a quota, so one wrong deployment in rutas-norte-dev — one replica too many, a restart loop, a pod asking for 16 GiB — can consume the cluster capacity and leave rutas-norte-pro with nowhere to run. And the requests and limits we have been writing since module 2 are set by eye, with no criteria at all. The next lesson, Resource Quotas and Limits, tackles it: you will see who uses requests (the scheduler) and who uses limits (the kubelet through cgroups), the crucial difference between CPU, which is throttled, and memory, which kills the process with OOMKilled, what happens when the sum of limits exceeds the cluster capacity, and you will put a ResourceQuota on each of the three Rutas Norte environments so that dev cannot frighten pro ever again.