The previous lesson took the web-store configuration out of the image and put it into a ConfigMap, but it ended with a warning: a ConfigMap protects nothing. Anyone who can list ConfigMaps in rutas-norte-pro reads their full contents in plain text. And the debt we have been dragging since module 2 is precisely one that cannot tolerate that: the bookings-postgres password is written in the clear in a manifest versioned in Git, and that database holds the name, ID number, phone number and email of every Rutas Norte customer. This lesson settles it with the Secret object, but above all it teaches you not to trust that object more than it deserves: you will see why base64 is not encryption, why etcd encryption at rest is not enabled by default, who can read a secret without you noticing, and what the ecosystem uses when the native Secret falls short.

Important warning. This lesson explains the mechanisms Kubernetes offers for managing credentials and gives you a solid starting point, but it does not replace a professional security design. bookings-postgres stores personal data belonging to real customers (name, ID number, phone number, email), which in the European Union falls squarely within the scope of the GDPR and, in Spain, of the LOPDGDD. The specific design of credential management, encryption at rest, the rotation policy, the audit logs and the applicable security measures must be reviewed and approved by a security professional or by your organisation's compliance officer. Treat everything that follows as necessary technical knowledge, not as a compliance recommendation.

1. The module 2 debt and why it hurts
2. What a Secret is and how it differs from a ConfigMap
3. Base64 is not encryption: the demonstration
4. The Secret types
5. Creating Secrets: imperative and declarative with stringData
6. Consuming a Secret as a volume
7. The bookings-postgres-credentials secret in Rutas Norte
8. imagePullSecrets for the private registry
9. Encryption at rest in etcd
10. Who can read a secret
11. Credential rotation
12. What you must never do
13. The real solutions in the ecosystem

This is the manifest we have in the Rutas Norte repository today:

The problem is not merely cosmetic. Let us enumerate the real damage:

1. Git does not forget. Even if you delete the line today and commit, the password is still in the history. Getting it out of there requires rewriting the repository history (git filter-repo) and force-pushing to every clone. In practice, a credential that has been in Git is considered compromised and must be rotated, not deleted.
2. It spreads uncontrollably. Every developer who cloned the repository has the production password on their laptop. So do the CI system, the repository backup system, the internal code search engine and any static analysis tool that keeps a cache.
3. There is no traceability. There is no way to know who has read that password, or when.
4. Rotating it means a code deployment. Changing the password requires a commit, a review, a merge and a deployment. That pushes people towards never rotating it.
5. Automated bots will find it. If the repository becomes public for even a single minute, there are crawlers that locate credentials in seconds.

The goal of this lesson is for that manifest to end up like this:

The manifest can be published without leaking anything. That is the twelve-factor bar we saw in the previous lesson.

A Secret is, structurally, almost identical to a ConfigMap: metadata plus data, no spec or status, namespaced, and with the same 1 MiB limit. The difference lies in the treatment Kubernetes gives it and in the expectations it creates.

The three differences that really matter:

First: stringData. It is a write-only field. You write plain text into it and the apiserver base64-encodes it and moves it to data before storing it. It is what makes writing Secrets by hand bearable.

Second: the tmpfs mount. When a Secret is mounted as a volume, the kubelet creates a file system in RAM, not on the node's disk. Practical consequence: if the node is powered off abruptly, the credential is not left written on any recoverable disk. It is a real protection, though a limited one.

Third: describe does not show the contents.

Only the sizes. This avoids the classic accident of showing the password in a shared session or leaving it in a console log. But it is a presentation detail, not a security control, as the next section demonstrates.

This is the most important idea in the lesson, and you need to internalise it by doing it.

We create the Secret:

We look at it in YAML:

At first glance it looks encrypted. It is not. Base64 is a reversible encoding with no key, designed in the 1980s to carry binaries over text channels. Anyone can undo it:

Three seconds and one command. No password, no key, no special permissions beyond being able to read the Secret.

We can dump the whole contents in one go:

Or, from Kubernetes 1.30 onwards, with the built-in viewer:

What this implies exactly:

So what is base64 for, then? For two legitimate things, and neither of them is security:

1. Allowing binary content. A certificate in DER format or a private key with non-printable bytes does not fit in a YAML text field. Base64 makes it possible.
2. Avoiding escaping problems. A password with quotes, line breaks or $ would break the YAML. Encoded, it does not.

The rule you must memorise: a Kubernetes Secret does not protect the data; what it does is give it a place where controls can be applied (RBAC, encryption at rest, auditing, in-memory mounting) and separate it from the application manifest. The protection comes from those controls, not from the object.

The type field is not decorative: it determines which keys Kubernetes demands and which components know how to use the Secret.

Usage notes for Rutas Norte:

• Opaque is the one we will use for bookings-postgres-credentials, for the payment gateway key and for the mail provider token of notifications-worker. It is the default type if you specify nothing.
• kubernetes.io/dockerconfigjson will be the one that lets us pull images from registry.rutasnorte.example (section 8).
• kubernetes.io/tls will be the one holding the www.rutasnorte.example certificate. You will see it in TLS and Certificates, where cert-manager will generate and renew it on its own.
• kubernetes.io/service-account-token: be careful with this one. Before Kubernetes 1.24, every ServiceAccount had an associated Secret of this type with a token that never expired. They are no longer created automatically, and creating them by hand is a bad idea except in very specific cases: an eternal token is a token that can never be cleanly revoked. You will see the current mechanism in ServiceAccounts.

The type is validated too. This fails:

The basic-auth type demands exactly the key username, not user. That validation is useful: it stops an Ingress or a controller from receiving a Secret with the wrong shape.

Watch out for the shell history. That command is written into ~/.bash_history with the password inside it. Two ways to avoid that:

The printf '%s' instead of echo is deliberate: echo adds a trailing newline that would end up inside the password. It is a classic and baffling mistake: the password "looks right" but PostgreSQL rejects it. If you use echo, add -n.

If you write the manifest by hand, always use stringData, never data:

Advantages of stringData over data:

• You write it and read it without encoding or decoding anything.
• It removes the mistake of encoding with echo without -n, which puts a \n into the value.
• A diff in a pull request is readable... which, mind you, is also the reason this file cannot go into Git as it stands.

If a key appears in both stringData and data, stringData wins.

And here comes the obvious question: if this file cannot go into Git, where does it live? Three legitimate answers, in order of maturity:

1. Only in the cluster. The file is applied once from a secure workstation and destroyed. Simple, but you lose traceability and you have to document separately which keys exist.
2. In Git, encrypted. Sealed Secrets or SOPS let you version a file that only the cluster can decrypt (section 13).
3. Outside Kubernetes. An external secrets manager (Vault, AWS Secrets Manager) is the source of truth and an operator synchronises it (section 13).

In the meantime, the bare minimum in the Rutas Norte repository:

Plus a secret-postgres.yaml.example with dummy values, versioned, so the team knows which keys are needed.

The mechanism is identical to that of the ConfigMaps in the previous lesson, swapping configMap for secret and name for secretName:

Inside the container:

Two things to highlight:

• type tmpfs: the volume is in RAM. It never touches the node's disk. When the pod dies, it disappears.
• The same chain of links to ..data as with ConfigMaps, with the same consequence: if the Secret changes, the mounted file is updated (with the same kubelet delay, and with the same subPath exception, which does not propagate).

That refresh capability is the main reason to prefer a volume over an environment variable when credentials are involved. A well-written application can re-read the file and adopt a rotated password without restarting. We will come back to this in section 11.

Consumption as an environment variable (secretKeyRef, envFrom with secretRef) is just as common and often more practical, above all with third-party images such as postgres:16, which expect POSTGRES_PASSWORD in the environment. You will see it in full detail in Environment Variables. Here is the comparison up front, because it shapes the design:

Many serious images accept the _FILE variant: POSTGRES_PASSWORD_FILE=/etc/secrets/postgres/password. When it exists, prefer it: it combines the compatibility of the variable with the safety of the file.

Let us settle the debt properly. The full design:

The Secret, in k8s/environments/dev/secret-postgres.yaml (kept out of Git):

The bookings-postgres Deployment, now with no plain-text passwords:

And bookings-api, which consumes it as a file:

Note the design detail: bookings-api receives only username and password thanks to items. It does not receive the url key, which contains the duplicated password, nor any other key that might be added to the Secret in the future. It is the principle of least privilege applied to the contents of an object.

Verification that the debt has been settled:

grep finds nothing in the repository and the application has its credential. That is the goal.

The Rutas Norte images live in registry.rutasnorte.example, which is private. Without credentials, the pod fails like this:

ImagePullBackOff with a 401 always means the same thing: registry credentials are missing.

You create a Secret of the right type:

What it stores inside is a .dockerconfigjson file:

Another demonstration that base64 protects nothing: the auth field is simply user:password encoded.

And it is referenced in the pod:

Three important details:

1. imagePullSecrets goes in the pod's spec, not inside containers. It applies to every image in the pod, including init containers.
2. You can list several, one per registry. The kubelet picks the one matching the image's server.
3. You have to repeat it in every pod in the namespace, which is tedious and easy to forget. The elegant solution is to declare it on the ServiceAccount: every pod using it will inherit it without writing it out. You will see it in ServiceAccounts.

Good security practice: the registry token must be image read-only (pull), never one with push permission. If the cluster is compromised, a pull token limits the damage to being able to download images; a push one would allow replacing them with malicious versions. The Image Security lesson goes deeper into this.

We reach the point most people are unaware of.

By default, Secrets are stored in etcd unencrypted. Base64-encoded, yes; encrypted, no. Whoever has access to the disk of a control-plane node, to an etcd backup or to a file system dump has every credential in the cluster.

Demonstration (on a cluster where you have control-plane access):

The password is read straight out of the dump. You do not even need to decode base64: etcd stores the serialised object with the values already in binary.

The fix is an EncryptionConfiguration on the apiserver:

And it is enabled by starting the apiserver with --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml.

The available providers:

Four critical rules:

1. Order matters. The first provider in the list is the one that encrypts on write. All of them in the list are tried on read. That is why identity must go last during the migration: it lets you read the Secrets that already existed unencrypted.
2. Enabling it does not encrypt what already exists. Only what is written from that moment on gets encrypted. You have to force a rewrite of everything:

3. The key in that file is the crown jewel. If you use aescbc or secretbox, the key sits in plain text on the control-plane node's disk. Whoever has that file and a copy of etcd has everything. That is why kms v2 is the only genuinely solid option in production: the data encryption key is wrapped with a key that lives in an HSM or in the cloud KMS and is never written to the node.
4. On managed Kubernetes (EKS, AKS, GKE) you enable this with a tick box, delegating to the provider's KMS. It is one of the strong reasons to use a managed cluster, as you will see in 10-06.

For Rutas Norte, which stores customer ID numbers and contact details, encryption at rest with a KMS is not optional: it is one of those technical measures the GDPR expects to see documented. And once again: the specific decision must be validated by the compliance officer.

A Secret has no protection of its own. Everything depends on RBAC, studied in depth in Role-Based Access Control. Here it is enough to understand why the two are inseparable.

Who can read bookings-postgres-credentials today:

The two entries that surprise people and must be underlined:

list leaks the contents. Many people grant list secrets thinking it only allows seeing the names. It does not: kubectl get secrets -o yaml with list permission returns the complete objects, data included. In RBAC, list on secrets is equivalent to get.

Being able to create pods is equivalent to being able to read every secret in the namespace. Anyone who can deploy a pod can mount any Secret in the namespace and dump it. There is no way to prevent this with RBAC on secrets: you have to restrict pod creation. It is the main reason why the namespace is the real unit of credential isolation and why Rutas Norte separates dev, pre and pro: nobody with development access should be able to create pods in rutas-norte-pro.

A good auditing exercise, anticipating module 8:

Rotating a credential means changing it periodically and, without exception, whenever you suspect it has leaked. And here a problem appears that Kubernetes does not solve on its own.

The problem: a pod that started yesterday holds the old password. If you change it in the Secret:

• If it consumes it as a volume: the file is updated within a couple of minutes. But the application will only notice if it re-reads the file. Most read at start-up and keep the connection open.
• If it consumes it as an environment variable: it is never updated. A process's environment variables are fixed when it is executed and there is no way to change them from outside.

The correct rotation sequence, with two valid credentials at once, which is what avoids a service outage:

In commands:

A warning about ALTER USER: in PostgreSQL a user has a single password, so step 1 replaces it and the old pods will break as soon as they open a new connection. In a system with serious rotation you use two users (rutasnorte_a and rutasnorte_b) and alternate between them, which is what Vault's dynamic secret engines do. Design it before you need it.

Points worth fixing in your mind:

• Automate change detection. The technique of hashing the Secret into an annotation on the template, which you will see in 03-03, makes updating the Secret trigger the rollout by itself, with no manual rollout restart.
• Always rotate after an exposure. If the password has been in Git, rotating it is mandatory even if you rewrote the history.
• Watch out for stragglers. A CronJob or a pod that is not part of a Deployment does not get restarted by rollout restart. Draw up a list of consumers before rotating.

On the args point, a demonstration of how exposed it stays:

Anyone with read permission on pods — a permission granted to a lot of people, monitoring systems included — has just read the mail password.

The native Secret solves where the credential lives inside the cluster. It does not solve how it gets there securely and auditably, nor how it is versioned, nor how it rotates itself. That is what these four families exist for. We are not going to run a tutorial on any of them — each one is worth a course of its own — but you must know when to look in each direction.

An example of what an ExternalSecret looks like, just so you recognise the pattern when you come across it:

The elegant part of the pattern: this file can go into Git. It does not contain the credential, it only says where to find it. And refreshInterval: 1h means that if someone rotates the password in Vault, the cluster's Secret updates itself within the hour.

Where each thing fits in the maturity of a platform like Rutas Norte:

1. Today (where we are): native Secrets created by hand, outside Git, with .gitignore and an example file. Acceptable in dev.
2. Next step: encryption at rest with a KMS in the cluster and strict RBAC on secrets in rutas-norte-pro.
3. Maturity: External Secrets Operator pointing at the corporate secrets manager, with automatic rotation and an audit trail of every read.

Final tips:

1. Treat every Secret as if it were going to leak. Design on the assumption that someone will read it: different credentials per environment and per component, with least privilege in the database.
2. Install a credential detector in your pre-commit. gitleaks or detect-secrets cost ten minutes of configuration and prevent 90% of accidents.
3. Keep an inventory of secrets. What exists, who consumes it, when it was last rotated, who owns it.
4. Document the keys, never the values. A secret-postgres.yaml.example with password: CHANGEME under version control is useful and safe.
5. And, as said at the start: have the design reviewed by a security professional or the compliance officer. With customers' personal data involved, this is not negotiable.

1. Create a payment-gateway Secret in rutas-norte-dev with the keys api_key (value sk_test_4eC39HqLyjWDarjtT1zdp7dc) and environment (value sandbox).
2. Show the object in YAML and check that the value cannot be read.
3. Decode api_key with a single chained command.
4. Check what kubectl describe shows for that Secret and explain why that is not a security measure.
5. Answer: if a colleague has the list verb on secrets in that namespace but not get, can they read the key? Prove it with kubectl auth can-i.

1. Write the manifest for the bookings-postgres-credentials Secret for rutas-norte-dev using stringData, with username, password and database, and with the Rutas Norte labelling scheme.
2. Modify the bookings-postgres Deployment so that it takes the three variables from the Secret.
3. Modify bookings-api so that it mounts only username and password as files in /etc/secrets/postgres with read-only permissions.
4. Verify that the volume is on tmpfs.
5. Prove with grep that no plain-text password is left in k8s/, and add the necessary rules to the .gitignore.

bookings-api runs with 4 replicas in rutas-norte-pro and the database password has to be rotated because a former team member knew it.

1. Write out the full sequence of steps, indicating which of them carries a risk of an outage and how you avoid it.
2. Run the Secret update with a single command that works whether or not the Secret already exists.
3. Make the pods adopt the new credential without losing service at any point.
4. Verify that the 4 new pods are up and that none of them still holds the previous credential.
5. Identify which consumers of the Secret are not restarted by kubectl rollout restart and what you would do about them.

Solution 1

The decoding:

describe only shows the size. It is not a security measure, but a matter of visual hygiene: it stops the credential being shown by accident on a shared screen or in the output of a script. Whoever has permission to run describe has the get verb, and with get -o yaml they obtain the value. The real protection comes from RBAC and encryption at rest, not from the output format.

About list:

Yes, they can read it. list returns the complete objects, not just their names. It is a very widespread permission-granting mistake.

Solution 2

The bookings-postgres Deployment is the one from section 7. The bookings-api fragment:

Verifying the tmpfs and that there are only two files:

Two files, not three: database has not been projected because it is not in items. And the volume is tmpfs, that is, RAM.

No results. The .gitignore:

The !k8s/**/*.yaml.example line lets you version the templates with dummy values, which are what document the keys needed.

Solution 3

The sequence, with the risk flagged:

1. Create the new password in PostgreSQL. High risk if the engine only accepts one password per user: the live pods will break on their next connection. You avoid it by creating a new user (rutasnorte_b) instead of changing the current one's password, leaving both valid.
2. Update the Secret. No risk: the live pods already have the credential loaded in memory.
3. rollout restart. No risk if maxUnavailable is set properly: module 2 guarantees progressive replacement.
4. Revoke the old credential. Risk if any consumer has not been restarted; that is why it comes after verifying.

The --dry-run=client -o yaml | kubectl apply -f - trick is the idiomatic one: it generates the object and applies it, working both for creating and for updating, which kubectl create on its own does not do (it would fail with AlreadyExists).

All four pods are new (same pod-template-hash, AGE under a minute and RESTARTS 0). Verifying the effective credential:

Consumers that are not restarted by a rollout restart of the Deployments:

• CronJobs such as occupancy-reports (module 6): their pods are created on each run, so they will pick up the new credential by themselves; but if there is a run in progress during the rotation, it will fail. Check kubectl get jobs before revoking.
• Loose pods created by hand for debugging. You find them with kubectl get pods -n rutas-norte-pro -o json | jq '.items[] | select(.metadata.ownerReferences == null) | .metadata.name'.
• StatefulSets (module 6): rollout restart works, but it is ordinal and slower; you have to wait for it to finish.
• Clients outside the cluster: reporting tools, backups, dashboards. Kubernetes cannot see them and they are the ones that cause the incident when you revoke. That is why the consumer inventory from section 11 is not bureaucracy.

You have settled the most serious debt from module 2. The bookings-postgres password is no longer in any Git manifest: it lives in a Secret, bookings-api receives it as a file mounted in memory with only the keys it needs, and bookings-postgres takes it from secretKeyRef in order to initialise. A grep over k8s/ returns nothing.

But the important part of this lesson is not the object, it is the judgement. You know that a Secret differs from a ConfigMap in stringData, in the tmpfs mount, in describe hiding the contents and in the existence of a type field that validates the shape. And you know, because you have proved it with a single command, that base64 is not encryption: the real protection comes from RBAC, encryption at rest and namespace isolation, not from the object itself. You know the seven Secret types and what each one is for, the three ways of creating them, and why stringData is always preferable to data when writing by hand.

You know that encryption at rest in etcd is not enabled, you have seen the password readable in an etcd dump, you know the five EncryptionConfiguration providers and why kms v2 is the only genuinely solid one, and that enabling it does not retroactively encrypt what already exists. You are clear about who can read a secret — with the two surprises: list leaks contents and being able to create pods is equivalent to being able to read everything — how a credential is rotated without cutting off the service, and why environment variables never refresh. You have the list of what must never be done and a map of the real ecosystem — Sealed Secrets, SOPS, External Secrets Operator, Vault, the cloud KMS offerings — for when the native Secret falls short. And do not forget the warning from the start: with the personal data Rutas Norte holds, the final design must be validated by a security professional or the compliance officer.

The two lessons have left a common loose end. We have seen the objects that store configuration and secrets, and one of the ways of getting them into the container: the mounted file. The other one is missing, the one that shows up in 80% of real-world manifests and that both lessons have mentioned without developing. The next one, Environment Variables, covers it in full: env, envFrom with prefix, valueFrom with configMapKeyRef and secretKeyRef, the optional flag, the Downward API so that bookings-api records in its traces which pod and which node served each request, $(VAR) expansion and its rules, the collisions when the same key arrives by several routes, and the technique of the configuration hash in an annotation that finally solves the problem of environment variables never refreshing.