We closed the previous lesson with an uncomfortable observation: we have given the Rutas Norte components their configuration, their credentials and their resources, but we have not given them an identity. Right now, the pods of web-store, bookings-api, bookings-postgres, redis-cache and notifications-worker are all using the same default ServiceAccount of their namespace, with a Kubernetes API access token mounted inside that none of them needs and that is the first thing an attacker who managed to run code in one of those containers would go looking for. This lesson fixes it: you will understand what a ServiceAccount is and how it differs from a user, how modern projected tokens work, why automountServiceAccountToken: false should be your default choice, and you will talk to the API from inside a pod with curl to see with your own eyes both an authorised response and a well-deserved 403 Forbidden.

1. The identity of a workload
2. Users versus ServiceAccounts
3. The default ServiceAccount and why it must not be used
4. Creating dedicated ServiceAccounts for Rutas Norte
5. The projected token: how it has changed
6. What lives in /var/run/secrets/kubernetes.io/serviceaccount/
7. automountServiceAccountToken: false
8. Talking to the API from inside a pod
9. The 403 Forbidden and what it means
10. imagePullSecrets on the ServiceAccount
11. The ServiceAccount says who you are, RBAC says what you can do
12. Identity federation with the clouds

Every request reaching the kube-apiserver goes through three phases you already know from module 1:

When you run kubectl get pods, the first phase is resolved with the certificate or the token in your kubeconfig. But what about when the one asking is a pod?

The cases where a pod needs to talk to the API are more common than they seem:

And in Rutas Norte there is a specific case coming in module 6: occupancy-reports, the nightly task, will have to query the API to find out how many bookings-api replicas were active each hour and correlate that with occupancy. It needs an identity and it needs permissions.

But the main reason for studying this is not the pods that do talk to the API: it is the five that do not and yet carry a valid credential around without needing it. That is free attack surface.

Kubernetes distinguishes two kinds of identity, and the difference is fundamental:

The first point always comes as a surprise: Kubernetes has no users. There is no user database, and no command to create one. When your kubeconfig uses a client certificate, Kubernetes reads the CN (Common Name) field of the certificate and takes it as the user name, trusting that the cluster's certificate authority only signs legitimate certificates. With OIDC, it delegates to the identity provider. Managing people is, deliberately, somebody else's problem.

ServiceAccounts, by contrast, are first-class objects:

There is only one, the default, and all our pods are using it. Notice the SECRETS 0: it is the sign that we are on modern Kubernetes, where ServiceAccounts no longer carry an associated Secret with a permanent token. We will come back to that in section 5.

The full name of a ServiceAccount in the authorization system has this shape:

That is the exact identifier you will use in the RoleBindings of RBAC, and also the one that appears in the audit logs. In addition, every ServiceAccount automatically belongs to the group system:serviceaccounts and to its namespace's group, system:serviceaccounts:rutas-norte-pro, which lets you grant permissions to all of an environment's accounts at once (something that is almost never a good idea, but worth knowing exists).

Every namespace gets a ServiceAccount called default the moment it is created. If a pod specifies none, it is assigned that one.

Five reasons not to use it:

Reason 1: it is shared. Every pod in the namespace has the same identity. If tomorrow occupancy-reports needs permission to read Deployments and you grant it to default, you are also granting it to web-store, to redis-cache and to bookings-postgres. It is the opposite of least privilege.

Reason 2: it destroys traceability. In the audit log every request shows up as system:serviceaccount:rutas-norte-pro:default. Faced with a suspicious access, there is no way of knowing which component made it.

Reason 3: it prevents selective revocation. If notifications-worker is compromised, you want to withdraw its permissions immediately. With the default, withdrawing them means withdrawing them from the whole namespace.

Reason 4: it mounts a token almost nobody needs. By default, automountServiceAccountToken is true, so all our pods carry a valid API token inside. Check it:

An nginx server that only serves static files has a cluster API access credential sitting there. If someone finds an arbitrary file read vulnerability in that application (hardly exotic), the first thing they will do is read that token.

Reason 5: the temptation to grant it permissions. It is so convenient that, in a rush, somebody ends up binding the default to a powerful ClusterRole "just to make it work". And that never gets reverted.

The rule is simple:

One ServiceAccount per component, and automountServiceAccountToken: false on all of them except those that genuinely talk to the API.

Let us start with the object. It is one of the simplest in Kubernetes:

Or imperatively, to generate the YAML:

The complete plan for Rutas Norte, with the reasoned decision for each component:

Five out of six need no token. That is the normal outcome: the vast majority of business applications do not talk to the Kubernetes API. And even so, the default configuration mounts one for all of them.

So why create a ServiceAccount for those that will not use it, if they are not going to talk to the API? For three reasons:

1. Traceability: if one day one of them makes a request, you will know which one.
2. imagePullSecrets: the ServiceAccount is the right place to declare them (section 10).
3. Preparation: when tomorrow bookings-api needs a specific permission, the identity already exists and there is no need to touch the Deployment.

The six objects, in k8s/base/serviceaccounts.yaml:

And the assignment on the pod, with the serviceAccountName field:

A historical detail you will see in old manifests: there is also a serviceAccount field (without Name). It has been deprecated for many versions and is kept only for compatibility. Always use serviceAccountName.

We apply and verify:

Each component with its own identity. And the token has disappeared from where it was not needed:

That error is exactly the desired outcome. There is no credential to steal.

This part has a history and it is important to know it, because you will come across manifests and tutorials from the earlier era.

When a ServiceAccount was created, a controller automatically generated a Secret of type kubernetes.io/service-account-token with a JWT inside. That token:

• Never expired.
• Was not tied to any pod: anyone who copied it could use it from anywhere, even from outside the cluster.
• Was stored in etcd, so it was in the backups.
• Could not be revoked without deleting the Secret and the ServiceAccount.

It was, in practice, an eternal password in every namespace. If a token leaked into a log, a dump or a repository, it was still valid months later.

The current mechanism is completely different and much better:

The properties of the modern token:

Let us look at the real contents of a token. Inside a pod that does have one mounted:

It is a standard JWT. What matters:

• sub is the identifier RBAC will use.
• exp - iat = 3600 seconds: a one-hour lifetime.
• kubernetes.io.pod ties the token to a specific pod. If that pod is deleted, the apiserver rejects the token even if it has not expired.
• aud limits who it is valid for.

An important practical warning: if your application reads the token once at start-up and keeps it in memory, it will stop working in an hour. The official client libraries (client-go, the Python kubernetes client, and so on) re-read the file automatically. If you talk to the API with curl or with a generic HTTP library, you have to re-read the file on every request. It is a source of baffling 401 errors that appear exactly one hour after the deployment.

The projection can be customised with an explicit volume:

The audience field is especially useful for an advanced pattern: issuing a token that is only valid for one specific external service, so that if it leaks it cannot be used to talk to the Kubernetes API.

And if you ever need a permanent token (integrating an external tool that cannot rotate), it has to be created explicitly:

Avoid it whenever you can. It reintroduces every problem of the old model. For one-off cases, the correct alternative is to request a short-lived token:

When the token is mounted, the directory contains three files:

You recognise two things from earlier lessons: the chain of links to ..data that makes atomic updates possible (here it is what enables token rotation) and the tmpfs, that is, RAM.

The ca.crt is the piece people usually forget. Without it, a client talking to https://kubernetes.default.svc cannot validate the apiserver's certificate and has to fall back on --insecure, which is exactly what you must not do: with no validation, an attacker with access to the pod's network could impersonate the apiserver and capture the token.

Together with the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables that Kubernetes injects into every pod (you saw them in the environment variables lesson), these three files are everything needed to talk to the API. It is precisely what the client libraries call in-cluster configuration (InClusterConfig).

This field can be set in two places, and the precedence matters:

The pod always has the last word. That enables the recommended pattern: set false on the ServiceAccount as a safety net, and an explicit true only on the specific pod that genuinely needs it.

Why this matters so much, in security terms: if an attacker manages to run code in a container — through an application vulnerability, a compromised dependency or an injection — the first thing an automated exploitation script does is read that token and probe what permissions it has. It is a standard step in any Kubernetes privilege escalation tool.

Let us compare the attack surface:

With no token, even a careless RBAC configuration is neutralised for that pod: there is no credential with which to exercise it.

Checking the state across the whole cluster, which is an audit worth doing:

The system ones legitimately need it. From Rutas Norte, only occupancy-reports. That is the goal.

An operational note: if you add automountServiceAccountToken: false to a ServiceAccount, the pods already running keep their token. The mount is decided when the pod is created. They have to be recreated:

We are going to do it by hand to understand what the client libraries do internally. We will use occupancy-reports, the only Rutas Norte component that genuinely needs to talk to the API.

Step 1: a pod with the ServiceAccount and the token mounted.

Step 2: prepare the variables inside the pod.

kubernetes.default.svc is the apiserver's Service, which exists in the default namespace of every cluster. Any pod can resolve it via DNS, as you saw in module 2.

Step 3: the first request, to the version endpoint (which requires no permissions).

It worked. The three elements of the request:

1. --cacert $SA/ca.crt: verifies that the server really is the cluster's apiserver. Never use -k or --insecure.
2. Authorization: Bearer $TOKEN: the authentication. It is the standard JWT mechanism.
3. The internal Service URL, resolved by DNS.

Step 4: ask who I am. This endpoint is extremely useful for debugging:

There is the complete identity: the username in the format we anticipated and the three automatic groups. From outside, the equivalent is:

Step 5: try something that requires permissions.

And here comes the instructive part.

Let us look at the full body of that response:

This message is a small lesson in itself. Let us break it down:

The distinction between 401 and 403 is fundamental and must be clear:

If you get a 401, the problem is the token: check you are reading it correctly, that it has not expired (remember the one-hour lifetime) and that the pod has the volume mounted. If you get a 403, the token is fine and what is missing is permissions.

And that 403 is exactly what should happen right now. Our occupancy-reports ServiceAccount has no permissions granted at all, because granting permissions is RBAC's job, and RBAC is the subject of lesson 08-01. A freshly created ServiceAccount, with no RoleBindings, can do absolutely nothing beyond the few public endpoints such as /version.

This is worth underlining because it is the correct design:

Kubernetes denies by default. Creating an identity grants no permissions. Everything a ServiceAccount can do has to be granted explicitly.

We can check permissions without making the request, with kubectl auth can-i:

And the complete list of what it can do:

Only the bare minimum: ask who it is, ask what it can do, and check health and version. No reading pods, no deployments, and certainly not the Secrets holding the bookings-postgres password.

As a preview of module 8, this is what the minimal grant occupancy-reports will need looks like, so you know where we are heading:

Look at the subjects: that is where the ServiceAccount we created today connects with the permissions. The two pieces fit together, but they are different objects with different responsibilities.

We clean up the test pod:

We pick up a loose end from the Secrets lesson. There we declared imagePullSecrets on each pod so we could pull images from registry.rutasnorte.example:

Repeating it in every Deployment, every Job and every CronJob is tedious and, above all, easy to forget: the failure shows up weeks later, when somebody adds a new component and runs into an incomprehensible ImagePullBackOff.

The ServiceAccount is the right place:

And the Deployment stays clean:

How it works: when a pod is created, an admission controller copies the ServiceAccount's imagePullSecrets into the pod's spec. You can see it:

The manifest did not have it and the object does. It is the same phenomenon as with the LimitRanger from the previous lesson: an admission controller has mutated the object.

Three details:

1. They are added, not replaced. If the pod declares its own, they are combined with those of the ServiceAccount.
2. The Secret must exist in the same namespace. It has to be created in each of the three environments.
3. It is not retroactive. Pods already created do not inherit it; they have to be recreated.

And a very common trick to avoid repeating it per component: add it to each namespace's default ServiceAccount, so that any pod not specifying another inherits it.

It is useful, but remember that it does not exempt you from creating dedicated ServiceAccounts: imagePullSecrets is the only reasonable thing the default should carry.

This is the sentence that sums up the lesson and it is worth engraving:

The ServiceAccount is the identity; RBAC is the permissions. They are two independent things and both are needed.

The possible combinations and what they mean:

Look at the last row, because it is the most important conclusion in practical terms: the best way to manage the permissions of a pod that does not need the API is for it to have no token. No misconfigured RBAC can hurt it if there is no credential to use.

What is left for 08-01: the Role and ClusterRole objects (which verbs on which resources), the RoleBinding and ClusterRoleBinding objects (to whom), role aggregation, the cluster's predefined roles (view, edit, admin, cluster-admin), and the privilege escalation to be avoided. All of that rests on the identity you created today.

One last note so you know it exists, because it is what you will use if Rutas Norte moves to a managed cluster.

The problem: occupancy-reports has to write its reports to a cloud storage bucket (s3://reports-rutasnorte/). The old way would be to create a permanent access key at the provider and store it in a Secret. That reintroduces every problem we saw in the Secrets lesson: a static credential that has to be rotated by hand and never expires.

The modern solution is called workload identity federation: the cloud provider trusts the Kubernetes cluster's token issuer and accepts the ServiceAccount token as proof of identity, handing back temporary credentials in exchange.

What it is called at each provider:

An example, just so you recognise the pattern:

With that annotation, the AWS SDK inside the pod obtains temporary credentials automatically, with no static key in any Secret. The Kubernetes ServiceAccount becomes an identity recognised outside the cluster too.

The advantages are the same ones you already know from the projected token: nothing permanent, automatic rotation, limited scope, and immediate revocation when the association is deleted. The detail of configuring it at each provider is the subject of Managed Kubernetes.

Tips:

1. Create the ServiceAccount next to the Deployment, in the same file. So there is never a component without its own identity.
2. automountServiceAccountToken: false by default in the base template. Make mounting the token a conscious decision, not inherited behaviour.
3. Audit periodically which pods carry a token. The jq command from section 7 should return only system components and those that genuinely need it.
4. Use kubectl auth can-i --list --as=... in reviews. It is the quick way to see an identity's effective permissions before approving a change.
5. Never grant permissions to the default. It is the fastest route to a privilege escalation across the whole namespace.

1. Create the six Rutas Norte ServiceAccounts in rutas-norte-pro with the module 2 labelling scheme, setting automountServiceAccountToken: false on all of them except occupancy-reports.
2. Modify the five existing Deployments so that they use their ServiceAccount and recreate the pods.
3. Show with a single command that each pod uses its own.
4. Show that web-store no longer has the token directory and that occupancy-reports does.
5. Write a command that lists every pod in the cluster that does have the token mounted and explain which of them are legitimate.

1. Create a pod called api-client with the occupancy-reports ServiceAccount and the token mounted.
2. From inside, query /version and /apis/authentication.k8s.io/v1/selfsubjectreviews and show the full identity.
3. Try to list the namespace's pods, capture the HTTP code and the full message, and identify the verb, the resource, the API group and the scope in it.
4. Decode the token payload and find sub, exp, the pod it is bound to and its lifetime in minutes.
5. Try the same request without the Authorization header and with a tampered token (change one character). Explain the codes you get in each case and why they differ.

You are asked to run an identity audit of the platform ahead of a security review.

1. Draw up a table with the six components showing: ServiceAccount, whether it mounts a token, whether it has imagePullSecrets and what effective permissions it has.
2. Identify which pods in the cluster (including kube-system) mount a token and classify them as "legitimate" or "to review".
3. Simulate an incident: an attacker manages to run code inside the web-store container. List what they can do with the current configuration and what they could have done before this lesson.
4. Explain why imagePullSecrets on the default ServiceAccount is acceptable but granting RBAC permissions to the default is not.
5. Propose three further hardening measures, saying in which lesson of the course each one is covered.

Solution 1

The k8s/base/serviceaccounts.yaml file is the one from section 4. Applying it and modifying the Deployments:

The audit of mounted tokens:

All legitimate: CoreDNS watches Services and Endpoints, kube-proxy does too, metrics-server publishes an aggregated API, the Ingress controller reads Ingress objects, and occupancy-reports is ours. From Rutas Norte, only one of six.

Solution 2

Inside the pod:

Breaking the message down:

The token payload:

With no header and with a tampered token:

The difference is subtle and instructive:

• No header → 403. It is not that the apiserver does not know who you are: it classifies you as the anonymous user system:anonymous, which is a valid identity with no permissions. It is a case of successful authentication as anonymous and denied authorization. (If the cluster had anonymous access disabled, you would indeed get a 401.)
• Tampered token → 401. The JWT signature does not validate, so the apiserver cannot establish any identity. It is a pure authentication failure.

That distinction is what will tell you, in a real incident, whether your problem is the credential or the permissions.

Solution 3

1. The identity table:

2. Classifying the pods with a token, using the command from solution 1: everything in kube-system and the Ingress controller are legitimate (they need to watch API objects in order to work). occupancy-reports is legitimate and expected. If any other rutas-norte-* pod showed up, it would be an immediate "to review".

3. The simulated incident. An attacker with code execution in web-store:

The fourth row is what gives the measure of the risk: in a cluster where somebody had granted permissions to the default "just to test", a vulnerability in a static file server would have allowed reading the database credentials holding the personal data of every Rutas Norte customer. Removing the token eliminates that route entirely.

The last row points at what this lesson does not solve: the network is still flat. Any pod in the namespace can open a TCP connection to bookings-postgres:5432. That is closed by the network policies.

4. imagePullSecrets on the default is acceptable because it grants pods no capability at all: it only lets the kubelet pull images from the private registry, which is an operation that happens before start-up and not an action the container process can exercise. The contents of that Secret are not even visible from inside the pod.

Granting RBAC permissions to the default is not acceptable because the default is, by definition, used by every pod that does not specify otherwise: any new pod, any debugging pod, any manifest pasted from the internet. A permission granted there is given to an open and unpredictable set of workloads, present and future. It is the opposite of an explicit decision.

5. Three further hardening measures:

All three, together with today's work, form the layers of a defence in depth: minimal identity, minimal permissions, minimal network and minimal container.

You have closed module 3 by giving the platform an identity. You know that Kubernetes distinguishes between users, which are not cluster objects and are managed by an external system, and ServiceAccounts, which are, are namespaced and identify themselves as system:serviceaccount:<ns>:<name>. You know the five reasons not to use the default — it is shared, it destroys traceability, it prevents selective revocation, it mounts a token almost nobody needs and it tempts people to grant it permissions — and you have created a dedicated ServiceAccount for each of the six Rutas Norte components, assigning it with serviceAccountName.

You understand how the token has changed: from those Secrets with eternal JWTs, tied to no pod and stored in etcd, to the projected tokens of the TokenRequest API, which expire in an hour, are rotated by the kubelet at 80% of their lifetime, are bound to a specific pod UID, live only in tmpfs and are invalidated when the pod is deleted. You know what lives in /var/run/secrets/kubernetes.io/serviceaccount/ — token, ca.crt and namespace — and what each file is for, including the ca.crt that saves you from using --insecure. And you are clear about the precedence of automountServiceAccountToken, with the recommended pattern: false on the ServiceAccount as a safety net and an explicit true only where it is genuinely needed. In Rutas Norte, five of six components no longer carry any credential inside, which eliminates at the root the first move of any privilege escalation script.

You have talked to the API from inside a pod with curl, assembling the request by hand with the CA, the token and the kubernetes.default.svc Service; you have asked who you are with SelfSubjectReview; and you have received a 403 Forbidden that you can break down into verb, resource, API group and scope, distinguishing it from the 401 that signals a credential problem rather than a permissions one. That 403 was not a failure: it was the demonstration that Kubernetes denies by default and that creating an identity grants absolutely nothing. You have moved imagePullSecrets onto the ServiceAccount so it is neither repeated nor forgotten, and you know that identity federation with the clouds exists so that even external credentials need not be static.

With this you close module 3 and the Rutas Norte platform is in far better health than it was six lessons ago. The configuration lives outside the image, in per-environment ConfigMaps; the bookings-postgres password is no longer in Git but in a Secret mounted in memory with only the keys each component needs; the environment variables are catalogued component by component and environment by environment, with the Downward API giving traceability to every request; the three environments have a quota and a LimitRange, so a mistake in development cannot touch production; every pod has a QoS class assigned on business grounds, with bookings-postgres as Guaranteed and last survivor; and each component has its own identity before the API, almost always with no credential mounted.

But the platform still has a very visible hole: the network is completely flat. Any pod in rutas-norte-pro can open a connection to bookings-postgres:5432, and no client on the internet can yet reach www.rutasnorte.example or api.rutasnorte.example, because our Services are all ClusterIP and exist only inside the cluster. Module 4, Networking in Kubernetes, tackles that from end to end: how the cluster network really works and the "every pod with its own IP" model, the Service types beyond ClusterIP, the internal DNS we have been using without fully explaining it, the Ingress controllers that will finally publish the store and the API on their domains, the TLS certificates managed automatically with cert-manager, and the network policies that will stop anyone who is not bookings-api or notifications-worker from even attempting to connect to the database holding the customers' personal data.