When we closed module 7 we left the Rutas Norte platform fully observable: probes that detect a sick process, metrics in Prometheus, dashboards in Grafana, alerts routed by Alertmanager and every log centralised in Elasticsearch. We can see absolutely everything that happens. And yet, right now, anyone with access to the cluster can read the Secret holding the credentials of bookings-postgres —the database that stores the name, the ID number, the phone and the email of every customer who has bought a ticket—, can deploy a privileged container that escapes to the node, or can push an image nobody has reviewed. And we would not know who did it.

This whole module is devoted to closing those doors. And the first one, the most important, is this: deciding who can do what. In Kubernetes that is called RBAC (Role-Based Access Control) and it is the mechanism that turns a cluster where everyone can do everything into a cluster where each person and each process has exactly the permissions they need and not one more.

In module 3 we left two loose threads that we now pick up. When talking about Secrets we said that who can read a secret is decided by RBAC. When talking about ServiceAccounts we said that the ServiceAccount answers who you are and RBAC answers what you can do, and we saw a 403 Forbidden proving that Kubernetes denies by default. This lesson explains exactly why that 403 happened and how to grant, with surgical precision, only what is strictly necessary.

Important warning. This lesson teaches the mechanics of RBAC and proposes an example design for a fictional company. The real permission design of a production cluster must be reviewed by a security professional, and if the cluster handles personal data —as is the case with bookings-postgres— also by the organisation's compliance officer. A permissions mistake goes unnoticed until somebody takes advantage of it.

1. The three gates: authentication, authorization and admission
2. The subjects: users, groups and ServiceAccounts
3. The four RBAC objects
4. Anatomy of a rule: apiGroups, resources, verbs and resourceNames
5. Subresources: the hidden power of pods/exec and pods/log
6. Default roles and aggregated roles
7. Checking permissions with kubectl auth can-i
8. The Rutas Norte RBAC design
9. Why list on secrets is the same as reading them
10. Privilege escalation and Kubernetes' protections
11. Auditing and periodic permission reviews
12. Common mistakes and tips
13. Exercises
14. Conclusion

Everything in Kubernetes goes through the apiserver. When you type kubectl get pods, when the kubelet reports a node's status, when the Deployment controller creates a ReplicaSet, when occupancy-reports queries the list of pods: they are all HTTP requests to the same API. And every request crosses three gates before reaching etcd.

Let us go gate by gate.

It answers who are you?. The apiserver examines the credentials that come with the request and produces an identity: a user name, a list of groups and, optionally, some extra attributes. The usual mechanisms are:

If none of them works, the request stays as system:anonymous and is normally rejected with 401 Unauthorized.

It answers may you do this?. The apiserver takes the identity from the previous gate and the details of the request (verb, API group, resource, namespace, name) and consults the configured authorizers. RBAC is the authorizer that practically everyone uses, although others exist (Node, ABAC, Webhook, AlwaysAllow). If no authorizer says "yes", the answer is 403 Forbidden. There is no implicit permission: Kubernetes denies by default. That was exactly the 403 we saw in 03-06.

One essential detail: RBAC only grants, it never denies. There are no deny rules. A subject's effective permission is the union of everything granted by all of their bindings. That is why removing a permission means removing the binding that grants it, not adding a rule that forbids it.

It answers is this object acceptable exactly as it arrives?. This is where the mutating admission controllers act (they can modify the object, like the one that injects the default ServiceAccount) and the validating ones (which only accept or reject). It is where the ResourceQuota of module 3, Pod Security Admission and the Kyverno policies live.

This third gate is the topic of 08-03; here we only name it so you can see where RBAC fits. Remember the key difference:

RBAC decides whether you have the right to perform the operation. Admission decides whether the object you send complies with the rules. You may have permission to create pods (RBAC says yes) and still see a privileged pod rejected (admission says no).

RBAC grants permissions to subjects. There are exactly three kinds.

Here comes the surprise that confuses almost everybody at the start:

Kubernetes does not store users. There is no User object. You cannot run kubectl create user anna. There is no database of people inside the cluster.

A user is simply a text string produced by the authenticator. If you authenticate with a certificate whose CN=anna.garcia, then as far as RBAC is concerned you are the user anna.garcia. If you authenticate with OIDC and the provider returns the email [email protected], that will be your user name. The cluster trusts the authenticator, full stop.

The practical consequence matters: adding and removing people is the identity provider's responsibility, not Kubernetes'. If Anna leaves the company and you only delete her RoleBinding, but her certificate is still valid and she had permissions left through another group binding, she still gets in.

Same as users: they are strings produced by the authenticator. With an X.509 certificate, each O (organisation) field becomes a group. With OIDC, a claim such as groups is usually configured.

You should almost always grant permissions to groups, not to users. If Rutas Norte hires someone new on the platform team, you add them to the platform group in the identity provider and they already have the permissions. Not a single cluster YAML needs touching.

Kubernetes reserves some groups with a meaning of their own:

Extreme care with system:masters. The apiserver grants it full permission before consulting RBAC, so it cannot be limited or revoked with a manifest. It is the identity of the administrator kubeconfig generated by kubeadm. That file must be guarded like a master key: kept off everybody's laptop, with logged access, and used only to recover the cluster when RBAC is broken.

They are the identities of the processes running in the cluster, and they are indeed real Kubernetes objects (we created them in 03-06). In RBAC they are referenced in two equivalent ways:

Comparative summary:

RBAC has only four object types, and all of them live in the API group rbac.authorization.k8s.io/v1. The idea has a very clean symmetry:

• Role and ClusterRole describe what can be done. They are lists of permissions, and they mention nobody.
• RoleBinding and ClusterRoleBinding describe who can do it. They join some subjects to a role.

Look closely, because 90% of the confusion with RBAC is right here:

The second row is the surprising one and at the same time the most useful. A ClusterRole grants nothing on its own: it is just a permission template. When you reference it from a RoleBinding, its rules apply only inside that binding's namespace. This lets you write the role once and reuse it in many namespaces.

With those two manifests, the support group can list pods in rutas-norte-pro and nowhere else. If tomorrow we wanted to give it read access in rutas-norte-pre too, another identical RoleBinding with a different namespace would be enough: the ClusterRole is not touched.

Some resources do not belong to any namespace: Node, PersistentVolume, StorageClass, Namespace, ClusterRole, CustomResourceDefinition... To grant permissions over them a ClusterRole referenced from a ClusterRoleBinding is mandatory. A RoleBinding can never give access to a global resource, no matter how much the ClusterRole mentions it.

One detail about roleRef worth knowing: it is immutable. You cannot change which role an existing binding points to; you have to delete it and create it again. The subjects list, on the other hand, can be modified.

An RBAC rule is the intersection of four dimensions. All of them must match for the permission to apply.

Every Kubernetes resource belongs to an API group. The core group (also called legacy) is historical and is represented by the empty string "", not by core. That is where Pods, Services, ConfigMaps, Secrets, ServiceAccounts, Nodes, PersistentVolumeClaims and Namespaces live.

Writing apiGroups: ["core"] is a very common silent mistake: it does not fail when applying the manifest, it simply grants nothing, because there is no group called core.

To find out the group and the exact plural name of any resource:

When APIVERSION is just v1 (no slash), the group is the empty one "". When it is apps/v1, the group is apps. The NAMESPACED column tells you whether you need a Role or necessarily a ClusterRole.

In a rule you always use the lowercase plural name from the NAME column, never the KIND nor the short name. resources: ["Pod"] or resources: ["po"] grant nothing.

And three special verbs that do not correspond to a normal operation:

All three are, in practice, privilege escalation routes. We will come back to them in section 10.

An important warning about deletecollection: many people overlook it when building a role for "being able to delete specific things". If you grant delete with resourceNames but also deletecollection without restriction, you have opened the door to deleting everything, because resourceNames does not apply to collection operations.

It lets you confine a rule to objects with specific names:

With this role you can run kubectl get configmap web-store-config but not kubectl get configmap any-other-one.

The fundamental limitation of resourceNames. It does not work with list, watch, create or deletecollection. The reason is simple: when you ask for a list, you do not yet know which objects exist, so the authorizer cannot filter by name —it can only allow or deny the whole request—. RBAC is not a content filter.

Practical consequence: resourceNames restricts individual reads, but it does not prevent a listing if you have granted list. If your intention is "they should only see this ConfigMap", you have to grant get with resourceNames and not grant list. The trade-off is that kubectl get configmaps (without a name) will fail with a 403, which baffles users; it has to be documented.

Even with that limitation, resourceNames is very valuable in one specific case: giving a process permission to update exactly one object. We will use it with ci-rutasnorte.

Some resources have subresources: child API paths with their own access control. They are written with a slash in the resources field.

The key point that is constantly forgotten:

get pods and get pods/log are different permissions. Being able to see that a pod exists gives no right to read its logs. And pods/exec is not granted by the get verb on pods: it has to be named explicitly. This is good news, because it lets you give visibility without giving access.

An example directly applicable to Rutas Norte: the support team needs to see whether a pod is down and read its logs to answer a customer who says the confirmation email never arrived. It does not need to get inside the container.

Notice what is not there. A security role is judged as much by what it omits as by what it includes. If support had pods/exec on bookings-api, it could open a shell, read the process's environment variables and obtain the connection string to the customer database. The boundary between "seeing the status" and "reading personal data" is exactly that line of YAML.

And be careful with pods/log even so: in 07-05 we explicitly forbade logging personal data precisely because a log-read permission is far easier to grant than a database-read one. The two decisions reinforce each other.

Every cluster ships with dozens of predefined ClusterRole objects. Many are internal (system:kube-scheduler, system:node...), but four are meant for people:

Two nuances that decide entire designs:

1. edit can read Secrets. Many people grant edit in production thinking "it only edits, it does not administer", and with that they have handed over the customer database credentials. If this matters to you —and at Rutas Norte it matters a great deal—, you cannot use edit as-is in rutas-norte-pro.
2. view does not see Secrets (that was an explicit project decision), but it does see ConfigMaps. This reinforces what we said in 03-01 and 03-02: sensitive data goes in Secrets, never in ConfigMaps, even if it is "only a connection string without a password".

Checking exactly what a role does before using it is a mandatory habit:

Some default roles have no rules of their own: they are composed automatically from the rules of every ClusterRole carrying certain labels. That is the aggregationRule.

The aggregation controller watches the ClusterRole objects with that label and copies their rules into view. This is enormously useful when you install CRDs: you can make whoever has view also see your custom resources without editing the view role (which would be overwritten on every cluster upgrade).

A real example for Rutas Norte: in module 7 we installed the Prometheus operator, which brings ServiceMonitor and PrometheusRule. We want support and development to be able to see them.

Once applied, anyone who has view in any namespace starts seeing the ServiceMonitor objects of that namespace, without touching a single binding. Note the hierarchy: admin aggregates what edit has, and edit aggregates what view has, so if you only set aggregate-to-view, all three inherit it anyway. Setting all three is explicit and does no harm.

Writing RBAC without verifying it is like writing tests without running them. kubectl auth can-i answers the exact question the apiserver asks itself.

The truly powerful part is --as and --as-group, which ask on behalf of another identity (this uses the impersonation mechanism, so you need the impersonate verb or to be an administrator):

You can also check ServiceAccounts, which is how a process's permissions are audited:

And a full view of everything a subject can do in a namespace:

That output is exactly the "permission sheet" to review in every audit. If something you did not expect shows up, you have one binding too many.

Stable since 1.28, it answers "who does the cluster think I am?". It is the tool for diagnosing authentication problems, which are constantly mistaken for authorization problems.

A useful diagnostic pattern:

A verification script worth keeping in the repository next to the manifests, so it can be run after every RBAC change:

Running this script on every change turns RBAC into something verifiable. Without it, a badly copied roleRef can go unnoticed for months.

Now we apply everything to the company's four groups of people and to the processes' ServiceAccounts. The guiding principle is least privilege: each subject gets just enough for its job.

Summary of the design:

In rutas-norte-dev they need to work freely. We use the edit ClusterRole confined with a RoleBinding:

It is acceptable because in rutas-norte-dev there is no real customer data: only synthetic data generated for testing. That statement is a design decision that must be documented and verified, not an assumption. If one day somebody copied a production dump into development, this binding would become a serious incident. It is exactly the kind of rule the compliance officer must be aware of.

In rutas-norte-pro the approach changes completely. Developers need to investigate —see the status, read logs, check events— but not touch anything or read credentials.

And the same ClusterRole, plus the log permission, is reused in rutas-norte-pre without writing new rules. That is the advantage of the RoleBinding → ClusterRole combination.

It administers the three namespaces. We could give them cluster-admin, but that is excessive and makes it impossible to tell a routine operation from an anomalous one in the audit log (08-06). We prefer admin per namespace plus a ClusterRole for reading infrastructure:

cluster-admin is reserved for an emergency procedure with logged access and prior approval. In many organisations this is implemented with temporary access (just-in-time): an external system creates the ClusterRoleBinding, announces it on a public channel and deletes it automatically after two hours.

We already wrote its ClusterRole in section 5. Only the binding is missing, exclusively in production:

This is the most interesting case, because CI is a very tempting target: if somebody compromises the CI system and it has cluster-admin, they have the cluster. We give it exactly what it needs in order to deploy: update the image of the existing Deployments and StatefulSets. Nothing else.

Let us analyse the decisions:

• No create: new objects are created by someone from platform after review. CI only updates what already exists and has been reviewed.
• No delete or deletecollection: a compromised CI cannot delete the platform.
• No secrets: CI does not read cluster credentials. Its own live in the pipeline's secret manager.
• No pods/exec: there is no way for CI to open a shell in production.

If we wanted to be stricter still, we could confine it with resourceNames to the exact workload names:

That is the correct pattern when listing is needed: two rules, a restrictive one for the by-name operations and another with only list.

The occupancy-reports CronJob generates a nightly report on seat occupancy. It does not need to talk to the Kubernetes API at all, so the right thing is for it not even to mount the token:

A process having no binding at all is the ideal situation and more common than people think. web-store, notifications-worker and redis-cache are in the same position.

bookings-api does need to read one specific ConfigMap in order to reload its route configuration on the fly:

And the PostgreSQL operator from module 6, which does need broad permissions because its job is to manage StatefulSets, Services and PVCs. Remember what we said in 06-07: before adopting an operator you have to read what RBAC it asks for. This is the reasonable minimum, confined to one namespace:

If the operator you are evaluating demands a ClusterRole with secrets across the whole cluster and its documentation does not explain why, that is a legitimate reason to reject it or to deploy it scoped to a single namespace, if it supports that.

This is one of the points most people misunderstand, and it has direct consequences for Rutas Norte's customer data.

When you run kubectl get secrets, kubectl issues a GET /api/v1/namespaces/<ns>/secrets. The apiserver returns a SecretList with the complete objects, including the data field with all the values. That kubectl shows you a summarised table without the values is merely a client-side presentation decision. The data already travelled over the network to your machine.

As we said in 03-02, base64 is not encryption: it is encoding. A single command reverses it.

A rule that admits no exceptions: granting list on secrets is granting read access to every secret in that scope. There is no way to list secrets "without seeing their contents". If a role has list on secrets in rutas-norte-pro, whoever holds it can read the customer database credentials. Treat it as seriously as handing over the password.

The same applies to watch, which delivers the complete objects on every change.

That is cluster-admin. But even "small" wildcards are dangerous:

The last row is especially treacherous: a wildcard in verbs on pods does not include the subresources (they must be named), but resources: ["pods/*"] does include exec, attach and portforward. The difference is worth memorising.

And there is a time-related problem: a wildcard grants permissions over resources that do not yet exist. If you install a new CRD next month, anyone with resources: ["*"] will control it from day one, without anybody having decided anything.

The correct alternative is always to enumerate. It takes longer to write and is vastly easier to audit.

By default, Kubernetes stops you from creating or modifying a role with permissions you do not have. If you could, anyone with permission over roles would be a de facto cluster-admin: they would write a role with * and assign it to themselves.

The exact rule is: to create or modify a role, you must already hold every permission that role grants (in the corresponding scope). There are two deliberate exceptions:

bind, confined with resourceNames, is the elegant mechanism for delegation. At Rutas Norte we could let team leads grant the support role without being administrators:

Whoever holds this role can create bindings to the rutasnorte-support role and to no other. They cannot bind admin, nor cluster-admin. It is a safe delegation.

This is probably the most underestimated implication in all of RBAC:

If you can create pods in a namespace, you can run code with the identity of any ServiceAccount in that namespace, simply by putting its name in spec.serviceAccountName.

Creating pods is not protected by the "you cannot grant what you do not have" rule, because technically you are not creating a role: you are creating a pod. But the practical effect is identical to inheriting that ServiceAccount's permissions.

Concrete consequences for our design:

1. create pods in rutas-norte-pro is a high-privilege permission. That is why ci-rutasnorte does not have it and neither does development in production.
2. Never put a highly privileged ServiceAccount and ordinary workloads in the same namespace. The PostgreSQL operator, which can read secrets, shares a namespace with web-store. Whoever can create a pod there inherits the operator's permissions. The right thing is to isolate operators in their own namespace.
3. Verbs that create pods indirectly count just the same: creating Deployments, StatefulSets, DaemonSets, Jobs or CronJobs is creating pods by delegation. edit includes all of them.

Other escalation routes worth knowing so you can close them:

Auditing these specific permissions is a routine task with a very good return:

That list should fit on one screen and every line should have a known justification. The day it no longer fits, the trouble begins.

RBAC decays over time. A temporary permission nobody withdrew, a binding created to debug an incident that stayed, a group that grew. The only defence is systematic review.

The second line is exactly what needs catching: an emergency access that was meant to last two hours and is still there. The convention of putting the date in the binding's name makes it stand out.

Only platform. That is the result we were after and we must be able to prove it at any moment: this output, saved with a date, is valid evidence for a compliance audit.

When a person leaves the company, their binding stays. When a ServiceAccount is deleted, its bindings stay too. A simple check:

There are dedicated utilities that make this job far more comfortable:

A reasonable cadence for a platform like Rutas Norte:

Every RBAC manifest must live in the repository, under k8s/base/rbac/, and be applied only from there. A binding created by hand with kubectl create rolebinding is invisible to code review and is exactly where the problems come in. In module 10 we will see, with GitOps, how to make the cluster effectively reject anything that is not in Git.

Writing apiGroups: ["core"]. It does not exist. The core group is the empty string "". The manifest applies without error and grants nothing; then you spend an hour looking for the reason behind a 403.

Using the Kind or the short name in resources. You must use the lowercase plural: pods, not Pod or po; deployments, not Deployment. Check it with kubectl api-resources.

Believing that a RoleBinding to a ClusterRole gives global permissions. It does not: it confines them to the binding's namespace. That is the exact opposite of what the name suggests, and it is the most useful of the four combinations.

Trying to change the roleRef of an existing binding. It is immutable. You have to delete and recreate. A kubectl apply that changes the roleRef fails with a rather uninformative error.

Granting edit in production "because it only edits". edit reads and creates Secrets, and creates pods (which means it can impersonate any ServiceAccount in the namespace). In a namespace holding personal data it is an administration permission in disguise.

Assuming that resourceNames protects listings. It does not apply to list, watch, create or deletecollection. If you grant list without names, everything is visible.

Forgetting that list on Secrets is reading them. There is no way to list them without seeing the contents.

Adding wildcards "to get by for now". The temporary permission stays, and on top of that it grants access to resources that do not yet exist. If you need to unblock somebody quickly, grant the exact permission they are missing and open a ticket to review it.

Granting permissions to users instead of groups. Every joiner and every leaver becomes a manifest change, and leavers are always forgotten.

Confusing 401 with 403. 401 is authentication (I do not know who you are), 403 is authorization (I know who you are and you may not). kubectl auth whoami tells the two cases apart in a second.

Not verifying. Always write the negative checks: it is not enough to confirm that support can read logs, you have to confirm that it cannot exec or read Secrets. Security failures live in the extra permissions, not the missing ones.

Putting a privileged operator in the same namespace as the applications. Whoever can create a pod there inherits the operator's permissions. Separate namespace.

Golden tip: always start from zero permissions and add only what fails. It is slower on the first day and vastly safer on all the others. With kubectl auth can-i --list you will know exactly where you stand at any moment.

Rutas Norte takes on an intern in the support team. You need a ClusterRole called rutasnorte-support-junior and a binding that applies it only to rutas-norte-pre. It must be able to:

• List and view pods, services and deployments.
• Read the pods' logs.

And it must not be able to:

• Read Secrets.
• Run kubectl exec or kubectl port-forward.
• Modify anything.

Write the manifests and the verification commands that prove the three prohibitions.

A new metrics agent, inventory-agent, runs as a Deployment in rutas-norte-pro with its own ServiceAccount. It needs to list pods and nodes across the whole cluster in order to build a resource inventory, and nothing more. Write the ServiceAccount, the ClusterRole, the ClusterRoleBinding and a check that proves it cannot read Secrets or create anything.

Additional question: why does this case really need a ClusterRoleBinding rather than just a RoleBinding?

You find this Role applied in rutas-norte-pro:

It is bound to the analytics group, whose sole remit is to consult the occupancy metrics through the ServiceMonitor objects and to see the status of the pods.

1. List all the security problems with this role.
2. Rewrite it with least privilege.
3. Write the checks that prove the new version still serves its purpose and no longer exposes personal data.

Solution 1

Verification:

The last check is the one that proves the RoleBinding confines the ClusterRole: in rutas-norte-pro it has absolutely nothing.

Solution 2

Verification:

Why a ClusterRoleBinding is needed: for two independent reasons.

1. nodes is a non-namespaced resource. A RoleBinding can never grant access to global resources, not even by referencing a ClusterRole that mentions them.
2. The agent must list pods from every namespace. A RoleBinding would confine the read access to the binding's namespace, and it would have to be replicated in every existing and future namespace.

Had the requirement been only "pods in rutas-norte-pro", a RoleBinding would have been the correct and safer option.

Solution 3

1. Problems with the original role:

2. Least-privilege version:

Notice that we have also removed pods/log: the stated remit is to consult metrics, and logs could contain customer information (even though we forbade it in 07-05, the minimum permission does not depend on that prohibition always being honoured).

3. Verification:

Since the old role granted access to the customer database credentials, fixing the role is not enough: you have to review the audit log to find out whether anybody actually read that Secret, and if so, rotate the credentials and notify the compliance officer. How that query is done is the subject of 08-06.

RBAC is the first and most important line of defence of a Kubernetes cluster, and you now have the complete mental model:

• Every request crosses three gates: authentication (who you are), authorization (what you may do, where RBAC acts) and admission control (whether the object is acceptable).
• The subjects are users and groups, which Kubernetes does not store and which arrive from the authenticator, and ServiceAccounts, which really are cluster objects.
• Four objects: Role/ClusterRole say what, RoleBinding/ClusterRoleBinding say who. The RoleBinding → ClusterRole combination is the most useful: write the role once, confine it wherever you like.
• A rule is the intersection of apiGroups (with the empty core group ""), resources (including subresources such as pods/log and pods/exec), verbs and optionally resourceNames (which does not apply to list).
• list on Secrets is reading them. Wildcards grant more than you think, including over resources that do not yet exist.
• Creating pods is equivalent to being able to use any ServiceAccount in the namespace; that is why privileged operators go in their own namespace.
• kubectl auth can-i --as/--as-group and kubectl auth whoami turn RBAC into something verifiable, and those checks must run automatically on every change.

At Rutas Norte we have gone from "anyone can read the customer database credentials" to a design where only platform can do so, support sees the status and the logs without being able to get inside the containers, development investigates production without touching it, and the continuous integration pipeline can only update the image of workloads that already exist.

But we have just uncovered an uncomfortable limit: RBAC decides whether you have the right to create a pod, not what that pod looks like. Someone from platform, with perfectly legitimate permissions, can deploy a container today with privileged: true, mounting the node's / with hostPath and running as root. RBAC will say yes, because they have permission to create pods. And from there the whole node —and every pod running on it, bookings-postgres included— is laid bare.

The next lesson, 08-02, Security Contexts and Container Hardening, tackles exactly that problem: what a container really isolates, why it is not a virtual machine, and how the securityContext of each Rutas Norte component is configured so that, even if somebody manages to run code inside a container, they cannot get out of it.