We closed the previous lesson pointing at a loose thread that has run through the whole module. The selector of a ReplicaSet adopting pods, the selector of a Service finding its destinations, the environment label network policies will use to select namespaces, the kubernetes.io/change-cause annotation documenting every revision, the pod-template-hash the Deployment adds by itself, and all the -l queries we have been using since the first lesson: all of that is labels and annotations. We have been using them out of necessity, without any system. This lesson puts them in order, and it is no minor matter: in a cluster with six components across three environments, the labelling scheme is what separates a navigable platform from a chaos of anonymous objects. You will see the exact difference between a label and an annotation, the syntax and constraints of keys and values, the labels Kubernetes recommends and how they take shape in Rutas Norte's official scheme, equality- and set-based selectors in all their forms, who consumes them, why a Deployment's selector is immutable, the real use of annotations and the everyday queries that will make you productive.

1. Labels versus annotations
2. Syntax and constraints of keys and values
3. The labels Kubernetes recommends
4. The Rutas Norte labelling scheme
5. Equality-based selectors
6. Set-based selectors
7. Selectors in manifests: matchLabels and matchExpressions
8. Who consumes selectors
9. The golden rule: a Deployment's selector is immutable
10. Annotations in practice
11. kubectl label and kubectl annotate
12. Useful everyday queries

Both live in metadata and both are key-value pairs. That is where the resemblance ends.

The decision rule, which resolves 100 % of doubts:

Will you want to search for objects by this data, or will a selector use it? It is a label. Is it information that gets read but not searched? It is an annotation.

Examples applied to Rutas Norte:

Kubernetes is strict about the format, and knowing the rules avoids baffling rejections.

A key has a name and, optionally, a prefix separated by /:

Name rules (the mandatory part):

• Maximum 63 characters.
• It must begin and end with an alphanumeric character.
• It can contain -, _, . and alphanumerics in between.

Prefix rules (optional):

• It must be a valid DNS subdomain: rutasnorte.example, app.kubernetes.io.
• Maximum 253 characters.
• It is separated from the name by /.
• Reserved: the prefixes kubernetes.io/ and k8s.io/ are reserved for Kubernetes itself and its core components. Do not use them for your own labels.

• Maximum 63 characters (or empty, which is valid).
• If it is not empty, it must begin and end with an alphanumeric.
• Only alphanumerics, -, _ and . in between.
• Forbidden: spaces, /, :, @, accents, ñ, emojis.

Valid and invalid examples:

Trigger the rejection yourself:

The correct version uses an annotation:

Keys follow the same rules as label keys. Values, on the other hand, are almost free: any string, including JSON, line breaks and special characters, with a combined limit of 256 KiB for all of an object's annotations.

That generous limit is what lets kubectl apply store the whole manifest in kubectl.kubernetes.io/last-applied-configuration, as we saw in Objects and Manifests.

Kubernetes defines a set of common labels under the app.kubernetes.io/ prefix. They are not mandatory and nothing enforces them, but they are the shared vocabulary understood by Helm, Kustomize, Argo CD, Prometheus and practically every tool in the ecosystem.

The difference between name and instance is the hardest one:

• name is what it is: postgres. The same across every installation.
• instance is which one it is: bookings-postgres-pro. It tells two installations of the same thing apart.

If Rutas Norte had two PostgreSQL databases —one for bookings and another for billing—, both would carry name: postgres but instance: bookings-postgres and instance: billing-postgres.

A complete example with all six:

A practical note: version: "2.5.0" goes in quotes. Without them, YAML could read values such as 1.0 as a number and cause a type error, since label values must be strings.

An important warning that ties in with section 9: app.kubernetes.io/version must not go in the selector. If it did, changing version would stop the selector matching its own pods, and since the selector is immutable, you would have to recreate the Deployment on every rollout. The version is informative; identity comes from name and instance.

The time has come to settle the project's definitive scheme. So far we have used three minimum labels (app, app.kubernetes.io/part-of and environment); we consolidate and extend them.

The scheme's golden rule, which follows from everything learned in the module:

Only app and environment go into the selectors. They are the only two labels that identify the component unambiguously and that never change.

Everything else —version, component, criticality, manager— is informative and can change without recreating anything.

This is how bookings-api looks with the full scheme applied:

Note the asymmetry, exactly the one we learned about with ReplicaSets: the template carries nine labels and the selector demands only two. It is allowed and it is correct: the selector must be a minimal, stable subset; the other labels travel with the pods so they can be queried.

Apply it and observe the immediate power:

A selector is a query over labels. The simplest kind compares values.

They are used with -l (or --selector), and several conditions separated by commas are combined with a logical AND:

A detail about != worth knowing: it also selects objects that do not have that label at all. environment!=pro returns those in dev and pre as well as those lacking the environment label.

There is no logical OR in equality-based selectors. That is what set-based ones are for.

More expressive. They operate on sets of values and on the existence of the key.

Applied examples:

A note on quoting: set-based selectors carry parentheses and spaces, which the shell would interpret. Always wrap them in single quotes.

And they can be combined with equality-based ones, separated by commas:

Inside a YAML, selectors have two forms equivalent to the two families you have just seen.

It is a map: every key must match (logical AND). It is what we have used throughout the module.

Both can be used at once; every condition is combined with a logical AND:

Translation: "pods with app=bookings-api and with environment equal to pre or pro".

Mind the Service exception, which we already saw in its lesson and is worth repeating here because it is a real asymmetry in the API:

The reason is historical: Services belong to the core v1 group, which predates the enriched selector, and they only support equality.

Selectors are not only for manual queries. They are the coupling mechanism of the whole of Kubernetes: the way some objects find others without knowing their names.

The underlying lesson: a pod's labels are not decoration, they are its interface with the rest of the system. Changing a pod's label can take it out of a Service, leave it outside a network policy or make it invisible to its ReplicaSet. We will see this in exercise 3.

Check it:

The same goes for ReplicaSets, StatefulSets and DaemonSets. Services are the exception: their selector can be modified, and that makes them the tool behind module 11's blue-green strategies.

The reason is the safety of the data... of the pods. Imagine changing it were allowed:

1. The Deployment has selector: app=bookings-api and governs 4 pods with that label.
2. You change the selector to app=bookings-api-v2.
3. Instantly, the 4 pods stop matching. They become orphans, exactly like those in the --cascade=orphan experiment of the ReplicaSets lesson.
4. The Deployment counts 0 pods of its own and creates 4 new ones.
5. Result: 8 pods running, 4 of them with no controller, invisible to the Deployment, consuming resources and with nobody to update them or bring them back if they crash.

Worse still: if the Service was still pointing at app=bookings-api, those 4 orphans would keep receiving production traffic with no controller watching them.

Kubernetes prefers to fail on apply rather than let you reach that state.

There are two routes, and the first is the good one:

Route A: replace the controller without interrupting the service.

Route B: a new name and a transition through the Service. You create a Deployment under another name with the correct selector, wait for it to be ready and change the Service's selector to point at the new pods. It is, in essence, a blue-green rollout (11-04).

Think the selector through before the first apply. It is the least reversible decision in a manifest.

Hence the project rule you already know: in the selector, only app and environment. Never the version, never the manager, never the criticality, never part-of. Anything that can change over time stays out.

Annotations look less important than labels until you discover how many things work thanks to them.

This is the most powerful use: many controllers are configured through annotations, not through spec fields. The textbook case is the Ingress controller:

None of those lines is in the Ingress schema: they are instructions the NGINX controller and cert-manager read and apply. The reason for this design is that each Ingress implementation has different capabilities, and the standard API cannot cover them all. They are covered in Ingress Controllers and TLS and Certificates.

And the simplest, most profitable use: information that saves a night on call.

Note two things impossible with labels: values with @, : and /, and a multi-line value. That is why annotations exist.

At three in the morning, this is worth its weight in gold:

Never store passwords, tokens, API keys or private certificates in annotations.

The reasons:

• They are read in plain text with kubectl get -o yaml or kubectl describe, permissions that are usually granted widely.
• They show up in logs: any dump of an object includes them.
• They are copied into last-applied-configuration, ending up duplicated.
• They travel to backups and GitOps tooling, spreading everywhere.
• They are not encrypted at rest inside etcd unless the cluster is configured for it, and even then the mechanism is designed for Secrets.

The same applies to labels, with the aggravating factor that they are also indexed and queryable.

That is what Secrets are for, in lesson 03-02, and their secure management in module 8. That plain-text POSTGRES_PASSWORD we left in the provisional bookings-postgres Deployment is exactly the debt we are going to pay in the next module.

Both commands share a syntax.

If the key already exists, the command fails unless you say otherwise:

It is a deliberate protection against accidentally trampling a label some selector is using.

That trailing hyphen is easy to miss when reading a script. Remember it: a key with a trailing hyphen means delete.

A safety warning: --all combined with --overwrite on a label that appears in some selector can pull pods out of their Service or their ReplicaSet on the spot. Always try --dry-run=server first:

Changing a label with kubectl label on a pod managed by a Deployment does not survive: on the next rollout, the pods are recreated from the template and the change disappears. To make it last, edit spec.template.metadata.labels in the manifest.

And there is a far more serious side effect, which we will explore in exercise 3: if you change a label that is in the ReplicaSet's selector, the pod is orphaned instantly and the ReplicaSet creates another to replace it.

This is where the labelling scheme stops being bureaucracy and becomes productivity. The most profitable queries combine -l with the output formats we saw in The Kubernetes CLI.

An important syntax trick: in custom-columns, dots that are part of the label key must be escaped with \.. Otherwise, kubectl reads them as JSONPath path separators and finds nothing.

The query you will be asked for most often:

Finding what breaks the project convention:

That --dry-run=client before a bulk delete is a habit worth acquiring.

The -L flag (capital L) is a small treasure: it adds columns with the value of the given labels, without filtering anything.

• Putting the version in the selector. On rolling out a new version the selector stops matching, and since it is immutable, it forces you to recreate the Deployment. The selector carries only app and environment.
• Overly broad selectors. An app.kubernetes.io/part-of: rutas-norte as a selector adopts and can delete other components' pods, as we saw in ReplicaSets.
• Using forbidden characters in label values. No @, :, /, spaces or accents. If the data needs them, it is an annotation.
• Using the kubernetes.io/ prefix for your own labels. It is reserved. Use your organisation's domain: rutasnorte.example/.
• Storing secrets in annotations or labels. They are read in plain text with describe, they travel to backups and they end up in Git. They go in Secrets.
• Putting the labels only in metadata and not in spec.template.metadata. Pods inherit the template's, not the Deployment's. It is the mistake that breaks Services.
• Forgetting --overwrite and assuming the command worked. kubectl label fails if the key exists. Always read the output.
• Changing a label that is in a selector with kubectl label. The pod is orphaned and the controller creates another. It can double capacity without you noticing.
• Expecting a kubectl label change on a pod to last. It is lost on the next rollout. Edit the template.
• Forgetting the single quotes in set-based selectors. The shell interprets the parentheses and spaces.
• Not escaping the dots in keys within custom-columns. app.kubernetes.io/version must be written app\.kubernetes\.io/version.
• Tip: define the labelling scheme before deploying anything. Adding it later means editing every manifest and, if it touches selectors, recreating objects.
• Tip: kubectl get pods -A -L environment,app gives you a tabulated view of the whole platform without filtering. It is the query to start any review with.
• Tip: document the scheme in the repository's k8s/README.md. A scheme only its designer knows is not a scheme.

1. Update k8s/base/web-store-deployment.yaml so that the Deployment and its template carry the eight labels of the Rutas Norte scheme (app, environment, the five app.kubernetes.io/ ones and criticality), plus four annotations: owner, dashboard, runbook and change-cause.
2. Make sure the selector still has only app and environment, and explain in two lines why.
3. Apply it and prove with one command that the pods carry the eight labels.
4. Retrieve the runbook URL with a single command.

With the three environments deployed, write the exact command for each question:

1. Every platform pod in pre or pro, in any namespace.
2. Every platform Deployment that is not development.
3. The pods of components with critical or high criticality that are not in Running state.
4. A table with the environment, component, version and node of every platform pod.
5. The Deployments missing the rutasnorte.example/criticality label (an audit of the scheme).
6. Every pod, unfiltered, showing environment and app.kubernetes.io/component as extra columns.

This exercise shows why labels are a pod's interface with the rest of the system. Work in rutas-norte-dev with the web-store Deployment (3 replicas) and its Service.

1. Note down how many pods there are and how many endpoints the Service has.
2. Pick a pod and change its app label to web-store-isolated with kubectl label --overwrite.
3. Observe immediately: how many pods are there now? What has the ReplicaSet done? Is the modified pod still alive? Who owns it?
4. Check how many endpoints the Service has now and explain why.
5. Explain what this technique is for in the real world and what its danger is.
6. Leave the cluster as it was.

Solution 1

2. The selector carries only app and environment because it is immutable and must identify the component stably. If it included app.kubernetes.io/version, rolling out 1.9.0 would stop the selector matching the new pods, and since it cannot be modified, the Deployment would have to be deleted and recreated on every rollout. Besides, a minimal selector avoids the risk of improper adoptions if another object shares informative labels.

The eight from the scheme, plus the pod-template-hash the Deployment adds.

Solution 2

A note on question 3: -l filters by labels and --field-selector by object fields. They are different, complementary mechanisms; the fields available to --field-selector are limited (status.phase, spec.nodeName, metadata.name, metadata.namespace and little else).

Solution 3

3 pods, 3 endpoints.

There are now 4 pods. What happened, step by step:

• On changing app from web-store to web-store-isolated, the pod stopped matching the selector of the ReplicaSet.
• On the next turn of its reconciliation loop, the ReplicaSet counted 2 pods of its own where there should have been 3 and created a new one (h7pnk, 4 seconds old).
• The modified pod is still alive, but it has become an orphan: no controller is watching it.

Empty output: it has no owner. Kubernetes stripped its ownerReferences when it stopped matching the selector.

There are still 3, but one has changed. The isolated pod (10.244.0.81) has dropped out of the Service because its selector also demands app=web-store; the new pod (10.244.0.87) has taken its place. The orphaned pod no longer receives traffic even though it is perfectly alive and healthy.

5. What it is for in the real world: it is the classic technique of isolation for debugging in production. If one of five pods shows anomalous behaviour —a memory leak, erratic responses, a suspicious log—, changing one of its selector labels pulls it out of the Service and the ReplicaSet in an instant. It stops receiving real customer traffic, the ReplicaSet creates a replacement so no capacity is lost, and you keep the pod intact, with its memory and its state, to inspect at leisure: kubectl exec, kubectl logs, dumps, profiling. There is no need to reproduce the problem in a test environment: you have the patient alive on the table.

   The danger is twofold. First, that pod is no longer managed by anybody: if the node goes down or somebody evicts it, it disappears with no replacement, and above all it is easy to forget about it, consuming CPU and memory indefinitely. Second, if the label change is made by mistake or the other way round —giving a bare pod the labels of a governed component—, you trigger the adoption and possible immediate deletion we saw in the ReplicaSets lesson. The mandatory discipline is to label the isolated pod with the date and the reason, and delete it when you are done:

Three pods, three endpoints, all in order. And notice that deleting the orphan produced no replacement: the ReplicaSet already had its three pods and that one was not its own.

You close module 2 with the thread that ran through it all turned into a system. You can tell a label from an annotation without hesitation: the former identify and are queried, they are indexed, they have 63 characters and a restricted character set; the latter attach information that is read but not searched, they take up to 256 KiB and any content, including JSON and multi-line text. You know the full syntax of keys and values, you know kubernetes.io/ is a reserved prefix and that your organisation's domain is the right place for your own, and you have a command of the six labels Kubernetes recommends, including the distinction between name (what it is) and instance (which one it is).

You have settled Rutas Norte's definitive labelling scheme for its six components and three environments, with its golden rule: only app and environment go into the selectors, because a Deployment's selector is immutable and putting the version in it would condemn you to recreating the object on every rollout. You handle both families of selectors —equality with =, == and !=; sets with in, notin, exists and its negation— both on the command line with -l and in manifests with matchLabels and matchExpressions, without forgetting the Services' asymmetry, which only accepts a flat map. You know who consumes selectors, which is almost everybody: ReplicaSets, Deployments, Services, NetworkPolicies, PodDisruptionBudgets, affinity and the monitoring tooling. You use annotations for what they are good for —documenting the change, configuring Ingress controllers, keeping the runbook and the on-call number to hand— without ever making the mistake of storing a secret in them. And you have kubectl label and kubectl annotate on your belt with --overwrite and deletion with a trailing hyphen, plus the repertoire of queries that turns a large cluster into something navigable.

With this you close module 2 and the Rutas Norte platform is alive. web-store went from being that fragile pod in module 1 to a three-replica Deployment that recovers on its own; bookings-api, redis-cache and notifications-worker have theirs; the four components that receive connections have a stable address with balancing across replicas; you know how to deploy without downtime and to undo a broken rollout in under a minute; you have three environments separated into namespaces with the same manifest deployed to several of them; and everything is labelled with a coherent scheme the coming lessons are going to exploit.

But there are very visible outstanding debts. The PostgreSQL password is still written in plain text inside a manifest that lives in Git, and that database holds the name, ID number, phone and email of every Rutas Norte customer. The API URL is baked into the image instead of being injected per environment. No namespace has a quota, so a mistake in development can starve production of resources. And all our pods declare requests and limits by eye, with no criteria. Module 3, Configuration and Secret Management, tackles exactly that: we will separate configuration from code with ConfigMaps, we will get the credentials out of the manifests with Secrets, we will inject both as environment variables, we will put quotas and limits on each environment, we will understand the quality of service classes that decide which pod dies first when memory runs short, and we will give each component its own identity before the API with ServiceAccounts.