The previous lesson ended with a RutaProgramada CRD that was perfectly defined, validated and queryable, and on which absolutely nothing happened. And in lesson 06-01 we left another debt: bookings-postgres is a StatefulSet that gives its replicas an identity and a disk, but that cannot choose a primary, replicate data or fail over when that primary dies.
Both gaps have the same solution, and it comes down to one sentence worth memorising:
Operator = custom resource + controller that reconciles it.
An operator is an expert's operational knowledge — how a PostgreSQL replica is promoted, how a consistent backup is taken, how a major version is upgraded without losing data — encoded in software that runs inside the cluster and never sleeps.
This is the last lesson of the module. We will settle the two outstanding debts, look inside the reconciliation loop, replace our hand-rolled StatefulSet with a real PostgreSQL operator, sketch out the RutaProgramada controller, and finish by explaining why most teams should not write operators.
Contents
- What an operator is, precisely
- The real anatomy of the reconciliation loop
- Idempotency and no state in memory
- The five-level capability model
- Operators you have already used in this course
- A practical case: a PostgreSQL operator for
bookings-postgres - Where to find operators and what to check before adopting one
- How one is written: Kubebuilder and
controller-runtime ownerReferencesand cascading deletion- When NOT to write an operator
- What an operator is, precisely
An operator has exactly two pieces:
- One or more CRDs that define the vocabulary:
Cluster,Certificate,RutaProgramada. It is the declarative interface with which the user expresses what they want. - A controller — normally a Deployment running in the cluster itself — that watches those objects and does the work.
graph LR
U[User] -->|kubectl apply<br/>Cluster with 3 replicas| API[kube-apiserver]
API -->|watch| C[Operator controller<br/>Deployment in the cluster]
C -->|compares desired vs observed| R{Do they match?}
R -->|no| A[Act: create pods,<br/>promote a primary,<br/>adjust Services]
R -->|yes| N[Do nothing]
A -->|writes| API
C -->|updates status| API
API -->|change| C
The term was coined by CoreOS in 2016 with this idea: when an operations team has been administering PostgreSQL for years, it has accumulated a set of procedures — what to do if the primary stops answering, how to add a read replica, in what order to upgrade — that live in runbooks, in scattered scripts and in two people's heads. An operator turns all of that into a program that runs continuously.
The difference from the tools you already know:
| Tool | When it acts | What it maintains |
|---|---|---|
| Deployment script | When somebody runs it | Nothing: it is a one-shot |
| Helm (10-03) | On install and upgrade |
It renders templates; it does not watch afterwards |
| Kustomize (10-04) | When generating the manifests | Nothing at runtime |
| Operator | Continuously | The desired state, no matter what |
An operator does not install: it maintains. If somebody deletes a pod, it recreates it. If the primary goes down at three in the morning, it promotes a replica without waking anybody. If the disk fills up, it expands it. That permanent vigilance is its value.
- The real anatomy of the reconciliation loop
In lesson 01-02 we described the reconciliation loop as the central idea of Kubernetes: observe the desired state, observe the real one, act to bring them together. Now we look inside it, as any serious controller implements it.
graph TB
W[Informer: WATCH on the API<br/>+ local cache of the state] -->|add/update/delete event| Q[Work queue<br/>with deduplication and delay]
Q -->|takes one key<br/>namespace/name| REC[Reconcile ns/name]
REC --> LEER[Read the current object from the cache]
LEER --> OBS[Observe the real world:<br/>pods, PVCs, services]
OBS --> COMP{desired == observed?}
COMP -->|yes| ST[Update status<br/>and finish]
COMP -->|no| ACT[Carry out the next step]
ACT --> ST
ST --> RES{Result?}
RES -->|error| REQ[Requeue with<br/>exponential backoff]
RES -->|requeue after N s| REQ2[Requeue with a fixed delay]
RES -->|ok| FIN[Wait for the next event]
REQ --> Q
REQ2 --> Q
The watch and the informer
The controller does not poll the API in a loop: it opens a watch connection and receives a notification of every change. The standard library (client-go) wraps that in an informer, which also maintains a local cache of every object watched.
The cache matters for two reasons:
- The controller's reads do not hit the apiserver: on a cluster with hundreds of objects, the difference is substantial.
- The cache may be slightly out of date. A controller has to tolerate reading an object a couple of seconds behind, which reinforces the need for idempotency.
The work queue
Events are not processed directly: each is translated into a key (namespace/name) and put into a queue with three properties:
- Deduplication: if an object changes five times while it is being processed, the key appears only once. It will be reconciled once with the final state, not five times with intermediate ones.
- Delay: you can ask for "look at this again in 30 seconds", useful for waiting until something external makes progress.
- Rate limiting with exponential backoff: a failing object is retried after 5 ms, 10 ms, 20 ms… up to a maximum. A permanently broken object does not consume the whole controller.
The Reconcile function
It is the heart of it, and its signature says a lot:
func (r *RutaProgramadaReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)It receives only a key, not the object and not the event. That design decision is deliberate: Reconcile does not know what changed or why it was called. Its contract is always the same: given this name, read the desired state, look at the real one, and make them match.
It returns two things:
| Returns | Effect |
|---|---|
error != nil |
Requeue with exponential backoff |
Result{Requeue: true} |
Requeue immediately |
Result{RequeueAfter: 30*time.Second} |
Requeue in 30 seconds |
Result{}, nil |
Done; wait for the next event |
Updating the status
The last step of every reconciliation is writing what was observed into .status, through the subresource we studied in 06-06. Here a conventional field appears that deserves an explanation:
metadata.generation is incremented by the apiserver every time the spec changes (not when labels or annotations change). The controller stores in status.observedGeneration the generation it has already processed. By comparing the two, anybody can tell whether the controller is up to date:
kubectl get rutaprogramada rn-041-bilbao-santander -n rutas-norte-pro \
-o jsonpath='generation={.metadata.generation} observed={.status.observedGeneration}{"\n"}'If they differ, there is a spec change the controller has not attended to yet.
- Idempotency and no state in memory
Two non-negotiable properties of any Reconcile.
Idempotency
Reconcile can run many more times than you expect: because of a real change, because of a retry after an error, because of a periodic resync of the informer (every 10 hours by default), or simply because the controller restarted and is reconciling everything that exists.
That is why Reconcile must never think in terms of "create" but of "make sure it exists":
// BAD: fails on the second run with AlreadyExists
if err := r.Create(ctx, deployment); err != nil {
return ctrl.Result{}, err
}
// GOOD: idempotent
existing := &appsv1.Deployment{}
err := r.Get(ctx, client.ObjectKeyFromObject(desired), existing)
switch {
case apierrors.IsNotFound(err):
return ctrl.Result{}, r.Create(ctx, desired)
case err != nil:
return ctrl.Result{}, err
default:
if !reflect.DeepEqual(existing.Spec, desired.Spec) {
existing.Spec = desired.Spec
return ctrl.Result{}, r.Update(ctx, existing)
}
return ctrl.Result{}, nil // it was already fine: do nothing
}A practical corollary: a reconciliation that changes nothing is the normal case and must be cheap. If your Reconcile writes to the API on every pass even when nothing has changed, you will cause an infinite loop: the write generates an event, the event triggers another reconciliation, and so on indefinitely. It is the classic mistake in everybody's first operator, and it is spotted because the apiserver records thousands of updates per minute on the same object.
No state in memory
The controller cannot remember anything between reconciliations. Not in global variables, not in maps, not in local files.
The reasons:
- It can restart at any moment (an update, an eviction, a node failure) and would lose everything.
- There may be several replicas for high availability; only one is active thanks to leader election, but the handover can happen at any time.
- In-memory state drifts out of sync with the real world without anybody noticing.
All the state the controller needs to remember must live in the Kubernetes API: in .status, in annotations, in labels or in the objects it manages. If your operator needs to know "I already ran today's backup", that goes in status.lastBackup, not in a variable.
The positive consequence is that a well-written operator can be killed and started at any moment with no side effects. It is a property worth testing deliberately: delete the controller's pod halfway through an operation and check that it picks up correctly.
- The five-level capability model
The Operator Framework project defined a scale for measuring how much an operator knows how to do. It is the best tool for evaluating one before adopting it. Applied to a database like bookings-postgres:
| Level | Name | What it means for a database |
|---|---|---|
| 1 | Basic install | It creates the StatefulSet, the Service, the PVC and the Secret. Equivalent to what we did by hand in 06-01 |
| 2 | Seamless upgrades | It changes the minor version (16.4 → 16.6) in the right order, replicas before the primary |
| 3 | Full lifecycle | Read replicas, scheduled backups, restore, scaling, configuration changes with no downtime |
| 4 | Deep insights | It exposes metrics and alerts, and the status reflects the real replication state with its lag |
| 5 | Auto pilot | It detects the downed primary and fails over on its own, tunes parameters to the load, repairs corrupt replicas, scales itself |
What each jump adds, in detail:
Level 1 → 2: upgrading stops being a manual procedure. The operator knows the replicas have to be updated first and the promotion done afterwards, and that each one has to be waited for while it syncs.
Level 2 → 3: here is the bulk of the value. Scheduled backups with verification, point-in-time restore, adding a read replica with a single change in the spec, changing shared_buffers without dropping connections.
Level 3 → 4: the operator stops being a black box. It publishes metrics for replication lag, database size and backup state, and its status tells the truth about what is going on.
Level 4 → 5: automatic failover. It is the level that separates "it saves me work" from "I can sleep at night". It is also the hardest and the one requiring the most care: an operator that fails over badly can cause a split brain with two primaries accepting writes.
When evaluating an operator, ask about the level. Many flashy projects stop at 2, and for level 2 the added complexity is not worth it: a Helm chart already does that.
- Operators you have already used in this course
Without calling them that, we have been using operators for several modules.
cert-manager (lesson 04-05)
When we applied a Certificate for www.rutasnorte.example, the following happened without us doing anything else:
- The cert-manager controller observed the new
Certificateobject. - It created a
CertificateRequest, generated a private key and stored it in a Secret. - It created an
Orderand aChallengefor the ACME protocol. - It published a temporary Ingress with the HTTP-01 challenge token.
- It waited for Let's Encrypt to validate, obtained the certificate and wrote it into the Secret.
- And ever since it has been watching the expiry date and repeats the process 30 days before it expires.
Step 6 is the definition of an operator. A script would have done steps 1 to 5; only a controller that runs forever does step 6. In capability terms, cert-manager is at level 5 for its domain: it renews with no human intervention.
NAME READY STATUS RESTARTS AGE
cert-manager-6d8f7c9b54-p2m4x 1/1 Running 0 24d
cert-manager-cainjector-7b9d4f8c6-k7t2v 1/1 Running 0 24d
cert-manager-webhook-59c8d7b64-w3n8q 1/1 Running 0 24dThree Deployments: the controller, a CA certificate injector and the validating webhook for its CRDs. It is the typical anatomy of a mature operator.
The snapshot-controller (lesson 05-05)
When we created a VolumeSnapshot of the bookings data, a controller observed it, talked to the CSI driver, created the corresponding VolumeSnapshotContent and updated status.readyToUse. Same pattern: a CRD plus a controller.
Velero (lesson 05-06)
Its Backup, Restore and Schedule are CRDs, and its controller is what runs the backups, applies the hooks before and after, and honours the retention. A Velero Schedule is an operator creating Jobs, conceptually the same as what the CronJob controller does with our occupancy-reports.
Prometheus Operator (lesson 07-03)
Coming up. Its Prometheus, ServiceMonitor, PodMonitor and PrometheusRule CRDs let you declare "collect metrics from every Service with this label" and the operator generates and reloads the Prometheus configuration. Without it, adding a new target means hand-editing a configuration file hundreds of lines long.
And the native controllers
It is worth closing the circle: the Deployment controller, the ReplicaSet one and the Job one work in exactly the same way. They watch objects, compare desired with observed and act. The only difference is that their types come built in and their code lives inside the kube-controller-manager rather than in a separate Deployment. The pattern is identical; operators simply extend it to domains Kubernetes does not know about.
- A practical case: a PostgreSQL operator for
bookings-postgres
bookings-postgresWe come to the debt from 06-01. Our hand-rolled StatefulSet has one replica and cannot do anything else. We are going to replace it with CloudNativePG, a mature open-source PostgreSQL operator.
Installing the operator
kubectl apply --server-side -f \
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.1.yaml
kubectl get deployment -n cnpg-system
kubectl get crds | grep postgresqlNAME READY UP-TO-DATE AVAILABLE AGE
cnpg-controller-manager 1/1 1 1 47s
backups.postgresql.cnpg.io 2026-08-05T21:40:11Z
clusters.postgresql.cnpg.io 2026-08-05T21:40:11Z
poolers.postgresql.cnpg.io 2026-08-05T21:40:11Z
scheduledbackups.postgresql.cnpg.io 2026-08-05T21:40:12ZOne Deployment (the controller) and four CRDs (the vocabulary). Exactly the two pieces of the definition.
Declaring the cluster
bookings-postgres is now described like this:
# k8s/base/bookings-postgres-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: bookings-postgres
namespace: rutas-norte-pro
labels:
app: bookings-postgres
app.kubernetes.io/part-of: rutas-norte
environment: pro
spec:
instances: 3 # one primary and two replicas
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
primaryUpdateStrategy: unsupervised # the operator fails over on its own when updating
bootstrap:
initdb:
database: bookings
owner: rutasnorte
secret:
name: bookings-postgres-credentials
localeCollate: es_ES.UTF-8
localeCType: es_ES.UTF-8
storage:
size: 20Gi
storageClass: rutasnorte-fast
walStorage: # WAL on a separate volume: better performance
size: 5Gi
storageClass: rutasnorte-fast
postgresql:
parameters:
max_connections: "200"
shared_buffers: "512MB"
work_mem: "8MB"
log_min_duration_statement: "500" # log queries taking more than 500 ms
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "1"
memory: 2Gi # requests == limits: QoS Guaranteed (03-05)
affinity:
enablePodAntiAffinity: true
topologyKey: kubernetes.io/hostname
podAntiAffinityType: required # never two instances on the same node (06-05)
nodeSelector:
disk: ssd
monitoring:
enablePodMonitor: true # metrics for Prometheus (07-03)
backup:
retentionPolicy: "30d"
barmanObjectStore:
destinationPath: "s3://rutasnorte-backups/bookings-postgres"
s3Credentials:
accessKeyId:
name: backup-credentials
key: ACCESS_KEY_ID
secretAccessKey:
name: backup-credentials
key: SECRET_ACCESS_KEY
wal:
compression: gzip
maxParallel: 4
data:
compression: gzip
immediateCheckpoint: false
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: bookings-postgres-nightly-backup
namespace: rutas-norte-pro
labels:
app: bookings-postgres
app.kubernetes.io/part-of: rutas-norte
environment: pro
spec:
schedule: "0 30 2 * * *" # 02:30 (6-field format, with seconds)
backupOwnerReference: self
cluster:
name: bookings-postgresNAME AGE INSTANCES READY STATUS PRIMARY
bookings-postgres 3m42s 3 3 Cluster in healthy state bookings-postgres-1NAME READY STATUS RESTARTS AGE
pod/bookings-postgres-1 1/1 Running 0 3m
pod/bookings-postgres-2 1/1 Running 0 2m
pod/bookings-postgres-3 1/1 Running 0 2m
NAME TYPE CLUSTER-IP PORT(S)
service/bookings-postgres-rw ClusterIP 10.96.201.14 5432/TCP
service/bookings-postgres-ro ClusterIP 10.96.188.77 5432/TCP
service/bookings-postgres-r ClusterIP 10.96.140.22 5432/TCPThe three Services are the piece a StatefulSet cannot provide:
| Service | Points to | Use in Rutas Norte |
|---|---|---|
-rw |
Only the current primary | Writes from bookings-api |
-ro |
Only the replicas | Queries from occupancy-reports |
-r |
Any instance | Reads that tolerate lag |
And the essential part: when the primary changes, the operator rewrites the -rw Service so that it points to the new one. The application never notices. That is exactly what a StatefulSet cannot do, because its headless Service gives stable names but does not know which of those names is the primary.
The acid test: killing the primary
kubectl delete pod bookings-postgres-1 -n rutas-norte-pro
kubectl get cluster bookings-postgres -n rutas-norte-pro -wNAME INSTANCES READY STATUS PRIMARY
bookings-postgres 3 2 Failing over to bookings-postgres-2 bookings-postgres-1
bookings-postgres 3 2 Cluster in healthy state bookings-postgres-2
bookings-postgres 3 3 Cluster in healthy state bookings-postgres-2Within seconds: the outage is detected, bookings-postgres-2 is promoted, the -rw Service is rewritten, and the downed instance comes back as a replica. No human intervention, no runbook, no phone call at three in the morning.
What the operator solves that you would otherwise do by hand
| Capability | With a hand-rolled StatefulSet (06-01) | With an operator |
|---|---|---|
| Primary election | Decide by convention that it is ordinal 0 | The operator decides it, records it in status and publishes it |
| Failover | Detect the outage, promote, reconfigure replicas, change the Service: all manual | Automatic, in seconds |
| Read replicas | pg_basebackup by hand, primary_conninfo, replication slots |
instances: 3 |
| Read/write routing | Nothing: a single Service for everything | -rw, -ro and -r Services kept up to date |
| Scheduled backups | A CronJob with pg_dump (05-06) |
ScheduledBackup with continuous WAL |
| Point-in-time recovery | Impossible without hand-built WAL archiving | recoveryTarget.targetTime |
| Minor version upgrade | Edit the image and hope | Replicas first, failover, primary afterwards |
| Major version upgrade | A dump, a restore and hours of downtime | A procedure guided by the operator |
| Expanding the disk | kubectl patch on each PVC (05-05) |
Change storage.size |
| Replication slots | Manual configuration, and they break when pods are recreated | Managed |
| Real health probes | pg_isready, which does not tell a primary from a replica |
The operator knows each instance's role |
| Metrics | Add an exporter sidecar (06-04) | enablePodMonitor: true |
That is the whole argument in favour of operators for stateful software: the left-hand column is weeks of work, fragile procedures and night shifts; the right-hand one is fields in a YAML.
Migrating without losing the data
With what we learned in 05-06 and 06-01, the migration from our StatefulSet is done by logical restore, not by moving volumes:
spec:
bootstrap:
initdb:
database: bookings
owner: rutasnorte
import:
type: microservice
databases: ["bookings"]
source:
externalCluster: old-statefulset
externalClusters:
- name: old-statefulset
connectionParameters:
host: bookings-postgres-nodes.rutas-norte-pro.svc.cluster.local
user: rutasnorte
dbname: bookings
password:
name: bookings-postgres-credentials
key: passwordThe operator starts up, connects to the old StatefulSet, imports the database and builds the new cluster. Afterwards you verify the number of bookings, point bookings-api at the -rw Service and retire the StatefulSet.
- Where to find operators and what to check before adopting one
Where to look
| Source | What it holds |
|---|---|
| OperatorHub.io | A community catalogue with a declared capability level |
| Artifact Hub | Helm charts and operators, with download counts |
| The project's official repository | Almost always the most reliable and up-to-date source |
| Your cloud provider's marketplace | Operators validated for EKS, AKS or GKE (10-06) |
The checklist
Adopting an operator means adopting a dependency that will have broad permissions over your cluster and that your data will depend on. Before installing it:
1. Maintenance. When was the last commit? How many people contribute? Are there regular releases? How many open issues with no response? An abandoned operator managing your database is a serious problem, because uninstalling it without losing data is rarely trivial.
2. The RBAC permissions it asks for. This is the point most often overlooked. Look at it before applying the manifest:
The questions: does it ask for cluster-admin? (an immediate red flag) Does it ask for access to all the cluster's Secrets? Can it create ClusterRoleBindings, that is, widen its own permissions? A compromised operator with broad permissions amounts to a compromised cluster. We will come back to this in 08-01.
3. Maturity. What capability level does it declare and which one does it really meet? Are there documented production use cases? Is there an upgrade guide between versions of the operator itself?
4. What happens if you uninstall it. The decisive question:
- Do the objects it managed keep working, or do they stop?
- Do its CRDs have finalizers that would leave objects hanging in
Terminating? - Can you export the data to a standard format?
- Is there a documented exit procedure?
An operator you cannot get out of is a technological hostage situation. With CloudNativePG, for example, the backups in Barman format are restorable with standard PostgreSQL tools: there is a way out.
5. Resources and scope. How much CPU and memory does the controller use? Does it watch the whole cluster or can it be limited to particular namespaces? An operator watching every object in a large cluster can use a fair amount of memory.
6. Upgrade model. How is the operator upgraded without affecting what it manages? Is it backwards compatible with the CRDs already deployed?
- How one is written: Kubebuilder and
controller-runtime
controller-runtimeNobody writes an operator from scratch. The standard tools:
| Tool | What it provides |
|---|---|
controller-runtime |
A Go library with informers, queues, a cached client and a controller manager |
| Kubebuilder | Scaffolding: it generates the project, the CRDs from Go structs, the RBAC and the deployment |
| Operator SDK | It wraps Kubebuilder and adds Helm and Ansible as alternatives to Go |
With Operator SDK you can build an operator without writing Go, using a Helm chart or an Ansible playbook as the reconciliation logic. It is a reasonable route for simple operators, though it limits the capabilities to levels 1 and 2.
Starting a project
mkdir -p ~/projects/rutasnorte-operator && cd ~/projects/rutasnorte-operator
kubebuilder init \
--domain rutasnorte.example \
--repo github.com/rutasnorte/rutasnorte-operator
kubebuilder create api \
--group rutasnorte \
--version v1 \
--kind RutaProgramada \
--resource --controllerThe generated structure:
api/v1/rutaprogramada_types.go <- the Go types: the CRD comes from here
internal/controller/rutaprogramada_controller.go <- Reconcile goes here
config/crd/bases/ <- CRDs generated with "make manifests"
config/rbac/ <- Roles generated from the markers
config/samples/ <- sample instances
Makefile <- make manifests, make docker-build, make deployThe types, from which the CRD comes
The CRD from lesson 06-06 is not written by hand in a real project: it is generated from annotated Go structs.
// api/v1/rutaprogramada_types.go
type RutaProgramadaSpec struct {
// Commercial code of the route, format XX-999 (e.g. RN-041)
// +kubebuilder:validation:Pattern=`^[A-Z]{2}-[0-9]{3}$`
Code string `json:"code"`
// Town the journey starts from
// +kubebuilder:validation:MinLength=2
// +kubebuilder:validation:MaxLength=60
Origin string `json:"origin"`
// Town the journey ends at
// +kubebuilder:validation:MinLength=2
// +kubebuilder:validation:MaxLength=60
Destination string `json:"destination"`
// Total seats offered on each departure
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=90
Seats int32 `json:"seats"`
// Daily departure times in HH:MM format
// +kubebuilder:validation:MinItems=1
// +kubebuilder:validation:items:Pattern=`^([01][0-9]|2[0-3]):[0-5][0-9]$`
Timetables []string `json:"timetables"`
// Whether the route currently accepts sales
// +kubebuilder:default=true
// +optional
Active bool `json:"active,omitempty"`
}
type RutaProgramadaStatus struct {
// +optional
Phase string `json:"phase,omitempty"`
// +optional
SeatsSold int32 `json:"seatsSold,omitempty"`
// Generation of the spec the controller last processed
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=route;routes,categories=rutasnorte
// +kubebuilder:printcolumn:name="Code",type=string,JSONPath=`.spec.code`
// +kubebuilder:printcolumn:name="Origin",type=string,JSONPath=`.spec.origin`
// +kubebuilder:printcolumn:name="Destination",type=string,JSONPath=`.spec.destination`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
type RutaProgramada struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec RutaProgramadaSpec `json:"spec,omitempty"`
Status RutaProgramadaStatus `json:"status,omitempty"`
}The // +kubebuilder:... comments are markers the generator translates into the CRD's OpenAPI schema. make manifests produces exactly the YAML we wrote by hand in 06-06, and the ordinary comments become the descriptions that feed kubectl explain.
The skeleton of Reconcile
This is the heart of the RutaProgramada operator: every active route must have its own CronJob generating the occupancy report for that route.
// internal/controller/rutaprogramada_controller.go
// Permissions the controller needs. These markers generate config/rbac/role.yaml
// +kubebuilder:rbac:groups=rutasnorte.rutasnorte.example,resources=rutasprogramadas,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=rutasnorte.rutasnorte.example,resources=rutasprogramadas/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=rutasnorte.rutasnorte.example,resources=rutasprogramadas/finalizers,verbs=update
// +kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch
func (r *RutaProgramadaReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := logf.FromContext(ctx)
// ── 1. READ THE DESIRED STATE ────────────────────────────────────────
var route rutasnortev1.RutaProgramada
if err := r.Get(ctx, req.NamespacedName, &route); err != nil {
// NotFound means the route was deleted. There is nothing to do:
// the derived objects delete themselves through ownerReferences (section 9).
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// ── 2. INACTIVE ROUTE: withdraw whatever was created ─────────────────
if !route.Spec.Active {
log.Info("Route inactive; the reports CronJob is not maintained", "code", route.Spec.Code)
return ctrl.Result{}, r.updateStatus(ctx, &route, "Cancelled")
}
// ── 3. BUILD THE DESIRED STATE ───────────────────────────────────────
// A pure function: the same input data ALWAYS gives the same output object.
// That purity is what makes the comparison in step 5 possible.
desired := r.buildReportCronJob(&route)
// ── 4. ownerReferences: the CronJob belongs to the route ─────────────
// When the route is deleted, the garbage collector deletes the CronJob (section 9).
if err := ctrl.SetControllerReference(&route, desired, r.Scheme); err != nil {
return ctrl.Result{}, err
}
// ── 5. RECONCILE: create if missing, update if different, nothing if equal ──
var current batchv1.CronJob
err := r.Get(ctx, client.ObjectKeyFromObject(desired), ¤t)
switch {
case apierrors.IsNotFound(err):
log.Info("Creating the reports CronJob", "route", route.Spec.Code)
if err := r.Create(ctx, desired); err != nil {
// Returning the error makes the queue requeue with exponential backoff
return ctrl.Result{}, err
}
r.Recorder.Eventf(&route, corev1.EventTypeNormal, "CronJobCreated",
"Created the reports CronJob for route %s", route.Spec.Code)
case err != nil:
return ctrl.Result{}, err
default:
// IDEMPOTENCY: if nothing changed, do NOT write. Writing here would cause
// an event, which would cause another reconciliation: an infinite loop.
if !equality.Semantic.DeepDerivative(desired.Spec, current.Spec) {
log.Info("Updating the reports CronJob", "route", route.Spec.Code)
current.Spec = desired.Spec
if err := r.Update(ctx, ¤t); err != nil {
return ctrl.Result{}, err
}
}
}
// ── 6. OBSERVE THE REAL WORLD AND WRITE THE STATUS ───────────────────
sold, err := r.querySeatsSold(ctx, &route)
if err != nil {
// A transient failure querying the database: retry in a minute
// without marking the reconciliation as failed.
log.Error(err, "could not query the seats sold")
return ctrl.Result{RequeueAfter: time.Minute}, nil
}
route.Status.Phase = "Active"
route.Status.SeatsSold = sold
route.Status.ObservedGeneration = route.Generation // "already processed" marker
meta.SetStatusCondition(&route.Status.Conditions, metav1.Condition{
Type: "ReportsScheduled",
Status: metav1.ConditionTrue,
Reason: "CronJobActive",
Message: fmt.Sprintf("Reports for route %s scheduled", route.Spec.Code),
})
// Written through the status SUBRESOURCE: it does not touch the user's spec (06-06)
if err := r.Status().Update(ctx, &route); err != nil {
return ctrl.Result{}, err
}
// ── 7. PERIODIC RECONCILIATION ───────────────────────────────────────
// Even with no events, check every 10 minutes: it catches external changes
// and drift that generated no notification.
return ctrl.Result{RequeueAfter: 10 * time.Minute}, nil
}
func (r *RutaProgramadaReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&rutasnortev1.RutaProgramada{}).
// Watch our own CronJobs too: if somebody deletes one by hand,
// a reconciliation of its route is triggered and it is recreated.
Owns(&batchv1.CronJob{}).
Complete(r)
}The seven steps are the skeleton of any operator, whether for bus routes or for PostgreSQL: read the desired state, handle the deletion case, build what ought to exist, mark the ownership, create or update only if needed, observe reality and write the status, and decide when to look again.
The Owns(&batchv1.CronJob{}) in SetupWithManager deserves a comment: it makes the controller receive events for the CronJobs it created, and translate them into reconciliations of the owning RutaProgramada. It is what makes deleting the CronJob by hand cause its recreation within seconds. Without that line, the operator would only react to changes in its own custom resources.
The RBAC permissions
The +kubebuilder:rbac: markers above generate this ClusterRole:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rutasnorte-operator-manager-role
rules:
- apiGroups: ["rutasnorte.rutasnorte.example"]
resources: ["rutasprogramadas"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["rutasnorte.rutasnorte.example"]
resources: ["rutasprogramadas/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["batch"]
resources: ["cronjobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]Note the principle of least privilege: the operator can only touch CronJobs and its own resources. It cannot read Secrets, create Deployments or touch anything else. When you evaluate somebody else's operator (section 7), this is exactly what you should look at and compare against what the operator says it does. RBAC in detail is lesson 08-01.
Running and deploying
# Generate the CRDs and RBAC from the markers
make manifests generate
# Install the CRDs in the cluster
make install
# Run the controller LOCALLY against the cluster: the development cycle
make runINFO setup starting manager
INFO Starting EventSource {"controller": "rutaprogramada", "source": "kind source: *v1.RutaProgramada"}
INFO Starting Controller {"controller": "rutaprogramada"}
INFO Creating the reports CronJob {"route": "RN-041"}# And for production: an image and a deployment inside the cluster
make docker-build docker-push IMG=registry.rutasnorte.example/rutasnorte-operator:0.1.0
make deploy IMG=registry.rutasnorte.example/rutasnorte-operator:0.1.0make run is the practical advantage of this model: the controller runs on your laptop, with a debugger if you need one, talking to the real cluster. There is no need to build images for every iteration.
ownerReferences and cascading deletion
ownerReferences and cascading deletionIn lesson 02-02 we saw that a ReplicaSet puts ownerReferences on its pods, and that this is why deleting a Deployment deletes everything underneath. Operators use the same mechanism, and it is essential to understand it.
if err := ctrl.SetControllerReference(&route, desired, r.Scheme); err != nil {
return ctrl.Result{}, err
}That line writes into the generated CronJob:
ownerReferences:
- apiVersion: rutasnorte.rutasnorte.example/v1
kind: RutaProgramada
name: rn-041-bilbao-santander
uid: 3f1a9c04-8e2b-4c71-9a55-71b0e2d8c4f3
controller: true
blockOwnerDeletion: trueThe consequences:
- Automatic cascading deletion. When the
RutaProgramadais deleted, Kubernetes' garbage collector deletes the CronJob. The operator does not have to write any clean-up code: the cluster itself takes care of it. - Ownership is visible.
kubectl describe cronjobshowsControlled By: RutaProgramada/rn-041-bilbao-santander, which makes it traceable where each object came from. - Events towards the owner. Thanks to
Owns(), changes to the CronJob trigger reconciliations of the route.
Two important restrictions:
- The owner and the owned object must be in the same namespace. A namespaced object cannot be owned by one from a different namespace.
- A cluster-scoped object cannot be owned by a namespaced one. If your operator creates ClusterRoles or PersistentVolumes, you will have to clean them up yourself.
Finalizers: clean-up outside the cluster
When the operator creates external resources — a bucket, a DNS record, a database at a provider — ownerReferences are of no use: Kubernetes' garbage collector knows nothing about those resources.
That is what finalizers are for, the same mechanism we saw protecting PVCs in 05-03:
const finalizer = "rutasnorte.example/cleanup-external-resources"
if !route.DeletionTimestamp.IsZero() {
// The object is marked for deletion but has NOT been deleted:
// the finalizer holds it until we remove it.
if controllerutil.ContainsFinalizer(&route, finalizer) {
if err := r.deleteExternalResources(ctx, &route); err != nil {
return ctrl.Result{}, err // it will retry; the object stays held
}
controllerutil.RemoveFinalizer(&route, finalizer)
if err := r.Update(ctx, &route); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil // now Kubernetes completes the deletion
}
// A live object: make sure it has the finalizer
if !controllerutil.ContainsFinalizer(&route, finalizer) {
controllerutil.AddFinalizer(&route, finalizer)
if err := r.Update(ctx, &route); err != nil {
return ctrl.Result{}, err
}
}A first-order operational warning: if the operator stops working and there are objects carrying its finalizer, those objects stay in Terminating forever. kubectl delete hangs and there is no clean way out other than editing the object and removing the finalizer by hand:
kubectl patch rutaprogramada rn-041-bilbao-santander -n rutas-norte-pro \
--type=merge -p '{"metadata":{"finalizers":null}}'That leaves the external resources orphaned, and they will have to be cleaned up some other way. It is one of the reasons behind the "what happens if I uninstall it?" question of section 7: always uninstall the operator after deleting its objects, never before.
- When NOT to write an operator
Writing an operator is fun and almost always unnecessary. Before you start, put it through this filter.
Do not write one if your application is stateless
A Deployment, a Service, a HorizontalPodAutoscaler and a ConfigMap cover bookings-api, web-store and notifications-worker completely. An operator would add nothing: the Deployment controller already does the reconciliation.
Operators shine with stateful software and complex operational procedures: databases, message queues, consensus systems, distributed stores. If your application restarts with no consequences, you do not need one.
Do not write one if Helm or Kustomize will do
| Need | Tool |
|---|---|
| Deploying with different values per environment | Helm (10-03) or Kustomize (10-04) |
| Automatically applying what is in Git | GitOps with Argo CD or Flux (10-05) |
| Reacting to failures and running procedures continuously | An operator |
| Running something periodically | A CronJob (06-03) |
| Validating or modifying objects at admission | A webhook, not an operator |
The deciding question: "what has to happen when something breaks at three in the morning?" If the answer is "nothing, the Deployment recreates it", you do not need an operator. If it is "a replica has to be promoted, the routing reconfigured and somebody notified", then there is an operator there.
Do not write one if one already exists
For PostgreSQL, MySQL, Redis, Kafka, MongoDB, Elasticsearch, RabbitMQ and practically any well-known software mature operators already exist, maintained by teams that have been at it for years. Writing your own means reimplementing, worse, what others have already solved, and maintaining it on your own.
The real cost of maintaining an operator
What people underestimate:
- It is a production service with privileged permissions. It needs deployment, updates, monitoring, alerting and on-call.
- A failure in the operator affects everything it manages. A
Reconcilewith a bug can delete objects in cascade across the whole cluster. There are famous public incidents caused by this. - CRDs are a public API with the compatibility obligations we saw in 06-06. Changing the schema afterwards is expensive.
- Testing an operator is hard. You need tests with
envtestor an ephemeral cluster, and the reconciliation logic has many possible paths. - It requires Go and deep Kubernetes knowledge in the team, in perpetuity, not just while it is being written.
A sensible rule of thumb:
Write an operator when you have a documented, repetitive operational procedure that runs often and that goes wrong when a tired human does it. If you cannot write that runbook precisely, you will not be able to encode it either.
Cheaper alternatives
| Instead of an operator | Try this first |
|---|---|
| Automating a deployment | Helm + GitOps (10-03, 10-05) |
| Running something periodically | A CronJob (06-03) |
| Reacting to a one-off event | A Job triggered from the CI pipeline |
| Adding configuration to pods | A mutating webhook or an init container (06-04) |
| Managing a database | An existing operator or a managed service (10-06) |
| Validating manifests | Policies with Kyverno or schemas in CI |
Common Mistakes and Tips
The infinite reconciliation loop. Mistake number one: Reconcile writes to the API on every pass even when nothing has changed, the write generates an event, the event triggers another reconciliation. Symptom: thousands of updates per minute on the same object. Fix: compare before writing, with DeepDerivative or similar.
Keeping state in memory. Global variables, "already done it" maps. They are lost on restart and drift out of sync with the real world. All the state goes in .status, in annotations or in the managed objects.
Writing .status without the subresource. An Update of the whole object can overwrite a spec change the user has just made. Always use r.Status().Update() with the subresource enabled.
Forgetting ownerReferences. The objects created by the operator are left orphaned when the main resource is deleted. Over time the cluster fills with CronJobs, Services and Secrets nobody knows the origin of.
Uninstalling the operator before deleting its objects. If it uses finalizers, the objects stay in Terminating forever. The right order: delete the objects, check that they disappear, and only then uninstall the operator.
Adopting an operator that asks for cluster-admin. It is an open door to the whole cluster. Always review the ClusterRole before applying the manifest, and be suspicious of anyone asking for access to all Secrets without justifying it.
Confusing "it has CRDs" with "it is an operator". A CRD with no controller is a database with a form, as we saw in 06-06. Check that there is a Deployment running and that it writes to the objects' status.
Not reading the capability level. A level-2 operator is not going to save you from a night-time failover. If you adopt it believing it reaches level 5, the surprise will arrive at the worst moment.
Tip: use make run in development. The controller runs on your laptop against the real cluster, with a debugger. The iteration cycle goes from minutes to seconds.
Tip: emit events. r.Recorder.Eventf(...) makes the operator's actions show up in the object's kubectl describe. It is the difference between an operator you can diagnose and one that is a black box.
Tip: record observedGeneration. It lets anybody tell whether the controller has processed the latest spec change, and it is the basis for kubectl wait working reliably on your resources.
Tip: test by killing the controller. Delete it halfway through an operation and check that it picks up correctly when it comes back. If it does not, you have state in memory or you are missing idempotency.
Exercises
Exercise 1: identify the operators in your cluster
On your minikube (profile rutas-norte), identify which operators are installed. For each one, work out: which CRDs it provides, where its controller runs, and what ClusterRole permissions it has. Then reason about why cert-manager is an operator while the Deployment controller, being the same pattern, is not called one.
Exercise 2: the difference between reconciling and deploying
Show experimentally that a controller reconciles continuously and a deployment does not:
- Create a Deployment
reconciliation-demowith 3 replicas inrutas-norte-dev. - Delete a pod and observe what happens and how quickly.
- Manually change a pod's image with
kubectl edit podand observe the result. - Explain which controller acted in each case and with what information.
Exercise 3: design an operator (without writing it)
Rutas Norte wants an EntornoPruebas resource that, on being created, automatically brings about: a namespace of its own, a restore of bookings-postgres from the latest snapshot, a deployment of bookings-api pointing at that database, and complete deletion after 7 days.
Without writing any code, design:
- The CRD's
specandstatus(fields and types). - The steps of the
Reconcilefunction, in order. - The minimum RBAC permissions it would need.
- What you would use for the deletion after 7 days, and why.
- Whether a finalizer would be needed, and what for.
Solutions
Solution 1
# Which CRDs there are and whose they are
kubectl get crds -o custom-columns=NAME:.metadata.name,GROUP:.spec.group | sort -k2NAME GROUP
certificaterequests.cert-manager.io cert-manager.io
certificates.cert-manager.io cert-manager.io
clusterissuers.cert-manager.io cert-manager.io
issuers.cert-manager.io cert-manager.io
volumesnapshotclasses.snapshot.storage.k8s.io snapshot.storage.k8s.io
volumesnapshotcontents.snapshot.storage.k8s.io snapshot.storage.k8s.io
volumesnapshots.snapshot.storage.k8s.io snapshot.storage.k8s.io# Where the controllers run
kubectl get deployments -A | grep -Ei 'cert-manager|snapshot|controller'cert-manager cert-manager 1/1 1 1 24d
cert-manager cert-manager-cainjector 1/1 1 1 24d
cert-manager cert-manager-webhook 1/1 1 1 24d
kube-system snapshot-controller 1/1 1 1 17d# What permissions cert-manager has
kubectl get clusterrole -l app.kubernetes.io/instance=cert-manager \
-o custom-columns=NAME:.metadata.name --no-headers | head -5
kubectl describe clusterrole cert-manager-controller-certificates | head -20Name: cert-manager-controller-certificates
PolicyRule:
Resources Verbs
--------- -----
certificaterequests.cert-manager.io [create delete get list patch update watch]
certificates.cert-manager.io [get list patch update watch]
certificates.cert-manager.io/status [patch update]
secrets [create delete get list patch update watch]
events [create patch]Narrow permissions: its own CRDs, plus Secrets (necessary: that is where it stores keys and certificates) and events. It does not ask for cluster-admin.
Why cert-manager is called an operator and the Deployment controller is not, given that they are the same pattern:
The difference is not technical, it is one of origin. Both are reconciliation loops over API types. The Deployment controller manages a native type and its code lives inside the kube-controller-manager, a control-plane binary. cert-manager manages types added through CRDs and runs as just another cluster workload.
The term "operator" designates that second situation: the controller pattern applied to a domain Kubernetes does not know about out of the box, packaged as a deployable application. Conceptually, the Deployment controller is a Deployment operator; it is simply that nobody calls it that because it comes included.
Solution 2
# /tmp/reconciliation-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: reconciliation-demo
namespace: rutas-norte-dev
labels:
app: reconciliation-demo
app.kubernetes.io/part-of: rutas-norte
environment: dev
spec:
replicas: 3
selector:
matchLabels:
app: reconciliation-demo
environment: dev
template:
metadata:
labels:
app: reconciliation-demo
app.kubernetes.io/part-of: rutas-norte
environment: dev
spec:
automountServiceAccountToken: false
containers:
- name: nginx
image: nginx:1.27.2-alpine
resources:
requests:
cpu: 20m
memory: 32Mi
limits:
cpu: 100m
memory: 64Mikubectl apply -f /tmp/reconciliation-demo.yaml
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demoNAME READY STATUS RESTARTS AGE
reconciliation-demo-6b4d8f7c9-h2k4x 1/1 Running 0 22s
reconciliation-demo-6b4d8f7c9-p8m2v 1/1 Running 0 22s
reconciliation-demo-6b4d8f7c9-t5n7q 1/1 Running 0 22s2. Delete a pod:
kubectl delete pod reconciliation-demo-6b4d8f7c9-h2k4x -n rutas-norte-dev
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demoNAME READY STATUS RESTARTS AGE
reconciliation-demo-6b4d8f7c9-p8m2v 1/1 Running 0 2m
reconciliation-demo-6b4d8f7c9-t5n7q 1/1 Running 0 2m
reconciliation-demo-6b4d8f7c9-w3x9z 0/1 ContainerCreating 0 1sIn less than a second there is a replacement. Nobody ran any command: the ReplicaSet saw there were 2 pods where there should be 3 and created one.
3. Change a pod's image by hand:
kubectl set image pod/reconciliation-demo-6b4d8f7c9-p8m2v \
-n rutas-norte-dev nginx=nginx:1.26.2-alpine
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demo \
-o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[0].imagePOD IMAGE
reconciliation-demo-6b4d8f7c9-p8m2v nginx:1.26.2-alpine
reconciliation-demo-6b4d8f7c9-t5n7q nginx:1.27.2-alpine
reconciliation-demo-6b4d8f7c9-w3x9z nginx:1.27.2-alpineThe change sticks. This is surprising, but it is correct and very instructive.
4. Which controller acted and with what information:
| Case | Controller | What it compared | Result |
|---|---|---|---|
| Pod deleted | ReplicaSet | Pods matching its selector (2) vs replicas (3) |
It created a pod |
| Image changed by hand | None | — | The change sticks |
The key is what each controller reconciles:
- The ReplicaSet controller reconciles the number of pods matching its selector. It counts 2, it wants 3, it creates one. It does not care what image they run.
- The Deployment controller reconciles which ReplicaSets should exist according to the Deployment's template. Since the template did not change, it does nothing.
- Nobody reconciles the content of an individual pod against the Deployment's template. The template is used when creating the pod, not for watching it afterwards.
A practical lesson: reconciliation is about managed objects, not arbitrary fields. If you delete the modified pod, its replacement will be born from the template and will go back to nginx:1.27.2-alpine. And a kubectl rollout restart deployment/reconciliation-demo would recreate every pod from the template, correcting the drift.
kubectl rollout restart deployment/reconciliation-demo -n rutas-norte-dev
kubectl rollout status deployment/reconciliation-demo -n rutas-norte-dev
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demo \
-o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[0].imagePOD IMAGE
reconciliation-demo-7c9e5a2b4-b1k8m nginx:1.27.2-alpine
reconciliation-demo-7c9e5a2b4-j4t2p nginx:1.27.2-alpine
reconciliation-demo-7c9e5a2b4-r6n9w nginx:1.27.2-alpineSolution 3
1. CRD design:
spec:
requester: string # mandatory, the developer's email
gitBranch: string # mandatory, valid branch pattern
durationDays: integer # 1-14, default 7
dataSource: # where to restore from
type: string # enum: snapshot, veleroBackup, empty
name: string # name of the snapshot or the backup
components: # what to deploy
bookingsApi: boolean # default true
webStore: boolean # default false
databaseSize: string # default 5Gi
status:
phase: string # enum: Pending, Creating, Restoring, Ready, Expiring, Error
namespaceCreated: string
accessUrl: string
expiryDate: string (date-time) # calculated: creationTimestamp + durationDays
observedGeneration: integer
conditions: []Condition # NamespaceCreated, DataRestored, ComponentsReadyThe status subresource enabled. additionalPrinterColumns: Requester, Phase, Expiry, Age.
2. The steps of Reconcile:
- Read the
EntornoPruebas. If it does not exist, finish (IgnoreNotFound). - If it has a
DeletionTimestamp, run the finalizer logic (step 10) and finish. - Make sure it has the finalizer if it does not already.
- Check the expiry: if
now() > status.expiryDate, delete the object itself and finish. The cascade will do the rest. - Make sure the namespace
test-<name>exists with ownership labels. (It is a cluster-scoped object: it accepts noownerReferencesfrom a namespaced object; it is cleaned up in the finalizer.) - Make sure the PVC restored from the snapshot exists, with
dataSource(05-05). If it is notBoundyet, updatestatus.phase = "Restoring"and returnRequeueAfter: 30s. - Make sure the PostgreSQL StatefulSet/Cluster and its generated credentials Secret exist.
- Make sure the Deployments and Services for the components marked in
spec.componentsexist, and the Ingress if appropriate. - Observe reality: are all the pods
Ready? Writestatus.phase,accessUrl, conditions andobservedGenerationthrough the subresource. - Return a calculated
RequeueAfter: up to the expiry if it is far off, or 1 minute if it is close, so that step 4 fires in time.
3. Minimum RBAC:
rules:
- apiGroups: ["rutasnorte.example"]
resources: ["entornospruebas"]
verbs: ["get", "list", "watch", "update", "patch", "delete"]
- apiGroups: ["rutasnorte.example"]
resources: ["entornospruebas/status", "entornospruebas/finalizers"]
verbs: ["get", "update", "patch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["services", "secrets", "persistentvolumeclaims"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]Note what is not there: no cluster-admin, no permissions over nodes, none over RBAC. Secrets are unavoidable (it generates credentials), and that alone is reason enough to review the code before granting it those permissions.
4. The deletion after 7 days:
The right option is the reconciliation loop itself with RequeueAfter, not a clean-up CronJob:
- The operator is already watching the object: checking a date on every reconciliation is free.
RequeueAfterguarantees it wakes up in time even if there is no other event.- An external CronJob would be a second component with its own logic, its own permissions and its own way of failing, and it could be out of step with what the operator believes.
- Besides,
ttlSecondsAfterFinished(06-03) does not apply: that is for Jobs, not for custom resources.
An implementation detail: the operator deletes its own object (r.Delete(ctx, &environment)), and the cascading deletion plus the finalizer take care of everything else. It is cleaner than deleting resources one by one.
5. A finalizer: yes, and it is essential.
The concrete reasons:
- The namespace is a cluster-scoped object from the point of view of the
ownerReferencesof an object living in another namespace: ownership cannot be established, so it has to be deleted explicitly. - You have to verify that the PVC is released before declaring the deletion done, avoiding orphan volumes with the
Retainclass. - There may be external resources: a DNS record
test-xyz.rutasnorte.example, an entry in the internal billing system. Kubernetes' garbage collector knows about none of that. - It is worth notifying the requester that their environment has expired before completing the deletion.
With the warning from section 9 firmly in mind: if the operator goes down with objects pending finalization, those objects stay in Terminating forever. That is why the finalizer must be fast, fault-tolerant and have a documented escape route.
Conclusion
An operator is a custom resource plus a controller that reconciles it: an expert's operational knowledge encoded in software that runs inside the cluster and never sleeps. Unlike a script or Helm, which act when somebody invokes them, an operator maintains the desired state continuously.
Its engine is the reconciliation loop of 01-02, now seen from the inside: a watch with a local cache, a work queue with deduplication and exponential backoff, and a Reconcile function that receives only a name and whose contract is always the same — read the desired, look at the real, bring them together — and that ends by writing the status through its subresource. Two non-negotiable properties: idempotency, because Reconcile will run many more times than you expect, and no state in memory, because the controller can restart at any moment.
The five-level model — install, upgrades, lifecycle, insights and auto pilot — is the tool for evaluating an operator before adopting it, along with the project's maintenance, the RBAC permissions it asks for and the decisive question of what happens if you uninstall it.
We have settled the debt we opened in 06-01: a PostgreSQL operator replaces our hand-rolled StatefulSet and provides what a StatefulSet can never give on its own — primary election, automatic failover in seconds, read replicas, -rw and -ro Services that rewrite themselves, scheduled backups and point-in-time recovery — with a YAML instead of weeks of work and night shifts. And we have sketched the RutaProgramada controller with Kubebuilder: the markers that generate the CRD and the RBAC, the seven steps of Reconcile, ownerReferences for cascading deletion and finalizers for whatever lives outside the cluster.
And we have ended up where we had to: most teams should not write operators. For stateless applications they add nothing, for well-known software they already exist and are better than the one you would write, and maintaining one means operating a privileged service in perpetuity. Write one only when you have a repetitive, documented procedure that goes wrong at three in the morning.
Closing module 6
This lesson closes the advanced concepts module. The Rutas Norte platform has changed a great deal:
| Component | How it entered module 6 | How it leaves |
|---|---|---|
bookings-postgres |
A 1-replica Deployment with Recreate |
A cluster managed by an operator, 3 instances, automatic failover |
redis-cache |
A Deployment | A StatefulSet with a disk per replica and a warm start |
occupancy-reports |
It did not exist | A nightly CronJob with its PVC, its SA and its NetworkPolicy |
| Logs from every node | Not collected | A collector DaemonSet on every node |
bookings-api |
One container | With a waiting init container and a payments ambassador |
notifications-worker |
A proprietary log in a file | With an adapter emitting structured JSON |
| Pod placement | Wherever they landed | A complete plan: affinity, anti-affinity, a dedicated analytics node |
| A vocabulary of its own | None | The RutaProgramada CRD extending the API |
It is a powerful platform. And it is, right now, completely opaque.
There is no way of knowing whether bookings-api is healthy, beyond its pod saying Running — which only means the process started, not that it works. We do not know how much memory bookings-postgres really uses or whether the 2 GiB we reserved for it in 03-05 is too much or too little. If the reports CronJob failed last night at three in the morning, the only clue is some logs that will be deleted along with the history. If web-store starts answering in three seconds instead of a hundred milliseconds, we will find out from a complaint by a customer who could not buy their ticket.
Everything we have built is flying blind. That ends in module 7: Monitoring and Logging. We will start with the most basic and most important thing — teaching Kubernetes to tell a container that started from one that works — with the health checks and probes of lesson 07-01.
Kubernetes Course
Module 1: Introduction to Kubernetes
- What Is Kubernetes?
- Kubernetes Architecture
- Key Concepts and Terminology
- Setting Up a Kubernetes Cluster
- The Kubernetes CLI: kubectl
- Objects, YAML Manifests and the Declarative Model
- The Course Project: the Rutas Norte Platform
Module 2: Core Kubernetes Components
- Pods
- ReplicaSets
- Deployments
- Updates, Rollbacks and Deployment Strategies
- Services
- Namespaces
- Labels, Selectors and Annotations
Module 3: Configuration and Secret Management
- ConfigMaps
- Secrets
- Environment Variables
- Resource Quotas and Limits
- LimitRanges and Quality of Service (QoS) Classes
- ServiceAccounts and API Access from Pods
Module 4: Networking in Kubernetes
- Cluster Networking
- Service Types
- Internal DNS and Service Discovery
- Ingress Controllers
- TLS and Certificate Management with cert-manager
- Network Policies
Module 5: Storage in Kubernetes
- Volumes
- Persistent Volumes
- Persistent Volume Claims
- Storage Classes
- Dynamic Provisioning, Expansion and Snapshots
- Backup and Restore of Persistent Data
Module 6: Advanced Kubernetes Concepts
- StatefulSets
- DaemonSets
- Jobs and CronJobs
- Init Containers, Sidecars and Multi-Container Patterns
- Scheduling: Affinity, Taints and Tolerations
- Custom Resource Definitions (CRDs)
- Operators and the Controller Pattern
Module 7: Monitoring and Logging
- Health Checks and Probes
- Metrics Server and kubectl top
- Monitoring with Prometheus
- Visualization and Alerting with Grafana and Alertmanager
- Centralized Logging with Elasticsearch, Fluentd and Kibana (EFK)
- Application Debugging and Cluster Events
Module 8: Kubernetes Security
- Role-Based Access Control (RBAC)
- Security Contexts and Container Hardening
- Pod Security Policies and Pod Security Standards
- Network Security
- Image Security
- Auditing, Scanning and Vulnerability Management
Module 9: Scaling and Performance
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
- Cluster Autoscaling
- Event-Driven and Custom-Metric Scaling with KEDA
- High Availability: PodDisruptionBudgets and Topology
- Performance Tuning
Module 10: Kubernetes Ecosystem and Tooling
- Minikube and Local Environments with kind
- Kubeadm
- Helm
- Kustomize
- GitOps with Argo CD and Flux
- Managed Kubernetes: EKS, AKS and GKE
Module 11: Case Studies and Real-World Applications
- Deploying a Web Application
- Running Stateful Applications
- CI/CD with Kubernetes
- Deployment Strategies: Blue-Green and Canary
- Multi-Cluster Management
- Production Operations: Incidents, Runbooks and Costs
