The previous lesson ended with a RutaProgramada CRD that was perfectly defined, validated and queryable, and on which absolutely nothing happened. And in lesson 06-01 we left another debt: bookings-postgres is a StatefulSet that gives its replicas an identity and a disk, but that cannot choose a primary, replicate data or fail over when that primary dies.

Both gaps have the same solution, and it comes down to one sentence worth memorising:

Operator = custom resource + controller that reconciles it.

An operator is an expert's operational knowledge — how a PostgreSQL replica is promoted, how a consistent backup is taken, how a major version is upgraded without losing data — encoded in software that runs inside the cluster and never sleeps.

This is the last lesson of the module. We will settle the two outstanding debts, look inside the reconciliation loop, replace our hand-rolled StatefulSet with a real PostgreSQL operator, sketch out the RutaProgramada controller, and finish by explaining why most teams should not write operators.

Contents

  1. What an operator is, precisely
  2. The real anatomy of the reconciliation loop
  3. Idempotency and no state in memory
  4. The five-level capability model
  5. Operators you have already used in this course
  6. A practical case: a PostgreSQL operator for bookings-postgres
  7. Where to find operators and what to check before adopting one
  8. How one is written: Kubebuilder and controller-runtime
  9. ownerReferences and cascading deletion
  10. When NOT to write an operator

  1. What an operator is, precisely

An operator has exactly two pieces:

  1. One or more CRDs that define the vocabulary: Cluster, Certificate, RutaProgramada. It is the declarative interface with which the user expresses what they want.
  2. A controller — normally a Deployment running in the cluster itself — that watches those objects and does the work.
graph LR
  U[User] -->|kubectl apply<br/>Cluster with 3 replicas| API[kube-apiserver]
  API -->|watch| C[Operator controller<br/>Deployment in the cluster]
  C -->|compares desired vs observed| R{Do they match?}
  R -->|no| A[Act: create pods,<br/>promote a primary,<br/>adjust Services]
  R -->|yes| N[Do nothing]
  A -->|writes| API
  C -->|updates status| API
  API -->|change| C

The term was coined by CoreOS in 2016 with this idea: when an operations team has been administering PostgreSQL for years, it has accumulated a set of procedures — what to do if the primary stops answering, how to add a read replica, in what order to upgrade — that live in runbooks, in scattered scripts and in two people's heads. An operator turns all of that into a program that runs continuously.

The difference from the tools you already know:

Tool When it acts What it maintains
Deployment script When somebody runs it Nothing: it is a one-shot
Helm (10-03) On install and upgrade It renders templates; it does not watch afterwards
Kustomize (10-04) When generating the manifests Nothing at runtime
Operator Continuously The desired state, no matter what

An operator does not install: it maintains. If somebody deletes a pod, it recreates it. If the primary goes down at three in the morning, it promotes a replica without waking anybody. If the disk fills up, it expands it. That permanent vigilance is its value.

  1. The real anatomy of the reconciliation loop

In lesson 01-02 we described the reconciliation loop as the central idea of Kubernetes: observe the desired state, observe the real one, act to bring them together. Now we look inside it, as any serious controller implements it.

graph TB
  W[Informer: WATCH on the API<br/>+ local cache of the state] -->|add/update/delete event| Q[Work queue<br/>with deduplication and delay]
  Q -->|takes one key<br/>namespace/name| REC[Reconcile ns/name]
  REC --> LEER[Read the current object from the cache]
  LEER --> OBS[Observe the real world:<br/>pods, PVCs, services]
  OBS --> COMP{desired == observed?}
  COMP -->|yes| ST[Update status<br/>and finish]
  COMP -->|no| ACT[Carry out the next step]
  ACT --> ST
  ST --> RES{Result?}
  RES -->|error| REQ[Requeue with<br/>exponential backoff]
  RES -->|requeue after N s| REQ2[Requeue with a fixed delay]
  RES -->|ok| FIN[Wait for the next event]
  REQ --> Q
  REQ2 --> Q

The watch and the informer

The controller does not poll the API in a loop: it opens a watch connection and receives a notification of every change. The standard library (client-go) wraps that in an informer, which also maintains a local cache of every object watched.

The cache matters for two reasons:

  • The controller's reads do not hit the apiserver: on a cluster with hundreds of objects, the difference is substantial.
  • The cache may be slightly out of date. A controller has to tolerate reading an object a couple of seconds behind, which reinforces the need for idempotency.

The work queue

Events are not processed directly: each is translated into a key (namespace/name) and put into a queue with three properties:

  • Deduplication: if an object changes five times while it is being processed, the key appears only once. It will be reconciled once with the final state, not five times with intermediate ones.
  • Delay: you can ask for "look at this again in 30 seconds", useful for waiting until something external makes progress.
  • Rate limiting with exponential backoff: a failing object is retried after 5 ms, 10 ms, 20 ms… up to a maximum. A permanently broken object does not consume the whole controller.

The Reconcile function

It is the heart of it, and its signature says a lot:

func (r *RutaProgramadaReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

It receives only a key, not the object and not the event. That design decision is deliberate: Reconcile does not know what changed or why it was called. Its contract is always the same: given this name, read the desired state, look at the real one, and make them match.

It returns two things:

Returns Effect
error != nil Requeue with exponential backoff
Result{Requeue: true} Requeue immediately
Result{RequeueAfter: 30*time.Second} Requeue in 30 seconds
Result{}, nil Done; wait for the next event

Updating the status

The last step of every reconciliation is writing what was observed into .status, through the subresource we studied in 06-06. Here a conventional field appears that deserves an explanation:

status:
  observedGeneration: 4

metadata.generation is incremented by the apiserver every time the spec changes (not when labels or annotations change). The controller stores in status.observedGeneration the generation it has already processed. By comparing the two, anybody can tell whether the controller is up to date:

kubectl get rutaprogramada rn-041-bilbao-santander -n rutas-norte-pro \
  -o jsonpath='generation={.metadata.generation} observed={.status.observedGeneration}{"\n"}'
generation=4 observed=4

If they differ, there is a spec change the controller has not attended to yet.

  1. Idempotency and no state in memory

Two non-negotiable properties of any Reconcile.

Idempotency

Reconcile can run many more times than you expect: because of a real change, because of a retry after an error, because of a periodic resync of the informer (every 10 hours by default), or simply because the controller restarted and is reconciling everything that exists.

That is why Reconcile must never think in terms of "create" but of "make sure it exists":

// BAD: fails on the second run with AlreadyExists
if err := r.Create(ctx, deployment); err != nil {
    return ctrl.Result{}, err
}

// GOOD: idempotent
existing := &appsv1.Deployment{}
err := r.Get(ctx, client.ObjectKeyFromObject(desired), existing)
switch {
case apierrors.IsNotFound(err):
    return ctrl.Result{}, r.Create(ctx, desired)
case err != nil:
    return ctrl.Result{}, err
default:
    if !reflect.DeepEqual(existing.Spec, desired.Spec) {
        existing.Spec = desired.Spec
        return ctrl.Result{}, r.Update(ctx, existing)
    }
    return ctrl.Result{}, nil   // it was already fine: do nothing
}

A practical corollary: a reconciliation that changes nothing is the normal case and must be cheap. If your Reconcile writes to the API on every pass even when nothing has changed, you will cause an infinite loop: the write generates an event, the event triggers another reconciliation, and so on indefinitely. It is the classic mistake in everybody's first operator, and it is spotted because the apiserver records thousands of updates per minute on the same object.

No state in memory

The controller cannot remember anything between reconciliations. Not in global variables, not in maps, not in local files.

The reasons:

  • It can restart at any moment (an update, an eviction, a node failure) and would lose everything.
  • There may be several replicas for high availability; only one is active thanks to leader election, but the handover can happen at any time.
  • In-memory state drifts out of sync with the real world without anybody noticing.

All the state the controller needs to remember must live in the Kubernetes API: in .status, in annotations, in labels or in the objects it manages. If your operator needs to know "I already ran today's backup", that goes in status.lastBackup, not in a variable.

The positive consequence is that a well-written operator can be killed and started at any moment with no side effects. It is a property worth testing deliberately: delete the controller's pod halfway through an operation and check that it picks up correctly.

  1. The five-level capability model

The Operator Framework project defined a scale for measuring how much an operator knows how to do. It is the best tool for evaluating one before adopting it. Applied to a database like bookings-postgres:

Level Name What it means for a database
1 Basic install It creates the StatefulSet, the Service, the PVC and the Secret. Equivalent to what we did by hand in 06-01
2 Seamless upgrades It changes the minor version (16.4 → 16.6) in the right order, replicas before the primary
3 Full lifecycle Read replicas, scheduled backups, restore, scaling, configuration changes with no downtime
4 Deep insights It exposes metrics and alerts, and the status reflects the real replication state with its lag
5 Auto pilot It detects the downed primary and fails over on its own, tunes parameters to the load, repairs corrupt replicas, scales itself

What each jump adds, in detail:

Level 1 → 2: upgrading stops being a manual procedure. The operator knows the replicas have to be updated first and the promotion done afterwards, and that each one has to be waited for while it syncs.

Level 2 → 3: here is the bulk of the value. Scheduled backups with verification, point-in-time restore, adding a read replica with a single change in the spec, changing shared_buffers without dropping connections.

Level 3 → 4: the operator stops being a black box. It publishes metrics for replication lag, database size and backup state, and its status tells the truth about what is going on.

Level 4 → 5: automatic failover. It is the level that separates "it saves me work" from "I can sleep at night". It is also the hardest and the one requiring the most care: an operator that fails over badly can cause a split brain with two primaries accepting writes.

When evaluating an operator, ask about the level. Many flashy projects stop at 2, and for level 2 the added complexity is not worth it: a Helm chart already does that.

  1. Operators you have already used in this course

Without calling them that, we have been using operators for several modules.

cert-manager (lesson 04-05)

When we applied a Certificate for www.rutasnorte.example, the following happened without us doing anything else:

  1. The cert-manager controller observed the new Certificate object.
  2. It created a CertificateRequest, generated a private key and stored it in a Secret.
  3. It created an Order and a Challenge for the ACME protocol.
  4. It published a temporary Ingress with the HTTP-01 challenge token.
  5. It waited for Let's Encrypt to validate, obtained the certificate and wrote it into the Secret.
  6. And ever since it has been watching the expiry date and repeats the process 30 days before it expires.

Step 6 is the definition of an operator. A script would have done steps 1 to 5; only a controller that runs forever does step 6. In capability terms, cert-manager is at level 5 for its domain: it renews with no human intervention.

kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6d8f7c9b54-p2m4x              1/1     Running   0          24d
cert-manager-cainjector-7b9d4f8c6-k7t2v    1/1     Running   0          24d
cert-manager-webhook-59c8d7b64-w3n8q       1/1     Running   0          24d

Three Deployments: the controller, a CA certificate injector and the validating webhook for its CRDs. It is the typical anatomy of a mature operator.

The snapshot-controller (lesson 05-05)

When we created a VolumeSnapshot of the bookings data, a controller observed it, talked to the CSI driver, created the corresponding VolumeSnapshotContent and updated status.readyToUse. Same pattern: a CRD plus a controller.

Velero (lesson 05-06)

Its Backup, Restore and Schedule are CRDs, and its controller is what runs the backups, applies the hooks before and after, and honours the retention. A Velero Schedule is an operator creating Jobs, conceptually the same as what the CronJob controller does with our occupancy-reports.

Prometheus Operator (lesson 07-03)

Coming up. Its Prometheus, ServiceMonitor, PodMonitor and PrometheusRule CRDs let you declare "collect metrics from every Service with this label" and the operator generates and reloads the Prometheus configuration. Without it, adding a new target means hand-editing a configuration file hundreds of lines long.

And the native controllers

It is worth closing the circle: the Deployment controller, the ReplicaSet one and the Job one work in exactly the same way. They watch objects, compare desired with observed and act. The only difference is that their types come built in and their code lives inside the kube-controller-manager rather than in a separate Deployment. The pattern is identical; operators simply extend it to domains Kubernetes does not know about.

  1. A practical case: a PostgreSQL operator for bookings-postgres

We come to the debt from 06-01. Our hand-rolled StatefulSet has one replica and cannot do anything else. We are going to replace it with CloudNativePG, a mature open-source PostgreSQL operator.

Installing the operator

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.1.yaml

kubectl get deployment -n cnpg-system
kubectl get crds | grep postgresql
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
cnpg-controller-manager    1/1     1            1           47s

backups.postgresql.cnpg.io                    2026-08-05T21:40:11Z
clusters.postgresql.cnpg.io                   2026-08-05T21:40:11Z
poolers.postgresql.cnpg.io                    2026-08-05T21:40:11Z
scheduledbackups.postgresql.cnpg.io           2026-08-05T21:40:12Z

One Deployment (the controller) and four CRDs (the vocabulary). Exactly the two pieces of the definition.

Declaring the cluster

bookings-postgres is now described like this:

# k8s/base/bookings-postgres-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: bookings-postgres
  namespace: rutas-norte-pro
  labels:
    app: bookings-postgres
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  instances: 3                      # one primary and two replicas
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4

  primaryUpdateStrategy: unsupervised   # the operator fails over on its own when updating

  bootstrap:
    initdb:
      database: bookings
      owner: rutasnorte
      secret:
        name: bookings-postgres-credentials
      localeCollate: es_ES.UTF-8
      localeCType: es_ES.UTF-8

  storage:
    size: 20Gi
    storageClass: rutasnorte-fast

  walStorage:                        # WAL on a separate volume: better performance
    size: 5Gi
    storageClass: rutasnorte-fast

  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "512MB"
      work_mem: "8MB"
      log_min_duration_statement: "500"   # log queries taking more than 500 ms

  resources:
    requests:
      cpu: "1"
      memory: 2Gi
    limits:
      cpu: "1"
      memory: 2Gi                    # requests == limits: QoS Guaranteed (03-05)

  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: required    # never two instances on the same node (06-05)
    nodeSelector:
      disk: ssd

  monitoring:
    enablePodMonitor: true           # metrics for Prometheus (07-03)

  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: "s3://rutasnorte-backups/bookings-postgres"
      s3Credentials:
        accessKeyId:
          name: backup-credentials
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: backup-credentials
          key: SECRET_ACCESS_KEY
      wal:
        compression: gzip
        maxParallel: 4
      data:
        compression: gzip
        immediateCheckpoint: false
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: bookings-postgres-nightly-backup
  namespace: rutas-norte-pro
  labels:
    app: bookings-postgres
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  schedule: "0 30 2 * * *"          # 02:30 (6-field format, with seconds)
  backupOwnerReference: self
  cluster:
    name: bookings-postgres
kubectl apply -f k8s/base/bookings-postgres-cluster.yaml
kubectl get cluster -n rutas-norte-pro
NAME                AGE     INSTANCES   READY   STATUS                     PRIMARY
bookings-postgres   3m42s   3           3       Cluster in healthy state   bookings-postgres-1
kubectl get pods,svc -n rutas-norte-pro -l cnpg.io/cluster=bookings-postgres
NAME                      READY   STATUS    RESTARTS   AGE
pod/bookings-postgres-1   1/1     Running   0          3m
pod/bookings-postgres-2   1/1     Running   0          2m
pod/bookings-postgres-3   1/1     Running   0          2m

NAME                          TYPE        CLUSTER-IP       PORT(S)
service/bookings-postgres-rw  ClusterIP   10.96.201.14     5432/TCP
service/bookings-postgres-ro  ClusterIP   10.96.188.77     5432/TCP
service/bookings-postgres-r   ClusterIP   10.96.140.22     5432/TCP

The three Services are the piece a StatefulSet cannot provide:

Service Points to Use in Rutas Norte
-rw Only the current primary Writes from bookings-api
-ro Only the replicas Queries from occupancy-reports
-r Any instance Reads that tolerate lag

And the essential part: when the primary changes, the operator rewrites the -rw Service so that it points to the new one. The application never notices. That is exactly what a StatefulSet cannot do, because its headless Service gives stable names but does not know which of those names is the primary.

The acid test: killing the primary

kubectl delete pod bookings-postgres-1 -n rutas-norte-pro
kubectl get cluster bookings-postgres -n rutas-norte-pro -w
NAME                INSTANCES   READY   STATUS                            PRIMARY
bookings-postgres   3           2       Failing over to bookings-postgres-2   bookings-postgres-1
bookings-postgres   3           2       Cluster in healthy state              bookings-postgres-2
bookings-postgres   3           3       Cluster in healthy state              bookings-postgres-2

Within seconds: the outage is detected, bookings-postgres-2 is promoted, the -rw Service is rewritten, and the downed instance comes back as a replica. No human intervention, no runbook, no phone call at three in the morning.

What the operator solves that you would otherwise do by hand

Capability With a hand-rolled StatefulSet (06-01) With an operator
Primary election Decide by convention that it is ordinal 0 The operator decides it, records it in status and publishes it
Failover Detect the outage, promote, reconfigure replicas, change the Service: all manual Automatic, in seconds
Read replicas pg_basebackup by hand, primary_conninfo, replication slots instances: 3
Read/write routing Nothing: a single Service for everything -rw, -ro and -r Services kept up to date
Scheduled backups A CronJob with pg_dump (05-06) ScheduledBackup with continuous WAL
Point-in-time recovery Impossible without hand-built WAL archiving recoveryTarget.targetTime
Minor version upgrade Edit the image and hope Replicas first, failover, primary afterwards
Major version upgrade A dump, a restore and hours of downtime A procedure guided by the operator
Expanding the disk kubectl patch on each PVC (05-05) Change storage.size
Replication slots Manual configuration, and they break when pods are recreated Managed
Real health probes pg_isready, which does not tell a primary from a replica The operator knows each instance's role
Metrics Add an exporter sidecar (06-04) enablePodMonitor: true

That is the whole argument in favour of operators for stateful software: the left-hand column is weeks of work, fragile procedures and night shifts; the right-hand one is fields in a YAML.

Migrating without losing the data

With what we learned in 05-06 and 06-01, the migration from our StatefulSet is done by logical restore, not by moving volumes:

spec:
  bootstrap:
    initdb:
      database: bookings
      owner: rutasnorte
      import:
        type: microservice
        databases: ["bookings"]
        source:
          externalCluster: old-statefulset
  externalClusters:
    - name: old-statefulset
      connectionParameters:
        host: bookings-postgres-nodes.rutas-norte-pro.svc.cluster.local
        user: rutasnorte
        dbname: bookings
      password:
        name: bookings-postgres-credentials
        key: password

The operator starts up, connects to the old StatefulSet, imports the database and builds the new cluster. Afterwards you verify the number of bookings, point bookings-api at the -rw Service and retire the StatefulSet.

  1. Where to find operators and what to check before adopting one

Where to look

Source What it holds
OperatorHub.io A community catalogue with a declared capability level
Artifact Hub Helm charts and operators, with download counts
The project's official repository Almost always the most reliable and up-to-date source
Your cloud provider's marketplace Operators validated for EKS, AKS or GKE (10-06)

The checklist

Adopting an operator means adopting a dependency that will have broad permissions over your cluster and that your data will depend on. Before installing it:

1. Maintenance. When was the last commit? How many people contribute? Are there regular releases? How many open issues with no response? An abandoned operator managing your database is a serious problem, because uninstalling it without losing data is rarely trivial.

2. The RBAC permissions it asks for. This is the point most often overlooked. Look at it before applying the manifest:

curl -sL <manifest-url> | grep -A40 'kind: ClusterRole'

The questions: does it ask for cluster-admin? (an immediate red flag) Does it ask for access to all the cluster's Secrets? Can it create ClusterRoleBindings, that is, widen its own permissions? A compromised operator with broad permissions amounts to a compromised cluster. We will come back to this in 08-01.

3. Maturity. What capability level does it declare and which one does it really meet? Are there documented production use cases? Is there an upgrade guide between versions of the operator itself?

4. What happens if you uninstall it. The decisive question:

  • Do the objects it managed keep working, or do they stop?
  • Do its CRDs have finalizers that would leave objects hanging in Terminating?
  • Can you export the data to a standard format?
  • Is there a documented exit procedure?

An operator you cannot get out of is a technological hostage situation. With CloudNativePG, for example, the backups in Barman format are restorable with standard PostgreSQL tools: there is a way out.

5. Resources and scope. How much CPU and memory does the controller use? Does it watch the whole cluster or can it be limited to particular namespaces? An operator watching every object in a large cluster can use a fair amount of memory.

6. Upgrade model. How is the operator upgraded without affecting what it manages? Is it backwards compatible with the CRDs already deployed?

  1. How one is written: Kubebuilder and controller-runtime

Nobody writes an operator from scratch. The standard tools:

Tool What it provides
controller-runtime A Go library with informers, queues, a cached client and a controller manager
Kubebuilder Scaffolding: it generates the project, the CRDs from Go structs, the RBAC and the deployment
Operator SDK It wraps Kubebuilder and adds Helm and Ansible as alternatives to Go

With Operator SDK you can build an operator without writing Go, using a Helm chart or an Ansible playbook as the reconciliation logic. It is a reasonable route for simple operators, though it limits the capabilities to levels 1 and 2.

Starting a project

mkdir -p ~/projects/rutasnorte-operator && cd ~/projects/rutasnorte-operator

kubebuilder init \
  --domain rutasnorte.example \
  --repo github.com/rutasnorte/rutasnorte-operator

kubebuilder create api \
  --group rutasnorte \
  --version v1 \
  --kind RutaProgramada \
  --resource --controller

The generated structure:

api/v1/rutaprogramada_types.go        <- the Go types: the CRD comes from here
internal/controller/rutaprogramada_controller.go   <- Reconcile goes here
config/crd/bases/                     <- CRDs generated with "make manifests"
config/rbac/                          <- Roles generated from the markers
config/samples/                       <- sample instances
Makefile                              <- make manifests, make docker-build, make deploy

The types, from which the CRD comes

The CRD from lesson 06-06 is not written by hand in a real project: it is generated from annotated Go structs.

// api/v1/rutaprogramada_types.go

type RutaProgramadaSpec struct {
    // Commercial code of the route, format XX-999 (e.g. RN-041)
    // +kubebuilder:validation:Pattern=`^[A-Z]{2}-[0-9]{3}$`
    Code string `json:"code"`

    // Town the journey starts from
    // +kubebuilder:validation:MinLength=2
    // +kubebuilder:validation:MaxLength=60
    Origin string `json:"origin"`

    // Town the journey ends at
    // +kubebuilder:validation:MinLength=2
    // +kubebuilder:validation:MaxLength=60
    Destination string `json:"destination"`

    // Total seats offered on each departure
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=90
    Seats int32 `json:"seats"`

    // Daily departure times in HH:MM format
    // +kubebuilder:validation:MinItems=1
    // +kubebuilder:validation:items:Pattern=`^([01][0-9]|2[0-3]):[0-5][0-9]$`
    Timetables []string `json:"timetables"`

    // Whether the route currently accepts sales
    // +kubebuilder:default=true
    // +optional
    Active bool `json:"active,omitempty"`
}

type RutaProgramadaStatus struct {
    // +optional
    Phase string `json:"phase,omitempty"`
    // +optional
    SeatsSold int32 `json:"seatsSold,omitempty"`
    // Generation of the spec the controller last processed
    // +optional
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`
    // +optional
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=route;routes,categories=rutasnorte
// +kubebuilder:printcolumn:name="Code",type=string,JSONPath=`.spec.code`
// +kubebuilder:printcolumn:name="Origin",type=string,JSONPath=`.spec.origin`
// +kubebuilder:printcolumn:name="Destination",type=string,JSONPath=`.spec.destination`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
type RutaProgramada struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec   RutaProgramadaSpec   `json:"spec,omitempty"`
    Status RutaProgramadaStatus `json:"status,omitempty"`
}

The // +kubebuilder:... comments are markers the generator translates into the CRD's OpenAPI schema. make manifests produces exactly the YAML we wrote by hand in 06-06, and the ordinary comments become the descriptions that feed kubectl explain.

The skeleton of Reconcile

This is the heart of the RutaProgramada operator: every active route must have its own CronJob generating the occupancy report for that route.

// internal/controller/rutaprogramada_controller.go

// Permissions the controller needs. These markers generate config/rbac/role.yaml
// +kubebuilder:rbac:groups=rutasnorte.rutasnorte.example,resources=rutasprogramadas,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=rutasnorte.rutasnorte.example,resources=rutasprogramadas/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=rutasnorte.rutasnorte.example,resources=rutasprogramadas/finalizers,verbs=update
// +kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch

func (r *RutaProgramadaReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := logf.FromContext(ctx)

    // ── 1. READ THE DESIRED STATE ────────────────────────────────────────
    var route rutasnortev1.RutaProgramada
    if err := r.Get(ctx, req.NamespacedName, &route); err != nil {
        // NotFound means the route was deleted. There is nothing to do:
        // the derived objects delete themselves through ownerReferences (section 9).
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // ── 2. INACTIVE ROUTE: withdraw whatever was created ─────────────────
    if !route.Spec.Active {
        log.Info("Route inactive; the reports CronJob is not maintained", "code", route.Spec.Code)
        return ctrl.Result{}, r.updateStatus(ctx, &route, "Cancelled")
    }

    // ── 3. BUILD THE DESIRED STATE ───────────────────────────────────────
    // A pure function: the same input data ALWAYS gives the same output object.
    // That purity is what makes the comparison in step 5 possible.
    desired := r.buildReportCronJob(&route)

    // ── 4. ownerReferences: the CronJob belongs to the route ─────────────
    // When the route is deleted, the garbage collector deletes the CronJob (section 9).
    if err := ctrl.SetControllerReference(&route, desired, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }

    // ── 5. RECONCILE: create if missing, update if different, nothing if equal ──
    var current batchv1.CronJob
    err := r.Get(ctx, client.ObjectKeyFromObject(desired), &current)
    switch {
    case apierrors.IsNotFound(err):
        log.Info("Creating the reports CronJob", "route", route.Spec.Code)
        if err := r.Create(ctx, desired); err != nil {
            // Returning the error makes the queue requeue with exponential backoff
            return ctrl.Result{}, err
        }
        r.Recorder.Eventf(&route, corev1.EventTypeNormal, "CronJobCreated",
            "Created the reports CronJob for route %s", route.Spec.Code)

    case err != nil:
        return ctrl.Result{}, err

    default:
        // IDEMPOTENCY: if nothing changed, do NOT write. Writing here would cause
        // an event, which would cause another reconciliation: an infinite loop.
        if !equality.Semantic.DeepDerivative(desired.Spec, current.Spec) {
            log.Info("Updating the reports CronJob", "route", route.Spec.Code)
            current.Spec = desired.Spec
            if err := r.Update(ctx, &current); err != nil {
                return ctrl.Result{}, err
            }
        }
    }

    // ── 6. OBSERVE THE REAL WORLD AND WRITE THE STATUS ───────────────────
    sold, err := r.querySeatsSold(ctx, &route)
    if err != nil {
        // A transient failure querying the database: retry in a minute
        // without marking the reconciliation as failed.
        log.Error(err, "could not query the seats sold")
        return ctrl.Result{RequeueAfter: time.Minute}, nil
    }

    route.Status.Phase = "Active"
    route.Status.SeatsSold = sold
    route.Status.ObservedGeneration = route.Generation   // "already processed" marker
    meta.SetStatusCondition(&route.Status.Conditions, metav1.Condition{
        Type:    "ReportsScheduled",
        Status:  metav1.ConditionTrue,
        Reason:  "CronJobActive",
        Message: fmt.Sprintf("Reports for route %s scheduled", route.Spec.Code),
    })
    // Written through the status SUBRESOURCE: it does not touch the user's spec (06-06)
    if err := r.Status().Update(ctx, &route); err != nil {
        return ctrl.Result{}, err
    }

    // ── 7. PERIODIC RECONCILIATION ───────────────────────────────────────
    // Even with no events, check every 10 minutes: it catches external changes
    // and drift that generated no notification.
    return ctrl.Result{RequeueAfter: 10 * time.Minute}, nil
}

func (r *RutaProgramadaReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&rutasnortev1.RutaProgramada{}).
        // Watch our own CronJobs too: if somebody deletes one by hand,
        // a reconciliation of its route is triggered and it is recreated.
        Owns(&batchv1.CronJob{}).
        Complete(r)
}

The seven steps are the skeleton of any operator, whether for bus routes or for PostgreSQL: read the desired state, handle the deletion case, build what ought to exist, mark the ownership, create or update only if needed, observe reality and write the status, and decide when to look again.

The Owns(&batchv1.CronJob{}) in SetupWithManager deserves a comment: it makes the controller receive events for the CronJobs it created, and translate them into reconciliations of the owning RutaProgramada. It is what makes deleting the CronJob by hand cause its recreation within seconds. Without that line, the operator would only react to changes in its own custom resources.

The RBAC permissions

The +kubebuilder:rbac: markers above generate this ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rutasnorte-operator-manager-role
rules:
  - apiGroups: ["rutasnorte.rutasnorte.example"]
    resources: ["rutasprogramadas"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["rutasnorte.rutasnorte.example"]
    resources: ["rutasprogramadas/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]

Note the principle of least privilege: the operator can only touch CronJobs and its own resources. It cannot read Secrets, create Deployments or touch anything else. When you evaluate somebody else's operator (section 7), this is exactly what you should look at and compare against what the operator says it does. RBAC in detail is lesson 08-01.

Running and deploying

# Generate the CRDs and RBAC from the markers
make manifests generate

# Install the CRDs in the cluster
make install

# Run the controller LOCALLY against the cluster: the development cycle
make run
INFO  setup   starting manager
INFO  Starting EventSource  {"controller": "rutaprogramada", "source": "kind source: *v1.RutaProgramada"}
INFO  Starting Controller   {"controller": "rutaprogramada"}
INFO  Creating the reports CronJob  {"route": "RN-041"}
# And for production: an image and a deployment inside the cluster
make docker-build docker-push IMG=registry.rutasnorte.example/rutasnorte-operator:0.1.0
make deploy IMG=registry.rutasnorte.example/rutasnorte-operator:0.1.0

make run is the practical advantage of this model: the controller runs on your laptop, with a debugger if you need one, talking to the real cluster. There is no need to build images for every iteration.

  1. ownerReferences and cascading deletion

In lesson 02-02 we saw that a ReplicaSet puts ownerReferences on its pods, and that this is why deleting a Deployment deletes everything underneath. Operators use the same mechanism, and it is essential to understand it.

    if err := ctrl.SetControllerReference(&route, desired, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }

That line writes into the generated CronJob:

  ownerReferences:
    - apiVersion: rutasnorte.rutasnorte.example/v1
      kind: RutaProgramada
      name: rn-041-bilbao-santander
      uid: 3f1a9c04-8e2b-4c71-9a55-71b0e2d8c4f3
      controller: true
      blockOwnerDeletion: true

The consequences:

  1. Automatic cascading deletion. When the RutaProgramada is deleted, Kubernetes' garbage collector deletes the CronJob. The operator does not have to write any clean-up code: the cluster itself takes care of it.
  2. Ownership is visible. kubectl describe cronjob shows Controlled By: RutaProgramada/rn-041-bilbao-santander, which makes it traceable where each object came from.
  3. Events towards the owner. Thanks to Owns(), changes to the CronJob trigger reconciliations of the route.

Two important restrictions:

  • The owner and the owned object must be in the same namespace. A namespaced object cannot be owned by one from a different namespace.
  • A cluster-scoped object cannot be owned by a namespaced one. If your operator creates ClusterRoles or PersistentVolumes, you will have to clean them up yourself.

Finalizers: clean-up outside the cluster

When the operator creates external resources — a bucket, a DNS record, a database at a provider — ownerReferences are of no use: Kubernetes' garbage collector knows nothing about those resources.

That is what finalizers are for, the same mechanism we saw protecting PVCs in 05-03:

const finalizer = "rutasnorte.example/cleanup-external-resources"

if !route.DeletionTimestamp.IsZero() {
    // The object is marked for deletion but has NOT been deleted:
    // the finalizer holds it until we remove it.
    if controllerutil.ContainsFinalizer(&route, finalizer) {
        if err := r.deleteExternalResources(ctx, &route); err != nil {
            return ctrl.Result{}, err   // it will retry; the object stays held
        }
        controllerutil.RemoveFinalizer(&route, finalizer)
        if err := r.Update(ctx, &route); err != nil {
            return ctrl.Result{}, err
        }
    }
    return ctrl.Result{}, nil    // now Kubernetes completes the deletion
}

// A live object: make sure it has the finalizer
if !controllerutil.ContainsFinalizer(&route, finalizer) {
    controllerutil.AddFinalizer(&route, finalizer)
    if err := r.Update(ctx, &route); err != nil {
        return ctrl.Result{}, err
    }
}

A first-order operational warning: if the operator stops working and there are objects carrying its finalizer, those objects stay in Terminating forever. kubectl delete hangs and there is no clean way out other than editing the object and removing the finalizer by hand:

kubectl patch rutaprogramada rn-041-bilbao-santander -n rutas-norte-pro \
  --type=merge -p '{"metadata":{"finalizers":null}}'

That leaves the external resources orphaned, and they will have to be cleaned up some other way. It is one of the reasons behind the "what happens if I uninstall it?" question of section 7: always uninstall the operator after deleting its objects, never before.

  1. When NOT to write an operator

Writing an operator is fun and almost always unnecessary. Before you start, put it through this filter.

Do not write one if your application is stateless

A Deployment, a Service, a HorizontalPodAutoscaler and a ConfigMap cover bookings-api, web-store and notifications-worker completely. An operator would add nothing: the Deployment controller already does the reconciliation.

Operators shine with stateful software and complex operational procedures: databases, message queues, consensus systems, distributed stores. If your application restarts with no consequences, you do not need one.

Do not write one if Helm or Kustomize will do

Need Tool
Deploying with different values per environment Helm (10-03) or Kustomize (10-04)
Automatically applying what is in Git GitOps with Argo CD or Flux (10-05)
Reacting to failures and running procedures continuously An operator
Running something periodically A CronJob (06-03)
Validating or modifying objects at admission A webhook, not an operator

The deciding question: "what has to happen when something breaks at three in the morning?" If the answer is "nothing, the Deployment recreates it", you do not need an operator. If it is "a replica has to be promoted, the routing reconfigured and somebody notified", then there is an operator there.

Do not write one if one already exists

For PostgreSQL, MySQL, Redis, Kafka, MongoDB, Elasticsearch, RabbitMQ and practically any well-known software mature operators already exist, maintained by teams that have been at it for years. Writing your own means reimplementing, worse, what others have already solved, and maintaining it on your own.

The real cost of maintaining an operator

What people underestimate:

  • It is a production service with privileged permissions. It needs deployment, updates, monitoring, alerting and on-call.
  • A failure in the operator affects everything it manages. A Reconcile with a bug can delete objects in cascade across the whole cluster. There are famous public incidents caused by this.
  • CRDs are a public API with the compatibility obligations we saw in 06-06. Changing the schema afterwards is expensive.
  • Testing an operator is hard. You need tests with envtest or an ephemeral cluster, and the reconciliation logic has many possible paths.
  • It requires Go and deep Kubernetes knowledge in the team, in perpetuity, not just while it is being written.

A sensible rule of thumb:

Write an operator when you have a documented, repetitive operational procedure that runs often and that goes wrong when a tired human does it. If you cannot write that runbook precisely, you will not be able to encode it either.

Cheaper alternatives

Instead of an operator Try this first
Automating a deployment Helm + GitOps (10-03, 10-05)
Running something periodically A CronJob (06-03)
Reacting to a one-off event A Job triggered from the CI pipeline
Adding configuration to pods A mutating webhook or an init container (06-04)
Managing a database An existing operator or a managed service (10-06)
Validating manifests Policies with Kyverno or schemas in CI

Common Mistakes and Tips

The infinite reconciliation loop. Mistake number one: Reconcile writes to the API on every pass even when nothing has changed, the write generates an event, the event triggers another reconciliation. Symptom: thousands of updates per minute on the same object. Fix: compare before writing, with DeepDerivative or similar.

Keeping state in memory. Global variables, "already done it" maps. They are lost on restart and drift out of sync with the real world. All the state goes in .status, in annotations or in the managed objects.

Writing .status without the subresource. An Update of the whole object can overwrite a spec change the user has just made. Always use r.Status().Update() with the subresource enabled.

Forgetting ownerReferences. The objects created by the operator are left orphaned when the main resource is deleted. Over time the cluster fills with CronJobs, Services and Secrets nobody knows the origin of.

Uninstalling the operator before deleting its objects. If it uses finalizers, the objects stay in Terminating forever. The right order: delete the objects, check that they disappear, and only then uninstall the operator.

Adopting an operator that asks for cluster-admin. It is an open door to the whole cluster. Always review the ClusterRole before applying the manifest, and be suspicious of anyone asking for access to all Secrets without justifying it.

Confusing "it has CRDs" with "it is an operator". A CRD with no controller is a database with a form, as we saw in 06-06. Check that there is a Deployment running and that it writes to the objects' status.

Not reading the capability level. A level-2 operator is not going to save you from a night-time failover. If you adopt it believing it reaches level 5, the surprise will arrive at the worst moment.

Tip: use make run in development. The controller runs on your laptop against the real cluster, with a debugger. The iteration cycle goes from minutes to seconds.

Tip: emit events. r.Recorder.Eventf(...) makes the operator's actions show up in the object's kubectl describe. It is the difference between an operator you can diagnose and one that is a black box.

Tip: record observedGeneration. It lets anybody tell whether the controller has processed the latest spec change, and it is the basis for kubectl wait working reliably on your resources.

Tip: test by killing the controller. Delete it halfway through an operation and check that it picks up correctly when it comes back. If it does not, you have state in memory or you are missing idempotency.

Exercises

Exercise 1: identify the operators in your cluster

On your minikube (profile rutas-norte), identify which operators are installed. For each one, work out: which CRDs it provides, where its controller runs, and what ClusterRole permissions it has. Then reason about why cert-manager is an operator while the Deployment controller, being the same pattern, is not called one.

Exercise 2: the difference between reconciling and deploying

Show experimentally that a controller reconciles continuously and a deployment does not:

  1. Create a Deployment reconciliation-demo with 3 replicas in rutas-norte-dev.
  2. Delete a pod and observe what happens and how quickly.
  3. Manually change a pod's image with kubectl edit pod and observe the result.
  4. Explain which controller acted in each case and with what information.

Exercise 3: design an operator (without writing it)

Rutas Norte wants an EntornoPruebas resource that, on being created, automatically brings about: a namespace of its own, a restore of bookings-postgres from the latest snapshot, a deployment of bookings-api pointing at that database, and complete deletion after 7 days.

Without writing any code, design:

  1. The CRD's spec and status (fields and types).
  2. The steps of the Reconcile function, in order.
  3. The minimum RBAC permissions it would need.
  4. What you would use for the deletion after 7 days, and why.
  5. Whether a finalizer would be needed, and what for.

Solutions

Solution 1

# Which CRDs there are and whose they are
kubectl get crds -o custom-columns=NAME:.metadata.name,GROUP:.spec.group | sort -k2
NAME                                              GROUP
certificaterequests.cert-manager.io                cert-manager.io
certificates.cert-manager.io                       cert-manager.io
clusterissuers.cert-manager.io                     cert-manager.io
issuers.cert-manager.io                            cert-manager.io
volumesnapshotclasses.snapshot.storage.k8s.io      snapshot.storage.k8s.io
volumesnapshotcontents.snapshot.storage.k8s.io     snapshot.storage.k8s.io
volumesnapshots.snapshot.storage.k8s.io            snapshot.storage.k8s.io
# Where the controllers run
kubectl get deployments -A | grep -Ei 'cert-manager|snapshot|controller'
cert-manager    cert-manager              1/1   1   1   24d
cert-manager    cert-manager-cainjector   1/1   1   1   24d
cert-manager    cert-manager-webhook      1/1   1   1   24d
kube-system     snapshot-controller       1/1   1   1   17d
# What permissions cert-manager has
kubectl get clusterrole -l app.kubernetes.io/instance=cert-manager \
  -o custom-columns=NAME:.metadata.name --no-headers | head -5

kubectl describe clusterrole cert-manager-controller-certificates | head -20
Name:  cert-manager-controller-certificates
PolicyRule:
  Resources                              Verbs
  ---------                              -----
  certificaterequests.cert-manager.io     [create delete get list patch update watch]
  certificates.cert-manager.io            [get list patch update watch]
  certificates.cert-manager.io/status     [patch update]
  secrets                                 [create delete get list patch update watch]
  events                                  [create patch]

Narrow permissions: its own CRDs, plus Secrets (necessary: that is where it stores keys and certificates) and events. It does not ask for cluster-admin.

Why cert-manager is called an operator and the Deployment controller is not, given that they are the same pattern:

The difference is not technical, it is one of origin. Both are reconciliation loops over API types. The Deployment controller manages a native type and its code lives inside the kube-controller-manager, a control-plane binary. cert-manager manages types added through CRDs and runs as just another cluster workload.

The term "operator" designates that second situation: the controller pattern applied to a domain Kubernetes does not know about out of the box, packaged as a deployable application. Conceptually, the Deployment controller is a Deployment operator; it is simply that nobody calls it that because it comes included.

Solution 2

# /tmp/reconciliation-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reconciliation-demo
  namespace: rutas-norte-dev
  labels:
    app: reconciliation-demo
    app.kubernetes.io/part-of: rutas-norte
    environment: dev
spec:
  replicas: 3
  selector:
    matchLabels:
      app: reconciliation-demo
      environment: dev
  template:
    metadata:
      labels:
        app: reconciliation-demo
        app.kubernetes.io/part-of: rutas-norte
        environment: dev
    spec:
      automountServiceAccountToken: false
      containers:
        - name: nginx
          image: nginx:1.27.2-alpine
          resources:
            requests:
              cpu: 20m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi
kubectl apply -f /tmp/reconciliation-demo.yaml
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demo
NAME                                   READY   STATUS    RESTARTS   AGE
reconciliation-demo-6b4d8f7c9-h2k4x    1/1     Running   0          22s
reconciliation-demo-6b4d8f7c9-p8m2v    1/1     Running   0          22s
reconciliation-demo-6b4d8f7c9-t5n7q    1/1     Running   0          22s

2. Delete a pod:

kubectl delete pod reconciliation-demo-6b4d8f7c9-h2k4x -n rutas-norte-dev
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demo
NAME                                   READY   STATUS              RESTARTS   AGE
reconciliation-demo-6b4d8f7c9-p8m2v    1/1     Running             0          2m
reconciliation-demo-6b4d8f7c9-t5n7q    1/1     Running             0          2m
reconciliation-demo-6b4d8f7c9-w3x9z    0/1     ContainerCreating   0          1s

In less than a second there is a replacement. Nobody ran any command: the ReplicaSet saw there were 2 pods where there should be 3 and created one.

3. Change a pod's image by hand:

kubectl set image pod/reconciliation-demo-6b4d8f7c9-p8m2v \
  -n rutas-norte-dev nginx=nginx:1.26.2-alpine

kubectl get pods -n rutas-norte-dev -l app=reconciliation-demo \
  -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[0].image
POD                                   IMAGE
reconciliation-demo-6b4d8f7c9-p8m2v   nginx:1.26.2-alpine
reconciliation-demo-6b4d8f7c9-t5n7q   nginx:1.27.2-alpine
reconciliation-demo-6b4d8f7c9-w3x9z   nginx:1.27.2-alpine

The change sticks. This is surprising, but it is correct and very instructive.

4. Which controller acted and with what information:

Case Controller What it compared Result
Pod deleted ReplicaSet Pods matching its selector (2) vs replicas (3) It created a pod
Image changed by hand None The change sticks

The key is what each controller reconciles:

  • The ReplicaSet controller reconciles the number of pods matching its selector. It counts 2, it wants 3, it creates one. It does not care what image they run.
  • The Deployment controller reconciles which ReplicaSets should exist according to the Deployment's template. Since the template did not change, it does nothing.
  • Nobody reconciles the content of an individual pod against the Deployment's template. The template is used when creating the pod, not for watching it afterwards.

A practical lesson: reconciliation is about managed objects, not arbitrary fields. If you delete the modified pod, its replacement will be born from the template and will go back to nginx:1.27.2-alpine. And a kubectl rollout restart deployment/reconciliation-demo would recreate every pod from the template, correcting the drift.

kubectl rollout restart deployment/reconciliation-demo -n rutas-norte-dev
kubectl rollout status deployment/reconciliation-demo -n rutas-norte-dev
kubectl get pods -n rutas-norte-dev -l app=reconciliation-demo \
  -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[0].image
POD                                    IMAGE
reconciliation-demo-7c9e5a2b4-b1k8m    nginx:1.27.2-alpine
reconciliation-demo-7c9e5a2b4-j4t2p    nginx:1.27.2-alpine
reconciliation-demo-7c9e5a2b4-r6n9w    nginx:1.27.2-alpine
kubectl delete -f /tmp/reconciliation-demo.yaml

Solution 3

1. CRD design:

spec:
  requester: string                   # mandatory, the developer's email
  gitBranch: string                   # mandatory, valid branch pattern
  durationDays: integer               # 1-14, default 7
  dataSource:                         # where to restore from
    type: string                      # enum: snapshot, veleroBackup, empty
    name: string                      # name of the snapshot or the backup
  components:                         # what to deploy
    bookingsApi: boolean              # default true
    webStore: boolean                 # default false
  databaseSize: string                # default 5Gi

status:
  phase: string                       # enum: Pending, Creating, Restoring, Ready, Expiring, Error
  namespaceCreated: string
  accessUrl: string
  expiryDate: string (date-time)      # calculated: creationTimestamp + durationDays
  observedGeneration: integer
  conditions: []Condition             # NamespaceCreated, DataRestored, ComponentsReady

The status subresource enabled. additionalPrinterColumns: Requester, Phase, Expiry, Age.

2. The steps of Reconcile:

  1. Read the EntornoPruebas. If it does not exist, finish (IgnoreNotFound).
  2. If it has a DeletionTimestamp, run the finalizer logic (step 10) and finish.
  3. Make sure it has the finalizer if it does not already.
  4. Check the expiry: if now() > status.expiryDate, delete the object itself and finish. The cascade will do the rest.
  5. Make sure the namespace test-<name> exists with ownership labels. (It is a cluster-scoped object: it accepts no ownerReferences from a namespaced object; it is cleaned up in the finalizer.)
  6. Make sure the PVC restored from the snapshot exists, with dataSource (05-05). If it is not Bound yet, update status.phase = "Restoring" and return RequeueAfter: 30s.
  7. Make sure the PostgreSQL StatefulSet/Cluster and its generated credentials Secret exist.
  8. Make sure the Deployments and Services for the components marked in spec.components exist, and the Ingress if appropriate.
  9. Observe reality: are all the pods Ready? Write status.phase, accessUrl, conditions and observedGeneration through the subresource.
  10. Return a calculated RequeueAfter: up to the expiry if it is far off, or 1 minute if it is close, so that step 4 fires in time.

3. Minimum RBAC:

rules:
  - apiGroups: ["rutasnorte.example"]
    resources: ["entornospruebas"]
    verbs: ["get", "list", "watch", "update", "patch", "delete"]
  - apiGroups: ["rutasnorte.example"]
    resources: ["entornospruebas/status", "entornospruebas/finalizers"]
    verbs: ["get", "update", "patch"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["services", "secrets", "persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses", "networkpolicies"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]

Note what is not there: no cluster-admin, no permissions over nodes, none over RBAC. Secrets are unavoidable (it generates credentials), and that alone is reason enough to review the code before granting it those permissions.

4. The deletion after 7 days:

The right option is the reconciliation loop itself with RequeueAfter, not a clean-up CronJob:

  • The operator is already watching the object: checking a date on every reconciliation is free.
  • RequeueAfter guarantees it wakes up in time even if there is no other event.
  • An external CronJob would be a second component with its own logic, its own permissions and its own way of failing, and it could be out of step with what the operator believes.
  • Besides, ttlSecondsAfterFinished (06-03) does not apply: that is for Jobs, not for custom resources.

An implementation detail: the operator deletes its own object (r.Delete(ctx, &environment)), and the cascading deletion plus the finalizer take care of everything else. It is cleaner than deleting resources one by one.

5. A finalizer: yes, and it is essential.

The concrete reasons:

  • The namespace is a cluster-scoped object from the point of view of the ownerReferences of an object living in another namespace: ownership cannot be established, so it has to be deleted explicitly.
  • You have to verify that the PVC is released before declaring the deletion done, avoiding orphan volumes with the Retain class.
  • There may be external resources: a DNS record test-xyz.rutasnorte.example, an entry in the internal billing system. Kubernetes' garbage collector knows about none of that.
  • It is worth notifying the requester that their environment has expired before completing the deletion.

With the warning from section 9 firmly in mind: if the operator goes down with objects pending finalization, those objects stay in Terminating forever. That is why the finalizer must be fast, fault-tolerant and have a documented escape route.

Conclusion

An operator is a custom resource plus a controller that reconciles it: an expert's operational knowledge encoded in software that runs inside the cluster and never sleeps. Unlike a script or Helm, which act when somebody invokes them, an operator maintains the desired state continuously.

Its engine is the reconciliation loop of 01-02, now seen from the inside: a watch with a local cache, a work queue with deduplication and exponential backoff, and a Reconcile function that receives only a name and whose contract is always the same — read the desired, look at the real, bring them together — and that ends by writing the status through its subresource. Two non-negotiable properties: idempotency, because Reconcile will run many more times than you expect, and no state in memory, because the controller can restart at any moment.

The five-level model — install, upgrades, lifecycle, insights and auto pilot — is the tool for evaluating an operator before adopting it, along with the project's maintenance, the RBAC permissions it asks for and the decisive question of what happens if you uninstall it.

We have settled the debt we opened in 06-01: a PostgreSQL operator replaces our hand-rolled StatefulSet and provides what a StatefulSet can never give on its own — primary election, automatic failover in seconds, read replicas, -rw and -ro Services that rewrite themselves, scheduled backups and point-in-time recovery — with a YAML instead of weeks of work and night shifts. And we have sketched the RutaProgramada controller with Kubebuilder: the markers that generate the CRD and the RBAC, the seven steps of Reconcile, ownerReferences for cascading deletion and finalizers for whatever lives outside the cluster.

And we have ended up where we had to: most teams should not write operators. For stateless applications they add nothing, for well-known software they already exist and are better than the one you would write, and maintaining one means operating a privileged service in perpetuity. Write one only when you have a repetitive, documented procedure that goes wrong at three in the morning.

Closing module 6

This lesson closes the advanced concepts module. The Rutas Norte platform has changed a great deal:

Component How it entered module 6 How it leaves
bookings-postgres A 1-replica Deployment with Recreate A cluster managed by an operator, 3 instances, automatic failover
redis-cache A Deployment A StatefulSet with a disk per replica and a warm start
occupancy-reports It did not exist A nightly CronJob with its PVC, its SA and its NetworkPolicy
Logs from every node Not collected A collector DaemonSet on every node
bookings-api One container With a waiting init container and a payments ambassador
notifications-worker A proprietary log in a file With an adapter emitting structured JSON
Pod placement Wherever they landed A complete plan: affinity, anti-affinity, a dedicated analytics node
A vocabulary of its own None The RutaProgramada CRD extending the API

It is a powerful platform. And it is, right now, completely opaque.

There is no way of knowing whether bookings-api is healthy, beyond its pod saying Running — which only means the process started, not that it works. We do not know how much memory bookings-postgres really uses or whether the 2 GiB we reserved for it in 03-05 is too much or too little. If the reports CronJob failed last night at three in the morning, the only clue is some logs that will be deleted along with the history. If web-store starts answering in three seconds instead of a hundred milliseconds, we will find out from a complaint by a customer who could not buy their ticket.

Everything we have built is flying blind. That ends in module 7: Monitoring and Logging. We will start with the most basic and most important thing — teaching Kubernetes to tell a container that started from one that works — with the health checks and probes of lesson 07-01.

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

© Copyright 2026. All rights reserved