The previous lesson ended with a problem that neither Helm nor Kustomize solves. Both tools only act when somebody runs a command. If a colleague runs kubectl edit in rutas-norte-pro at three in the morning to get out of a jam, neither of them finds out. If the laptop of whoever deploys breaks, nobody knows how to deploy. And if the ci-rutasnorte pipeline needs cluster administrator credentials to run kubectl apply, we have opened a security hole that contradicts the least-privilege RBAC we were so careful about in 08-01.

GitOps is the answer, and it is the natural conclusion of everything we have covered so far. The idea is simple to state and transformative in practice: an agent that lives inside the cluster, watches a Git repository continuously, and makes sure the cluster looks like whatever that repository says. Always. Without anybody running anything.

In this lesson we will build that system for Rutas Norte with Argo CD, we will look at its Flux equivalent, we will finally resolve the conflict between the HPA and the replicas field flagged in 09-01, and we will close the problem of secrets in the repository that has been outstanding since 03-02.

1. What GitOps is and its four principles
2. Push model versus pull model
3. Configuration drift and self-healing
4. Argo CD: architecture and installation
5. The Application resource field by field
6. App of apps and ApplicationSet
7. Sync waves and hooks
8. Health and sync statuses
9. Flux: the controllers and their resources
10. Argo CD versus Flux
11. The secrets problem
12. Fields that change on their own: ignoreDifferences
13. Repository structure and promotion between environments
14. Common mistakes and tips
15. Exercises
16. Conclusion

GitOps is an operating model in which the system's desired state lives in Git and an automated agent makes it real continuously. The OpenGitOps working group formalised it in four principles:

1. Declarative: the whole system is described by what must exist, not by the steps to execute. We have had that since 01-06: Kubernetes is declarative by design.
2. Versioned and immutable: the desired state is stored so that the history cannot be altered. Git is the obvious implementation. The question "what changed in production on Tuesday?" goes from being a forensic investigation to git log.
3. Pulled automatically: approved changes are applied without manual intervention. Nobody runs kubectl apply or helm upgrade: a pull request is merged and the system converges.
4. Continuously reconciled: and this is the one that changes everything. The agent does not merely apply when there is a new commit: it continuously checks that the real state matches the desired one and corrects the differences.

That loop never stops. Every three minutes (configurable), the agent asks itself: "is the cluster what Git says?". If it is not, it acts.

This is the key architectural difference, and it has serious security implications.

In the push model, the pipeline runs kubectl apply against the cluster and for that it needs a kubeconfig with permissions. The problem, in detail:

1. That kubeconfig is a secret stored outside the cluster, often managed by another team or by an external provider.
2. To be able to deploy anything into any namespace, it usually ends up holding cluster-admin. That is the easiest route, and that is why it is what most companies have.
3. Anybody who can modify the pipeline definition — a YAML file in the code repository — can run arbitrary commands with those credentials. A malicious pull request against a CI file is equivalent to administrator access to production.
4. The apiserver has to be reachable from outside.
5. The credentials have to be rotated, audited and revoked, and with three clusters, three sets of them.

The least-privilege RBAC we defined for ci-rutasnorte in 08-01 mitigates the problem, but it does not eliminate it: the pipeline still holds a permanent credential pointing into the cluster.

The change runs deep: the most dangerous credential disappears. If somebody steals the Git write token, they can make a commit... which will be recorded, reviewable and revertible.

Configuration drift is the difference between what your files say and what is actually in the system. It accumulates without anybody noticing. Real cases from Rutas Norte:

• During an incident over the May bank-holiday weekend, somebody ran kubectl scale deploy/bookings-api --replicas=12. It worked. Nobody carried it back into the manifest. Three weeks later, a routine deployment returned the Deployment to 4 replicas and the service was overwhelmed.
• A memory limit was changed with kubectl edit to test a hypothesis. It stayed that way for six months.
• An annotation was added to the Ingress to tune a timeout. When the pre cluster was recreated, it vanished and nobody knew why the behaviour had changed.

With GitOps, kubectl scale deploy/bookings-api --replicas=12 in production has a different ending. Three minutes later:

It is back to 4. And that is exactly what we want. It is not that the machine is being stubborn: it is that if 12 replicas was the right decision, it has to be in Git, where it is reviewed, documented and present the next time the environment is recreated.

The cultural change this brings about matters more than the technology: the only way to change production is through a pull request. At first it creates friction; two months in, nobody wants to go back.

Self-healing is a decision, not an obligation. Argo CD lets you enable selfHeal per application. In rutas-norte-dev it can be worth leaving it off so people can experiment; in rutas-norte-pro it must be on.

A key detail: the application controller is the only one that needs permissions over the cluster's objects. The API server only manages Argo CD resources. That separation matters for RBAC.

The installation consistent with what we have learned is via Helm (10-03), with versioned values:

In production the admin user is disabled and an identity provider is used (OIDC, corporate SSO). Argo CD's RBAC is configured separately from Kubernetes's, in the argocd-rbac-cm ConfigMap:

And the repository is registered with a read-only credential: Argo CD never writes to the manifest repository. It is the least privilege of 08-01 applied here.

Application is a CRD (06-06). Each instance says: "from this repository, this path, to this cluster and namespace, with this policy".

An AppProject restricts what a group of applications can do. It is an essential security layer when several teams share an Argo CD.

That clusterResourceWhitelist: [] is important: it means that not even a mistake in a manifest can create a ClusterRole or a StorageClass. Without it, whoever can write to the repository can escalate privileges in the cluster.

Argo CD runs helm template internally, so it does not create Helm releases: you will see nothing in helm list. Argo CD holds the state, not Helm. It is consistent with GitOps (the state is in Git), but it surprises you the first time.

A very useful pattern is multiple sources: the chart comes from a public repository and the values from the company's repository, referenced with ref: values and valueFiles: [$values/platform/.../values-pro.yaml].

With twenty applications, creating each Application by hand and applying it with kubectl takes us back to the original problem. The solution: an Application that manages a directory full of Applications.

That is the last time anybody runs kubectl apply at Rutas Norte. From then on, adding a new application means adding a file to the applications/ directory and merging the pull request.

One definition, three applications. There are more generators, and some of them open up notable possibilities:

That last one is spectacular in practice: you open a PR with the preview label and a complete environment appears deployed; you close it and it disappears on its own.

Argo CD applies objects in an order based on kind (Namespace, CRDs, ConfigMaps, Secrets and finally the workloads). When you need your own ordering, you use waves: argocd.argoproj.io/sync-wave: "-5". They run from lowest to highest, and Argo CD waits for every object in a wave to be healthy before moving to the next.

Hooks are the equivalent of Helm's (10-03): PreSync, Sync, PostSync, SyncFail and Skip.

The deletion policies are HookSucceeded, HookFailed (do not use it: you lose the logs) and BeforeHookCreation (recommended). And a PostSync with a curl --fail against /health/ready is an excellent smoke test: if it fails, the application ends up Degraded and Argo CD tells you even though the pods started.

Argo CD handles two independent statuses. Confusing them is the most common conceptual mistake.

• Sync status — does the cluster match Git?: Synced, OutOfSync, Unknown.
• Health status — is the application working?: Healthy, Progressing, Degraded, Suspended, Missing, Unknown.

They are orthogonal. The four combinations that matter:

Argo CD knows how to evaluate the health of the standard kinds (Deployment, Service, Ingress, StatefulSet, Job, PVC). For CRDs you write your own evaluators in Lua, in the argocd-cm ConfigMap:

Careful with argocd app rollback: it reverts the cluster, but it does not revert Git. With selfHeal on, within three minutes the agent will return the cluster to whatever Git says and your rollback will vanish. Argo CD's rollback is an emergency measure; the real rollback in GitOps is a git revert followed by a pull request, and Argo CD syncs on its own.

Flux has a different philosophy: instead of one application with a web interface, it is a set of specialised controllers, each with its own CRD.

That bootstrap is an interesting design decision: Flux installs itself via GitOps. It writes its own manifests into the repository and is then managed from there. Upgrading Flux means making a commit.

For third-party charts there are HelmRepository and HelmRelease, and unlike Argo CD, Flux does create real Helm releases (they show up in helm list), with install.remediation and upgrade.remediation as the equivalent of --atomic.

A capability Flux has out of the box and Argo CD only through a separate project:

And in the manifest you mark which field to update: newTag: 2.4.1 # {"$imagepolicy": "flux-system:bookings-api:tag"}. Flux detects a new tag in the registry, makes a commit in Git updating it, and the normal GitOps cycle deploys it. The state stays in Git: the automation writes to Git, not to the cluster.

Choose Argo CD if you want development and support to be able to see the state of the deployment without learning kubectl (the web interface is a communication tool, not a whim), you manage several clusters from one central point, or you need the flexibility of ApplicationSet.

Choose Flux if you prefer a lightweight tool driven by CLI and Git, you want image automation out of the box, you need real Helm releases, or you have many small autonomous clusters.

For Rutas Norte we choose Argo CD, for two concrete reasons: the support team needs to see the state of deployments without being a Kubernetes expert, and ApplicationSet generates the three environments from one definition. Neither choice is irreversible: both read the same Kustomize manifests and the same Helm charts, so switching affects the files in k8s-gitops/, not k8s/.

Here comes the objection that always turns up: if everything goes into Git, where do I put the bookings-postgres password?

In 03-02 we said that Secrets are base64-encoded, not encrypted, and that version-controlling them requires a specific tool. The moment has arrived. And remember: a secret that has been in Git is compromised for ever, even if you commit to delete it. The history keeps it, and anybody who has cloned the repository has it.

SOPS encrypts the values of a YAML file while leaving the keys readable, using a key from a KMS, age or PGP.

An enormous advantage: the diffs stay readable, you can see which key changed even if not its value. Flux has native support (decryption: { provider: sops, secretRef: ... }); in Argo CD you need a plugin or ksops, which is a point against it.

A controller in the cluster generates a key pair. You encrypt with the public one; only that cluster can decrypt.

The resulting SealedSecret, with its encryptedData, is safe in a public repository. Trade-offs: the diff tells you nothing (the whole block changes even if you change one letter), the secret is tied to a specific namespace and name, and you have to back up the controller's master key or you will lose the ability to decrypt if you recreate the cluster.

Conceptually the cleanest: the secret is not in Git in any form, not even encrypted. Git holds only a reference.

Recommendation for Rutas Norte: External Secrets Operator with Vault. The platform handles customers' personal data and payment gateway credentials; automatic rotation and access auditing are not luxuries. Sealed Secrets is a reasonable alternative for getting started without extra infrastructure.

One rule does not change with any of the three: if a secret has reached Git in the clear, it has to be rotated. Always.

And so we reach the problem we flagged back in 09-01.

bookings-api has an HPA that adjusts spec.replicas between 4 and 20 depending on load. The manifest in Git says replicas: 4. When the May bank-holiday weekend arrives, the HPA raises it to 15. With nothing else in place:

Argo CD has dropped it to 4 replicas at peak hour. Thirty seconds later, the HPA raises them again. And three minutes after that, Argo CD lowers them once more. It is a tug of war between two controllers, and the one who suffers is the customer trying to buy a ticket.

The solution has two parts. First: do not declare replicas in the manifest, as we already did in 10-04. Second: ignoreDifferences, because even though the field is not in Git, Argo CD compares the live object with the generated one and spots it anyway.

Flux solves it differently: by removing the field from the desired object before applying, with a patch using op: remove on /spec/replicas, so that server-side apply does not claim ownership of it and the HPA is left free.

Operational tip: when an application shows up as OutOfSync persistently and you cannot see why, argocd app diff tells you in two seconds. It is nearly always a field from this list.

Five reasons to separate code and manifests:

The pipeline's final step, which only touches dev:

The digest is promoted, not the tag. It is the guarantee that what was tested in pre is bit for bit what goes to pro. The pull request that promotes to production is literally copying three lines from pre/ to pro/: a diff you can review in ten seconds.

That last row matters: reverting has to be fast. If the rollback procedure is as heavy as the deployment one, people will avoid reverting and fix things by hand, which is exactly what we set out to eliminate.

The mandatory checks on every PR are the ones from 10-04: kustomize build of the three overlays, kubeconform against the schema, kyverno apply of the policies (08-03) and argocd app diff posted automatically as a comment on the PR.

This is the anecdote that convinces the sceptics, and it happens in every company that adopts GitOps. Somebody from support, debugging a problem, runs kubectl delete deployment bookings-api -n rutas-norte-pro. Their blood runs cold.

Thirty-seven seconds. The pods are back, the configuration is exactly right, and there is a record of what happened.

That day the team understands that the repository is not a copy of the cluster's documentation: it is the cluster. The cluster is only the current projection, and it is replaceable. The definitive test, worth rehearsing once a year: destroy rutas-norte-pre, create a new cluster, install Argo CD and apply one Application, the root. Twenty minutes later it is fully rebuilt: namespaces, applications, policies, monitoring, certificates. The data is restored separately with Velero (05-06), but the platform rebuilds itself.

1. Forgetting the resources-finalizer.argocd.argoproj.io finalizer. Without it, deleting an Application leaves all of its objects orphaned in the cluster, with nobody managing them.

2. Enabling prune: true without understanding the scope. If you get path wrong and it points at an empty directory, Argo CD will delete everything it was managing. Protect yourself with allowEmpty: false and always test in dev first.

3. selfHeal fighting the HPA. The loop from section 12. Remove replicas from the manifest and add ignoreDifferences. Both.

4. targetRevision: HEAD or main in production. Any commit to main is deployed to rutas-norte-pro immediately. Use a tag or a pinned commit, and promote deliberately.

5. Putting secrets in the repository "for now, we will fix it later". It never gets fixed, and the secret stays in the history for ever. Set up SOPS, Sealed Secrets or External Secrets before the first deployment.

6. Giving Argo CD's credential write access to Git. It only needs to read.

7. Not configuring an AppProject and leaving everything in default. The default project allows deploying anything into any namespace from any repository: whoever can write to the manifest repository can escalate to cluster administrator.

8. Confusing Synced with Healthy. Synced + Degraded means the deployment worked and the application is broken: look at the pods, not the repository.

9. Using argocd app rollback believing it solves the problem. It reverts the cluster, not Git. With selfHeal, the bad state comes back within three minutes. The real rollback is git revert.

10. Everything in one giant Application. When something fails, the entire application goes Degraded and you cannot tell what. Split it by component or by layer, with waves for ordering.

11. Editing production "just this once, it is an emergency". It will revert itself. If the change is correct, the fast track is a PR with one approval, which takes two minutes. Have that procedure ready before you need it.

12. Not monitoring Argo CD itself. If the controller is down, nothing syncs and you do not find out. Alert on a sustained argocd_app_info{sync_status="OutOfSync"} and on the health of the argocd pods (07-04).

13. Sync waves without health checks. Argo CD moves to the next wave when the previous one is healthy; if an object has no health evaluator (a CRD with no resource.customizations), it moves on immediately and the ordering counts for nothing.

Write an ApplicationSet that generates the three Rutas Norte applications with these differences: dev syncs automatically from main without selfHeal; pre from main with selfHeal and pruning; pro from the release-2.4 tag with selfHeal, pruning and server-side apply. All three must ignore the Deployments' spec.replicas. Add the AppProject restricting deployment to the three namespaces and forbidding the creation of cluster-scoped resources.

rutas-norte-pro has been OutOfSync for two days even though nobody has touched anything. Describe the complete diagnostic procedure with the exact commands, identify at least three possible causes from what the course has covered, and write the ignoreDifferences that would resolve them.

Design the annotations needed for a deployment to happen in this order: (1) namespace and quotas, (2) ExternalSecrets and ConfigMaps, (3) the schema migration before anything else, (4) bookings-postgres and redis-cache, (5) bookings-api, (6) web-store, (7) Ingress, (8) HPA and PDB, and a final smoke test that marks the application as degraded if it fails. Explain what happens if the migration fails.

Solution 1

Solution 2

Solution 3

If the migration fails: the PreSync hook does not complete, so Argo CD aborts the sync without applying a single manifest. The cluster stays exactly as it was, serving the previous version. The application shows up as Sync Failed and the Job still exists (thanks to BeforeHookCreation), so kubectl logs job/schema-migration -n rutas-norte-pro gives you the reason. It is the same protective behaviour as Helm's hooks from 10-03.

GitOps closes the circle we opened at the start of this module. Rutas Norte has gone from 120 files applied by hand from a laptop to a system where the repository is the source of truth and the cluster converges towards it continuously.

• The four principles — declarative, versioned and immutable, pulled automatically, continuously reconciled — are the contract. The fourth is the one that changes everything.
• The pull model eliminates the most dangerous credential: ci-rutasnorte no longer needs access to the cluster, only write access to a Git repository, and the apiserver no longer has to be reachable from outside.
• Configuration drift is detected and corrected by itself. The cultural change it brings about — "the only way to change production is a pull request" — is worth more than the technology.
• Argo CD with its Application, its AppProjects limiting the blast radius, the app of apps pattern that reduces manual work to a single kubectl apply in the cluster's entire lifetime, and the ApplicationSets that generate the three environments from one definition. Waves and hooks order the deployment and protect the schema migration.
• Flux offers the same with specialised controllers, real Helm releases and built-in image automation. The choice is not irreversible: they read the same manifests.
• Secrets have three practical solutions: SOPS (readable diffs), Sealed Secrets (no extra infrastructure) and External Secrets Operator (the secret never reaches Git, with automatic rotation). For Rutas Norte, the third.
• And the conflict between the HPA and replicas we have been carrying since 09-01 is resolved: out of the manifest and ignoreDifferences in the Application, along with the whole family of fields other controllers modify.

One piece of the ecosystem remains. Everything we have built has to run somewhere, and in 10-02 we saw what it costs to operate your own cluster. In the next lesson, Managed Kubernetes: EKS, AKS and GKE, we will look at what a cloud provider saves you and what remains yours: the shared responsibility model, an honest comparison of the big three, the identity federation we left pending in 03-06, using spot instances for notifications-worker and occupancy-reports, and the final decision criteria for Rutas Norte.