We have protected the edge meticulously: a segmented network, a load balancer, a cache, minimal permissions and a patiently calibrated WAF. And all the while, the password of the app_catalogo user of the alpinashop-pedidos database is still exactly where we left it in 02-01: in an environment variable defined inside the startup-script of the MIG's instance template.

It is worth listing why that is security debt, and not simply a lack of elegance:

• It is readable by anybody with compute.instances.get. An instance's metadata, including the start-up script, is read with a gcloud compute instances describe. Dani, who only has a read role in production, can see the production password.
• It is in the Git history, because the template is generated from a versioned script. Deleting it from the file does not delete it from the repository.
• It cannot be rotated without redeploying. Changing the password requires a new template and a rolling update of the MIG. In practice, that means it is never rotated.
• It is duplicated. In the template, in the .env on Dani's laptop, in the development environment, probably in a chat message from eight months ago.
• It leaves no trail. There is no way of knowing who has read it or when.

This lesson solves that problem and also answers the question that comes right behind it: if Google stores my data, who encrypts it and with which key? You will see Secret Manager for application secrets and Cloud KMS for cryptographic material, and you will understand when it makes sense to take on the responsibility of managing your own keys and when it is a complication that buys no real security.

Warning. Encryption and secret management touch directly on regulatory obligations: GDPR, PCI DSS if payments are processed, and sector-specific data residency requirements. What follows is a technically correct teaching model, but any design destined for production with personal or payment data must be reviewed by a security and compliance professional before it is deployed. A failure here is not measured in minutes of downtime.

1. The concrete problem: the app_catalogo password
2. Secret Manager: the model of secrets and versions
3. Creating AlpinaShop's secrets
4. Accessing them from the Flask application
5. Permissions per secret: least privilege applied
6. Rotation: disable rather than destroy
7. Automatic versus regional replication: data residency
8. Integration with Cloud Run, GKE and Cloud Build
9. Encryption in Google Cloud: what happens by default
10. The key hierarchy: DEK and KEK
11. Google-managed, CMEK and CSEK: a decision table
12. Cloud KMS: key ring, keys and rotation
13. HSM and External Key Manager
14. Applying CMEK to the bucket and to Cloud SQL
15. Encryption in transit
16. What NEVER to do

This is how the MIG's template stands today, and this is not how it should stay:

See for yourself how exposed it is:

That command needs nothing more than compute.instances.get, a permission that forms part of roles/compute.viewer and that we granted to the gcp-desarrollo@ group in production (03-04). In other words: our own carefully built permission design is bypassed completely because of where the password is stored.

The destination is this, and the rest of the lesson explains each piece:

Secret Manager stores small strings or binaries — passwords, API keys, certificates, tokens — with IAM access control, encryption at rest, versioning and auditing.

The model has two levels, and confusing them is the source of almost every mistake:

Properties that follow from that model:

• Versions are immutable. You do not "update" a secret: you add a new version. Version 1 still exists.
• latest is a moving alias that always points at the most recent enabled version.
• A version can be in three states: enabled (it can be read), disabled (it exists but access fails, and it is reversible) and destroyed (the material is irreversibly deleted; only the metadata remains).
• Permissions are granted on the secret, not on the version.

Details that matter in these commands:

• --data-file=- with printf and no trailing newline. If you use echo, you add a \n to the value and the password the application reads is not the one you think. It is a classic bug that costs half an afternoon of debugging. If the secret is in a file, use --data-file=file and delete the file afterwards with shred -u.
• Never pass the value on the command line. There is a --data-file, but no --data: that is deliberate. A value on the command line stays in ~/.bash_history and in the process list.
• Label the secrets with the same convention as the rest of the resources (entorno, equipo, centro-coste, aplicacion). It serves for inventory and for periodic review.

Common operations:

The application authenticates with the VM's identity — the sa-catalogo-web service account — with no credential in the code at all. It is the pattern you already used with Cloud Storage in 02-02, now applied to secrets.

Three design decisions made explicit in that code:

1. It is read at start-up, not on every request. The cost and latency of a call per request are unacceptable.
2. latest is used, which means a restart automatically picks up the new version after a rotation. The alternative — pinning version 3 — is more deterministic but forces you to deploy code in order to rotate. For AlpinaShop, latest is the right choice; in a system where a wrong secret would be catastrophic, a pinned version has its argument.
3. The value is never logged. Not in a print, not in a debug log, not in an exception message. If the value reaches Cloud Logging (06-06), you have exposed it again, now somewhere that is retained and exported as well.

Here what you learned in 03-04 applies directly. The relevant roles:

The rule not to break: permissions are granted at secret level, never at project level. A roles/secretmanager.secretAccessor on alpinashop-prod would give access to every present and future secret, including the payment gateway key created next month. It is exactly the same reasoning we applied with the bucket and with the IAM conditions.

One detail that surprises people and is worth knowing: access to a secret is recorded in the data access audit logs, but those logs have to be enabled explicitly because they are not on by default. Without them you will not know who read what. Configuring audit logs is covered in 07-07; for a production secret, enable them.

Rotating means replacing the value with a new one. The correct procedure has one step that almost everybody skips, and it is the one that avoids an outage.

Step 4 is the key to this whole lesson in operational terms. disable is reversible: if something breaks, a gcloud secrets versions enable 1 brings the service back in seconds. destroy is irreversible. The rule: always disable first, destroy much later, and never both on the same day.

Secret Manager can also remind you that it is time to rotate:

There is a nuance here that is very often misread: Secret Manager rotates nothing. It publishes a message to a Pub/Sub topic when the date arrives. What rotates is your own automation — a Cloud Function subscribed to that topic (06-03) that generates the password, applies it in Cloud SQL, adds the version and triggers the MIG restart. Fully automatic rotation is a project in itself; the Pub/Sub reminder plus a written procedure is the minimum acceptable.

You can also set an expiry on a secret, useful for temporary credentials:

When you create a secret you choose where it is stored, and the decision is irreversible.

There is also the variant of regional secrets, created with --location=europe-west1, which live entirely in the regional control plane and offer the strictest isolation. They are accessed through a regional endpoint (secretmanager.europe-west1.rep.googleapis.com), which has to be taken into account in the code.

For AlpinaShop, whose billing and customer base are European, the reasonable choice is user-managed with European regions for the secrets tied to personal or payment data, and automatic for the rest. Which of the two the applicable regulatory framework demands is a question for the compliance officer, not for the platform team.

Outside VMs, the integration is even cleaner: the platform injects the secret and your code does not even call the API.

Cloud Run (which will be the catalogue's destination according to DA-001, in 07-02) offers both forms:

GKE Autopilot (the alpinashop-cluster cluster, tienda namespace) uses the Secret Manager add-on with the CSI driver, which mounts the secret as a volume:

This finally closes the warning we left in 02-05: a Kubernetes Secret is base64-encoded, which is not encryption. Anybody with read permission in the namespace decodes it in a second. With the CSI driver, the value does not exist as a Kubernetes object: it is fetched by Secret Manager, authenticated with Workload Identity, and only appears in the pod's file system.

Cloud Build (06-01) lets you use secrets in the pipeline without writing them into the configuration file:

We change subject, from application secrets to data encryption.

First of all, to remove unnecessary anxiety: all data at rest in Google Cloud is encrypted by default, always, without you doing anything and at no extra cost. Cloud Storage, persistent disks, Cloud SQL, BigQuery, Firestore: everything. There is no "enable encryption" checkbox because there is no way of disabling it.

So the right question is not "is my data encrypted?" but "who controls the key?". And that question is rarely answered by technical security: it is answered by regulatory compliance, by the sector and, sometimes, by the contract with a large customer.

Understanding this explains in one go why CMEK is cheap to enable and why rotating a key does not rewrite petabytes.

The mechanism, step by step:

1. The data is split into chunks and each chunk is encrypted with its own distinct DEK.
2. The DEK is not stored in the clear: it is encrypted ("wrapped") with a KEK and stored wrapped alongside the chunk.
3. The KEK lives in Cloud KMS and never leaves it. To decrypt, the service sends the wrapped DEK to KMS, which unwraps it and returns it; the plaintext DEK lives in memory for the bare minimum of time.

Consequences that answer very frequent questions:

• Rotating the KEK is instantaneous and rewrites not a single byte of data. It only changes the key that new DEKs are wrapped with; the old ones carry on being unwrapped with the old version, which is retained.
• Revoking access to the KEK renders the data unusable immediately. With no KEK there is no DEK, and with no DEK there is no data. This is at once CMEK's most powerful guarantee and its greatest operational danger.
• CMEK means replacing the KEK, not encrypting the data yourself. That is why enabling it has a negligible computational cost.

How to decide, without mysticism:

• Google-managed covers the vast majority of cases well. If nobody has asked you for anything else, this is the correct choice and it is not "less secure".
• CMEK when you need one of these three things: to be able to revoke access to the data unilaterally, to audit every use of the key, or to comply with a regulatory or contractual requirement that explicitly demands it. It adds a new failure mode — if the key is unavailable, the service does not start — and that has to be taken on consciously.
• CSEK is almost always the wrong answer. You take on the custody, the rotation and the availability of the key, and one mistake means definitive data loss. It exists for very specific requirements where the key cannot reside in the cloud under any circumstances.

For AlpinaShop, the reasoned decision: CMEK on alpinashop-catalogo and on the orders database, because they contain customer data and the company wants to be able to demonstrate control of the key to an auditor; Google-managed on alpinashop-dev, where there is no real data and adding a failure mode does not pay off.

The KMS hierarchy is: project → location → key ring (keyring) → key → key versions.

Four things to know about KMS that are not obvious:

1. The key's location must be compatible with the resource's. A key in europe-west1 for a bucket in europe-west1: correct. For an EU multi-region bucket you need a key in the europe location. If they do not match, the operation fails with an unclear error.
2. Key rings and keys cannot be deleted. Ever. Only their versions are destroyed. This is deliberate: it prevents an accidental deletion leaving data unrecoverable. The practical consequence: think about the naming before creating, because it will live forever.
3. Destroying a key version has a grace period, 30 days by default, during which it can be restored. It is the safety net that stops a mistake becoming a catastrophe.
4. Automatic rotation re-encrypts nothing. It creates a new version that becomes the primary one for future operations. The old versions are kept enabled so that what is already encrypted can be decrypted.

Permissions, again with a least-privilege criterion:

That last line is separation of duties applied to encryption: whoever administers the keys must not be whoever accesses the encrypted data, because in that case the encryption protects against nothing as far as that person is concerned.

KMS also serves to encrypt small pieces of data directly:

Although for passwords and API keys, Secret Manager is the right tool, not KMS. KMS is for keys; Secret Manager is for secrets. Internally, Secret Manager already uses KMS.

The protection level determines where the key material physically lives:

External Key Manager (EKM) is the extreme case: the key lives in an external manager (Thales, Fortanix, Equinix and the like) and Google calls that system for every cryptographic operation. If you cut off access, Google cannot decrypt your data, not even if it wanted to. It is the technical answer to "I want to be able to demonstrate that not even the cloud provider can read my data".

The price of that guarantee is high and it must be said plainly: you add an external dependency on the critical path to your data. If the EKM does not respond, your services do not start. For a small business like AlpinaShop, SOFTWARE with CMEK is the proportionate choice; HSM only if the payment gateway or a PCI audit demands it in writing.

Here is the concept most people get stuck on: the service agent. When Cloud Storage encrypts an object with your key, the one calling KMS is not you: it is an internal service account belonging to the service, managed by Google, which must have permission on your key.

The alpinashop-catalogo bucket:

One essential nuance: this only affects new objects. The 60 GB already uploaded remain encrypted with Google's key. To migrate them they have to be rewritten:

Cloud SQL alpinashop-pedidos: here there is a restriction that changes the work plan and that is worth knowing before promising anything.

CMEK on Cloud SQL can only be set when the instance is created. It cannot be added to an existing instance. Migrating alpinashop-pedidos to CMEK means creating a new instance and migrating the data, with the maintenance window that implies.

The migration would be done with the tools from 02-03: backup, restore into the new instance, synchronisation and changing the connection string. AlpinaShop's decision is to do it outside the autumn campaign, not before it.

And the operational warning that closes the section, because it is CMEK's real risk:

If you disable or destroy the key version, the bucket stops serving objects and the database stops starting, immediately. That is not a theoretical warning: it is exactly the designed behaviour. Before touching a key in production, check with gcloud logging read what is using it, and have a written recovery procedure.

Less configurable, but worth knowing the complete map:

The point that does depend on you is the first one, and it is the subject of the next lesson. The second also deserves a decision: the traffic between the load balancer and the MIG travels over Google's private network inside alpinashop-vpc, which is reasonable, but for payment data the recommended practice is to encrypt that leg too by configuring the backend service with the HTTPS protocol.

A short list, with no nuance, because each point has caused a real breach at some company:

• Secrets in Git. Not in a .env, not in a config.py, not in a terraform.tfvars, not in a test file. Deleting them is not enough: they remain in the history. If it happens, you have to rotate the credential, not just rewrite the history. Use git-secrets or your platform's secret scanning to stop it at the door.
• Secrets in container images. An ENV DB_PASSWORD=... or a file copied in the Dockerfile stays in the image layer and is readable by anybody who pulls it from europe-west1-docker.pkg.dev/alpinashop-prod/alpinashop/catalogo.
• Secrets in instance metadata or variables. The problem this lesson started with.
• Secrets in the logs. An exception that prints the complete connection string leaves the password in Cloud Logging, where it is retained and sometimes exported to BigQuery.
• Secrets in Kubernetes Secrets with no additional encryption. Base64 is not encryption.
• Secrets by chat or email. They end up indexed and synchronised on devices you do not control.
• The same password in development and in production. The development environment has laxer permissions by design; compromising it must not compromise production.
• Downloaded service account JSON keys (03-04). They are secrets that never expire.

• Using echo instead of printf when creating a version. It adds a newline to the value and authentication fails with an unhelpful message.
• Granting secretAccessor at project level. It gives access to every secret, including future ones. Always at secret level.
• Reading the secret on every HTTP request. Latency, cost and quota. Cache it in memory and consciously accept that rotation requires a restart.
• Destroying a version without having disabled it first. disable is reversible; destroy is not. Never both on the same day.
• Believing that Secret Manager rotates secrets. It only warns via Pub/Sub. You schedule the rotation yourself.
• Choosing the replication policy without thinking. It is irreversible and can have data residency implications.
• Logging the secret's value. A debug print takes it to Cloud Logging, which is retained and exported.
• Forgetting the service agent when enabling CMEK. The error is a permission denied on the key, not on the bucket, and it is very misleading.
• Expecting CMEK to encrypt what already exists. It only applies to new data; the rest has to be rewritten.
• Promising CMEK on an existing Cloud SQL instance. It can only be done at creation; it implies a migration.
• Disabling a key version in production "to test it". The bucket stops serving and the database stops starting instantly.
• Confusing KMS with Secret Manager. KMS manages keys; Secret Manager manages secrets. For a password, Secret Manager.
• Putting keys in an HSM as a precaution. It costs considerably more per key version and only helps if there is a specific regulatory requirement.
• Tip: separate who administers the keys from who accesses the data. Without that separation, encryption protects against nobody relevant.
• Tip: name secrets by their function, not by their value. db-password-catalogo, not password-prod-2026.

Write the complete plan for removing the password from tpl-catalogo-v3's startup-script without an outage: commands, code changes, permissions, order of operations and verification. Include what to do with the current password, which has been exposed for eight months in the metadata and in the Git history.

For each of AlpinaShop's data sets, choose between Google-managed, CMEK or CSEK, and justify it in two lines:

1. The 60 GB of product images in alpinashop-catalogo.
2. The alpinashop-pedidos database, with names, addresses and the last four digits of card numbers.
3. The CSV exports for Lucía in gs://alpinashop-catalogo/exportaciones/.
4. The disks of the MIG instances, which only contain the application code.
5. A copy of the signed supplier contracts, which by contract "may not be accessible by the cloud provider".

On a Monday morning, the repository's secret scanning raises an alert: the payment gateway's sk_live_... key is in a commit from three weeks ago, in an integration test file. The repository is private, with access for the six people on the technical team and two external collaborators.

Write the response plan ordered by priority, and the measures to stop it happening again.

Solution 1

Order of operations, with no outage:

STEP 3 — Code change: the application switches to using the secretos.py module from section 4 and stops reading os.environ["DB_PASSWORD"]. It is important that start-up fails loudly if it cannot read the secret, rather than carrying on with an empty value.

What to do with the old password. It was readable for eight months by anybody with compute.viewer and it is in the Git history. It is considered compromised: rotating it is mandatory and it is step 0, not the last step. Rewriting the Git history is advisable but secondary; what removes the risk is that the value no longer works for anything. In addition, it is worth reviewing the Cloud SQL access logs for the period in case there were connections from unexpected origins.

Solution 2

1. Product images → Google-managed, or CMEK for consistency. They are neither personal nor confidential data: they are photos from a public catalogue. Google-managed is enough. In AlpinaShop's case CMEK was applied for policy homogeneity and to be able to demonstrate control to an auditor, which is a valid argument provided the added failure mode is accepted.
2. alpinashop-pedidos → CMEK, without a doubt. It contains personal data subject to the GDPR and fragments of card data. Here the ability to revoke and to audit the use of the key is exactly what an audit asks for. With the known operational caveat: it requires creating a new instance and migrating.
3. CSV exports → CMEK, the same key as the orders. They are an extract of the same personal data. It would be inconsistent to protect the source and not the copy; in fact, exports are usually the weakest point in the chain.
4. MIG disks → Google-managed. They only contain code, which is also in Artifact Registry and in Git. CMEK here would only add a failure mode — if the key fails, the instances do not start — without protecting anything that is not already internally public.
5. Contracts with the inaccessibility clause → EKM (EXTERNAL), or rethink the requirement. It is exactly the case External Key Manager exists for: the key lives outside Google and without it Google cannot decrypt. CSEK would be technically possible in Cloud Storage, but manual custody of the key is too fragile for a contractual requirement. The honest alternative is to discuss with the provider whether the requirement is satisfied by CMEK plus access logs, which is considerably more operable.

Solution 3

Priority 1 — Invalidate the credential (minutes, not hours).

Priority 2 — Assess the impact. Review the last three weeks' operations in the gateway's dashboard looking for charges, refunds or queries from unusual origins. If there was misuse, there are notification obligations that depend on the applicable framework: that is decided by the compliance officer, not by the technical team.

Priority 3 — Clean up. Remove the secret from the Git history (git filter-repo or equivalent), force everybody on the team to update their copies and check that it was not left in derived artefacts: container images in Artifact Registry, Cloud Build logs, log exports.

Priority 4 — Stop it happening again.

The app_catalogo password has left the startup-script. You know why that place was unacceptable — readable with compute.viewer, present in the Git history, unrotatable without deploying, duplicated everywhere and with no traceability whatsoever — and you have built the complete alternative: the db-password-catalogo and api-key-pasarela-pago secrets in Secret Manager, with its model of a container secret and immutable versions, the latest alias, and a version's three states.

You have mastered access from the Flask application with the sa-catalogo-web identity and not a single credential in the code, with an in-memory cache and an awareness of what that cache implies for rotation. You have granted roles/secretmanager.secretAccessor at secret level and never at project level, and you know that the record of who reads each secret exists but has to be enabled. You know the five-step rotation procedure, with the rule that avoids catastrophes: always disable before destroying, and never on the same day; and you know that Secret Manager rotates nothing, it only warns via Pub/Sub and you write the automation yourself. You have chosen between automatic and regional replication understanding that it is irreversible and that data residency is a compliance question. And you have seen the integrations that make this cleaner still: Cloud Run with --set-secrets, with a mounted file better than an environment variable; the CSI driver on GKE, which closes the 02-05 warning about the base64 of Kubernetes Secrets; and availableSecrets in Cloud Build.

On encryption, you have understood the first thing that has to be understood: everything is encrypted at rest by default, so the real question is who controls the key. You know the DEK/KEK hierarchy, which explains why rotating is instantaneous, why revoking renders the data unusable straight away and why CMEK means replacing the KEK and not encrypting anything yourself. You have the decision table between Google-managed, CMEK and CSEK, with criteria rather than dogma: CMEK when you need to revoke, audit or comply; CSEK hardly ever. You have created the alpinashop-keyring key ring and the clave-catalogo and clave-pedidos keys with 90-day rotation, knowing that key rings and keys are never deleted and that the location must be compatible with the resource's. You have applied CMEK to the bucket by granting permission to the service agent — the step everybody forgets — you have discovered that it only affects new objects, and you have learned the restriction that changes the plans: CMEK on Cloud SQL can only be set when the instance is created. And you are clear about the HSM and EXTERNAL levels, with the real price of sovereignty: an external dependency on the critical path to your data.

Only one leg remains to be protected, and it is the most visible of all: the one from the customer's browser to alpinashop-lb-ip. Today AlpinaShop is served from a numeric IP address with no certificate; nobody buys a 200-euro backpack from a site with a crossed-out padlock. In the next lesson, 03-07, Cloud DNS, TLS Certificates and Publishing Services Securely, we close the module: we will create the public zone for alpinashop.example, point the domain at the global IP, provision a Google-managed certificate — with its DNS requirements and that wait in PROVISIONING that drives everybody to despair — tune the SSL policy and the application's security headers, and go over everything we have built in module 3 with a checklist.