Your system works. You have seen it work. You have created a booking, it reached BigQuery, the dashboard shows it and the certificate is valid.

And yet it is not finished, because there is an enormous difference between "I have seen it work" and "I know it works". The first statement rests on a one-off observation, made by you, under the best possible conditions: no concurrency, no errors, nothing falling over, with data you put in yourself. The second rests on tests.

This lesson answers one specific question: what does "finished" really mean? And the answer has seven parts, each of which answers a question somebody is going to ask you:

Question Answered by
"What if two people do the same thing at the same time?" The code tests
"Can you bring this back up from scratch?" The infrastructure tests
"Is anything exposed that should not be?" The security tests
"How much traffic does it take?" The load test
"What happens if the database goes down?" The reliability tests
"How do you deploy without breaking anything?" The deployment strategy
"And if something goes wrong?" The rollback procedure and the incident runbook

None of those seven is optional in a project you want to show off. And they can all be done over a long weekend, with free tools and without spending more than a couple of euros.

By the end, your project will not be "done": it will be launched, with evidence for every claim you make about it.

Contents

  1. What "finished" really means
  2. The test pyramid applied to this project
  3. Infrastructure tests
  4. Security tests
  5. Load tests
  6. Reliability tests
  7. The deployment strategy
  8. The pre-launch checklist
  9. The launch and the first 24 hours
  10. When something goes wrong: the incident runbook
  11. The worked example of RefugioReserva

  1. What "finished" really means

There are three levels of "finished", and it is worth knowing which one you are at:

Level Claim Evidence Typical score
Works on my machine "I have seen it work" A screenshot 40-55
Works deployed "It is in production and it responds" A URL 55-70
I know it works "Here are the tests, the measurements and what happens when it fails" Automated tests, measured numbers, drills carried out 75-95

The difference between the second and the third is this lesson. And it is where half the points live that separate an ordinary project from one an interviewer remembers.

The operational definition of "finished"

A deliverable is finished when you can tick the nine boxes:

  • [ ] There is at least one automated test that fails if you break the core logic
  • [ ] The infrastructure is recreated from scratch and the system starts up
  • [ ] The security checklist passes in full
  • [ ] You know your capacity limit as a measured number
  • [ ] You have deliberately switched off a dependency and you know what happens
  • [ ] You have restored a backup and timed it
  • [ ] You have triggered the alert and it arrived
  • [ ] You have rehearsed the rollback and you know how long it takes
  • [ ] The pre-launch checklist is complete

None of these needs paid tools. All of them need actually doing, not just ticking.

  1. The test pyramid applied to this project

flowchart TB
    E2E["End to end — 3-5 tests<br/>Against the real development environment<br/>Slow (min), expensive, brittle"]
    INT["Integration — 8-15 tests<br/>Against emulators or an ephemeral DB<br/>Medium (s)"]
    UNI["Unit — 20-40 tests<br/>Pure logic, no network or DB<br/>Fast (ms), free"]

    UNI --> INT --> E2E

    style UNI fill:#e6f4ea
    style INT fill:#fef7e0
    style E2E fill:#fce8e6

The proportion rule for a project of this size: around 30 unit tests, around 10 integration tests and 3-5 end-to-end tests. You do not need more, and chasing 100 % coverage is time badly invested in a portfolio.

What to test and what not to:

Do test Do not waste time testing
Business rules (capacity calculation, prices, states) That FastAPI returns 200 on a trivial route
Edge cases (zero, negative, maximum, empty) That the ORM knows how to do a SELECT
Concurrency at the critical point Getters and setters
Input validation That Terraform knows how to create a bucket
The complete flow, once Every permutation of the interface

2.1 Unit tests: the logic with no dependencies

# app/tests/test_unitarios.py
import pytest
from datetime import date
from src.dominio import calcular_disponibilidad, validar_reserva, ErrorValidacion

class TestDisponibilidad:
    """The core business rule: never more bookings than capacity."""

    def test_refugio_vacio_ofrece_toda_la_capacidad(self):
        assert calcular_disponibilidad(capacidad=40, ocupadas=0) == 40

    def test_descuenta_las_plazas_ocupadas(self):
        assert calcular_disponibilidad(capacidad=40, ocupadas=15) == 25

    def test_refugio_lleno_ofrece_cero(self):
        assert calcular_disponibilidad(capacidad=40, ocupadas=40) == 0

    def test_nunca_devuelve_negativo(self):
        """Edge case: if a bug left overbooking in the data, the function
        must return 0, not a negative number that the interface would
        display as '-3 places free'."""
        assert calcular_disponibilidad(capacidad=40, ocupadas=45) == 0

class TestValidacion:
    @pytest.mark.parametrize("plazas", [0, -1, 13, 999])
    def test_rechaza_plazas_fuera_de_rango(self, plazas):
        with pytest.raises(ErrorValidacion):
            validar_reserva(plazas=plazas, fecha=date(2026, 8, 15))

    def test_rechaza_fecha_pasada(self):
        with pytest.raises(ErrorValidacion, match="pasado"):
            validar_reserva(plazas=2, fecha=date(2020, 1, 1))

    @pytest.mark.parametrize("correo", ["", "sin-arroba", "a@", "@b.com"])
    def test_rechaza_correo_invalido(self, correo):
        with pytest.raises(ErrorValidacion):
            validar_reserva(plazas=2, fecha=date(2026, 8, 15), email=correo)

test_nunca_devuelve_negativo is the kind of test that earns its keep. It does not test the normal case — everyday use tests that — it tests the odd case that, when it happens in production at three in the morning, will produce absurd behaviour in the interface.

2.2 Integration tests: against real services or emulators

Google publishes local emulators for Firestore, Pub/Sub, Bigtable and Datastore. They are free, they start in seconds and they behave like the real service in everything that matters.

# Local emulators, free and without touching the cloud
gcloud emulators firestore start --host-port=localhost:8080
gcloud emulators pubsub    start --host-port=localhost:8085

# The client libraries detect them through an environment variable
export FIRESTORE_EMULATOR_HOST=localhost:8080
export PUBSUB_EMULATOR_HOST=localhost:8085

There is no emulator for PostgreSQL, but there is something better: an ephemeral container.

# app/tests/conftest.py
import pytest, subprocess, time, os
import psycopg

@pytest.fixture(scope="session")
def bd_efimera():
    """PostgreSQL in a container, created and destroyed by the test session.
    Cost: €0. Does not touch the cloud."""
    subprocess.run([
        "docker", "run", "-d", "--name", "pg-pruebas",
        "-e", "POSTGRES_PASSWORD=pruebas",
        "-e", "POSTGRES_DB=reservas",
        "-p", "55432:5432", "postgres:16-alpine",
    ], check=True)

    dsn = "postgresql://postgres:pruebas@localhost:55432/reservas"
    for _ in range(30):                       # wait until it accepts connections
        try:
            psycopg.connect(dsn).close()
            break
        except Exception:
            time.sleep(1)

    # Apply the SAME migrations as in production
    with psycopg.connect(dsn) as con:
        for f in sorted(os.listdir("data/migraciones")):
            with open(f"data/migraciones/{f}") as fh:
                con.execute(fh.read())
        con.commit()

    yield dsn

    subprocess.run(["docker", "rm", "-f", "pg-pruebas"], check=True)

Applying the same migrations as in production is what gives the test its value: if a migration is broken, you find out here and not at deployment time.

And the test that really matters in RefugioReserva — concurrency:

# app/tests/test_integracion.py
import pytest, psycopg
from concurrent.futures import ThreadPoolExecutor
from src.repositorio import crear_reserva, ErrorSinPlazas

def test_no_hay_sobreventa_con_concurrencia(bd_efimera):
    """The case that breaks badly built booking systems:
    twenty people try to book the last place at the same time."""
    with psycopg.connect(bd_efimera) as con:
        con.execute("INSERT INTO refugio (nombre, altitud_m, capacidad) "
                    "VALUES ('Refugio Prueba', 2000, 1)")
        con.commit()

    def intentar():
        try:
            with psycopg.connect(bd_efimera) as c:
                crear_reserva(c, refugio_id=1, fecha="2026-08-15", plazas=1,
                              titular="Fictitious", email="[email protected]")
            return "ok"
        except ErrorSinPlazas:
            return "sin_plazas"

    with ThreadPoolExecutor(max_workers=20) as ex:
        resultados = list(ex.map(lambda _: intentar(), range(20)))

    # EXACTLY one must succeed. Not zero, not two.
    assert resultados.count("ok") == 1, f"Overbooking: {resultados.count('ok')} bookings"
    assert resultados.count("sin_plazas") == 19

    with psycopg.connect(bd_efimera) as con:
        total = con.execute("SELECT COALESCE(SUM(plazas),0) FROM reserva "
                            "WHERE refugio_id=1 AND estado='confirmada'").fetchone()[0]
    assert total == 1

This test on its own justifies ADR-002 (choosing Cloud SQL for its transactions). And if you run it against a naive implementation — read availability, decide, insert, with no lock — it fails, which is exactly what it should do. The correct implementation uses SELECT ... FOR UPDATE on the refuge row:

def crear_reserva(con, refugio_id, fecha, plazas, titular, email):
    with con.transaction():
        # The row lock serialises the concurrent attempts
        cap = con.execute(
            "SELECT capacidad FROM refugio WHERE id = %s FOR UPDATE",
            (refugio_id,)).fetchone()[0]
        ocupadas = con.execute(
            "SELECT COALESCE(SUM(plazas),0) FROM reserva "
            "WHERE refugio_id=%s AND fecha=%s AND estado='confirmada'",
            (refugio_id, fecha)).fetchone()[0]
        if ocupadas + plazas > cap:
            raise ErrorSinPlazas()
        con.execute("INSERT INTO reserva (refugio_id, fecha, plazas, "
                    "nombre_titular, email_titular) VALUES (%s,%s,%s,%s,%s)",
                    (refugio_id, fecha, plazas, titular, email))

2.3 End-to-end tests

Few, slow and against the real development environment. Three or four are enough:

# app/tests/test_e2e.py
import os, time, uuid, pytest, requests
from google.cloud import bigquery

BASE = os.environ["URL_DEV"]

@pytest.mark.e2e
def test_flujo_completo_reserva_llega_a_analitica():
    """The whole critical path: book → event → BigQuery."""
    marca = str(uuid.uuid4())[:8]
    r = requests.post(f"{BASE}/api/reservas", timeout=30, json={
        "refugio_id": 1, "fecha": "2027-01-15", "plazas": 1,
        "nombre_titular": f"E2E {marca}",          # FICTITIOUS data
        "email_titular": f"e2e-{marca}@example.com",
    })
    assert r.status_code == 201
    reserva_id = r.json()["id"]

    # Propagation through Pub/Sub is not instant: poll with a limit
    bq = bigquery.Client()
    consulta = """
      SELECT COUNT(*) AS n
      FROM `refugio-datos.refugio_analitica.reservas_eventos`
      WHERE reserva_id = @rid AND DATE(ocurrido_en) = CURRENT_DATE()
    """
    cfg = bigquery.QueryJobConfig(query_parameters=[
        bigquery.ScalarQueryParameter("rid", "STRING", reserva_id)])

    for _ in range(12):                     # up to 2 minutes
        n = list(bq.query(consulta, job_config=cfg).result())[0].n
        if n == 1:
            return
        time.sleep(10)
    pytest.fail("The event did not reach BigQuery within 2 minutes")

@pytest.mark.e2e
def test_sondas_de_salud_responden():
    assert requests.get(f"{BASE}/salud/vivo", timeout=10).status_code == 200
    assert requests.get(f"{BASE}/salud/arranque", timeout=10).status_code == 200

2.4 Integration into Cloud Build

steps:
  - id: unitarias-e-integracion
    name: python:3.12-slim
    entrypoint: bash
    args:
      - -c
      - |
        pip install --no-cache-dir -r app/requirements.txt -r app/requirements-dev.txt
        cd app && python -m pytest tests/ -v -m "not e2e" --tb=short \
          --junitxml=/workspace/resultados.xml

  # ... build, publish, deploy ...

  - id: e2e
    name: python:3.12-slim
    entrypoint: bash
    args:
      - -c
      - |
        pip install --no-cache-dir -r app/requirements-dev.txt
        export URL_DEV=$(gcloud run services describe refugio-web \
          --region=europe-west1 --format='value(status.url)')
        cd app && python -m pytest tests/ -v -m e2e
    waitFor: [desplegar]

The unit and integration tests go before building; the end-to-end tests after deploying, because they need a live system.

  1. Infrastructure tests

3.1 Static validation

terraform fmt -check -recursive        # consistent formatting
terraform validate                     # syntax and references
terraform plan -detailed-exitcode      # 0=no changes, 2=changes, 1=error

-detailed-exitcode is useful in CI: it lets you fail the build if the deployed state does not match the code.

3.2 The plan reviewed in the pull request

# .github/workflows/plan.yml
name: Terraform plan
on: pull_request

permissions:
  contents: read
  id-token: write
  pull-requests: write

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.SA_PLAN }}      # READ-ONLY SA
      - uses: hashicorp/setup-terraform@v3
      - run: terraform -chdir=infra/envs/dev init
      - id: plan
        run: terraform -chdir=infra/envs/dev plan -no-color -out=plan.tfplan
      - name: Publish the plan as a comment
        uses: actions/github-script@v7
        with:
          script: |
            const salida = `#### Terraform plan
            \`\`\`
            ${{ steps.plan.outputs.stdout }}
            \`\`\``;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner, repo: context.repo.repo, body: salida
            });

Making the plan account read-only (roles/viewer + access to the state bucket) matters: a plan must not be able to change anything, and that way neither can a malicious pull request.

3.3 Policy analysis: tflint and checkov

# tflint: syntax errors, invalid arguments, provider best practices
tflint --init && tflint --recursive

# checkov: security and compliance checks over the code
checkov -d infra/ --framework terraform --compact

Examples of what they catch and that may well have slipped past you:

Tool Catches
tflint Non-existent machine type, deprecated attribute, unused variable
checkov Bucket without uniform access, DB without enforced SSL, logs disabled, public service with no justification

Do not chase zero findings. Some will be false positives for your context (for example, Cloud SQL not having high availability is deliberate in a portfolio). The right move is to document the exception:

# checkov:skip=CKV_GCP_79:High availability not applicable; portfolio project
# with a €12/month budget. See docs/adr/ADR-002.

Add it to the pipeline as an informational, non-blocking step, right at the start:

  - id: analisis-iac
    name: bridgecrew/checkov:latest
    args: ["-d", "infra/", "--compact", "--soft-fail"]

3.4 The definitive test: recreate the environment from scratch

This is the test that separates a project with Terraform from a project with real infrastructure as code. It is worth 4 points of the rubric and it is the one most people take for granted without ever having done it.

#!/usr/bin/env bash
# scripts/prueba-reconstruccion.sh
set -euo pipefail

ENTORNO="dev"
INICIO=$(date +%s)

echo "=== 1. Initial state ==="
terraform -chdir="infra/envs/${ENTORNO}" state list | wc -l

echo "=== 2. DESTROY ==="
terraform -chdir="infra/envs/${ENTORNO}" destroy -auto-approve

echo "=== 3. Check that nothing is left ==="
gcloud run services list      --project="refugio-${ENTORNO}" --format="value(name)"
gcloud sql instances list     --project="refugio-${ENTORNO}" --format="value(name)"
gcloud compute networks list  --project="refugio-${ENTORNO}" --format="value(name)"

echo "=== 4. REBUILD ==="
terraform -chdir="infra/envs/${ENTORNO}" apply -auto-approve

echo "=== 5. Migrations and data ==="
./scripts/migrar.sh "${ENTORNO}"
python data/seed/generar.py --entorno="${ENTORNO}"

echo "=== 6. Deploy the last known image ==="
./scripts/desplegar.sh "${ENTORNO}" "$(git rev-parse --short HEAD)"

echo "=== 7. Check that it works ==="
URL=$(gcloud run services describe refugio-web --region=europe-west1 \
      --project="refugio-${ENTORNO}" --format='value(status.url)')
CODIGO=$(curl -s -o /dev/null -w '%{http_code}' "${URL}/salud/arranque")
test "${CODIGO}" = "200" || { echo "FAILURE: the app does not respond"; exit 1; }

echo "=== 8. End-to-end test ==="
URL_DEV="${URL}" python -m pytest app/tests/ -m e2e -q

FIN=$(date +%s)
echo "✅ FULL REBUILD in $(( (FIN-INICIO)/60 )) minutes"

What this test uncovers, and no other one does:

Typical finding Why you cannot see it any other way
A resource created by hand in week 2 In an environment that already exists, you never miss it
Implicit dependencies in the wrong order In an incremental apply, the earlier resource was already there
A secret you filled in by hand The code creates it empty and nobody notices
Migrations that only work against the existing DB They are never run from scratch
A permission granted from the console The app works and you do not know why

Do it at least twice: once halfway through the project (to find the gaps while they are still cheap to fix) and once before the launch. And time it: the rebuild time is a number that looks very good in the presentation.

  1. Security tests

4.1 Image scanning

Artifact Registry includes vulnerability analysis:

gcloud artifacts docker images scan \
  "europe-west1-docker.pkg.dev/refugio-dev/refugio-imagenes/web:latest" \
  --format="value(response.scan)"

# And to list the findings
gcloud artifacts docker images list-vulnerabilities <SCAN_RESULT> \
  --format="table(vulnerability.effectiveSeverity, vulnerability.shortDescription)"

Realistic criterion: zero critical vulnerabilities and zero exploitable high ones in your code. Those in the base image are fixed by updating the base (python:3.12-slim to its latest version) and rebuilding. Medium and low ones in system dependencies you do not use are documented and accepted.

4.2 The executable checklist

This script is a deliverable in its own right. Save it as scripts/auditoria-seguridad.sh, run it, and save the output in docs/evidencias/:

#!/usr/bin/env bash
# scripts/auditoria-seguridad.sh
set -uo pipefail
PROY="${1:?Usage: $0 <project>}"
FALLOS=0
ok()   { echo "  ✅ $1"; }
falla(){ echo "  ❌ $1"; FALLOS=$((FALLOS+1)); }
avisa(){ echo "  ⚠️  $1"; }

echo "=== SECURITY AUDIT: ${PROY} ==="

echo "[1] Primitive roles on service accounts"
N=$(gcloud projects get-iam-policy "$PROY" --format=json |
    jq '[.bindings[] | select(.role|test("roles/(owner|editor)")) |
         .members[] | select(startswith("serviceAccount:"))] | length')
[ "$N" -eq 0 ] && ok "No SA with owner/editor" || falla "$N accounts with a primitive role"

echo "[2] User-managed service account keys"
TOTAL=0
for SA in $(gcloud iam service-accounts list --project="$PROY" --format="value(email)"); do
  K=$(gcloud iam service-accounts keys list --iam-account="$SA" --managed-by=user \
      --format="value(name)" 2>/dev/null | wc -l)
  TOTAL=$((TOTAL+K))
  [ "$K" -gt 0 ] && echo "     · $SA has $K key(s)"
done
[ "$TOTAL" -eq 0 ] && ok "Zero JSON keys" || falla "$TOTAL downloadable keys"

echo "[3] Publicly accessible buckets"
PUB=0
for B in $(gcloud storage buckets list --project="$PROY" --format="value(name)"); do
  if gcloud storage buckets get-iam-policy "gs://$B" --format=json |
     jq -e '.bindings[]?.members[]? | select(. == "allUsers" or . == "allAuthenticatedUsers")' >/dev/null 2>&1; then
    echo "     · gs://$B is PUBLIC"; PUB=$((PUB+1))
  fi
done
[ "$PUB" -eq 0 ] && ok "No public bucket" || falla "$PUB public bucket(s)"

echo "[4] Databases with a public IP"
for I in $(gcloud sql instances list --project="$PROY" --format="value(name)"); do
  IPV4=$(gcloud sql instances describe "$I" --project="$PROY" \
         --format="value(settings.ipConfiguration.ipv4Enabled)")
  [ "$IPV4" = "False" ] && ok "$I has no public IP" || falla "$I HAS A PUBLIC IP"
done

echo "[5] Secrets in the repository"
if git log -p --all 2>/dev/null | grep -Ei '(password|api[_-]?key|secret|BEGIN (RSA|PRIVATE))' \
   | grep -v 'secret_id\|secretAccessor\|SecretManager\|secret_key_ref\|refugio-db-password' \
   | head -5 | grep -q .; then
  falla "Possible secrets in the git history — REVIEW"
else
  ok "No obvious secrets in the history"
fi

echo "[6] Credential files in the tree"
if find . -name "*.json" -not -path "./node_modules/*" -not -path "./.git/*" \
   -exec grep -l '"type": *"service_account"' {} \; 2>/dev/null | grep -q .; then
  falla "There are service account key files in the repository"
else
  ok "No credential files"
fi

echo "[7] Cloud Run services without authentication"
for S in $(gcloud run services list --project="$PROY" --region=europe-west1 --format="value(name)"); do
  if gcloud run services get-iam-policy "$S" --project="$PROY" --region=europe-west1 \
     --format=json | jq -e '.bindings[]?.members[]? | select(. == "allUsers")' >/dev/null 2>&1; then
    avisa "$S is public (correct if it is the web app; review otherwise)"
  else
    ok "$S requires authentication"
  fi
done

echo "[8] Admin activity audit logs"
gcloud logging read 'logName:"cloudaudit.googleapis.com%2Factivity"' \
  --project="$PROY" --limit=1 --format="value(timestamp)" | grep -q . \
  && ok "Audit logs present" || avisa "No recent activity recorded"

echo
echo "=== RESULT: ${FALLOS} failure(s) ==="
exit $((FALLOS > 0))

4.3 Checking TLS and headers

DOMINIO="refugioreserva.example"

# Certificate: issuer and validity
echo | openssl s_client -connect "${DOMINIO}:443" -servername "${DOMINIO}" 2>/dev/null |
  openssl x509 -noout -subject -issuer -dates

# TLS versions: 1.2 and 1.3 yes; 1.0 and 1.1 must fail
for V in tls1 tls1_1 tls1_2 tls1_3; do
  printf "%-8s " "$V"
  echo | openssl s_client -"$V" -connect "${DOMINIO}:443" 2>/dev/null | \
    grep -q "Verify return code: 0" && echo "accepts" || echo "rejects"
done

# Security headers
curl -sI "https://${DOMINIO}" | grep -iE \
  'strict-transport|x-content-type|x-frame|content-security|referrer-policy'

# HTTP must redirect
curl -sI "http://${DOMINIO}" | head -1
Header Recommended value What it prevents
Strict-Transport-Security max-age=31536000; includeSubDomains Downgrade to HTTP
X-Content-Type-Options nosniff Wrong type interpretation
X-Frame-Options DENY Clickjacking
Content-Security-Policy At least default-src 'self' Script injection
Referrer-Policy strict-origin-when-cross-origin URL leakage to third parties

These are added in an application middleware with ten lines of code, and they are worth 2 points.

4.4 Reviewing effective permissions

# What can my application service account actually do?
gcloud projects get-iam-policy refugio-prod --format=json |
  jq -r --arg sa "serviceAccount:[email protected]" \
     '.bindings[] | select(.members[]? == $sa) | .role'

# Check a specific permission it should NOT have
gcloud policy-troubleshoot iam \
  "//cloudresourcemanager.googleapis.com/projects/refugio-prod" \
  --principal-email="[email protected]" \
  --permission="resourcemanager.projects.setIamPolicy"
# Expected: NOT_GRANTED

Proving that it does not have a dangerous permission is as valuable as proving that it has the ones it needs.

  1. Load tests

5.1 How to run an honest, cheap test

Three rules, in order of importance:

  1. Against development, never against production, unless it is the launch itself and you have planned it.
  2. With a spending cap: --max-instances limited and a duration budget. A load test with no cap can scale to a hundred instances and eat the month's budget in twenty minutes.
  3. Start small and go up. 10 users, 50, 100. Stop when something breaks: that is the number you are after.

Free tools: hey, k6, locust, wrk. For this project, k6 is the most complete and hey the quickest to use.

// pruebas/carga.js  —  run with: k6 run pruebas/carga.js
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';

const erroresNegocio = new Rate('errores_negocio');

export const options = {
  stages: [
    { duration: '1m', target: 10 },   // warm-up
    { duration: '2m', target: 50 },   // expected load
    { duration: '2m', target: 100 },  // double that
    { duration: '1m', target: 0 },    // cool-down
  ],
  thresholds: {
    http_req_duration:  ['p(95)<1000', 'p(99)<2000'],
    http_req_failed:    ['rate<0.01'],
    errores_negocio:    ['rate<0.05'],
  },
};

const BASE = __ENV.URL;

export default function () {
  // 80% queries, 20% bookings: a realistic profile, not all writes
  if (Math.random() < 0.8) {
    const r = http.get(`${BASE}/api/disponibilidad?refugio_id=1&fecha=2027-01-15`);
    check(r, { 'query 200': (x) => x.status === 200 });
    erroresNegocio.add(r.status >= 500);
  } else {
    const r = http.post(`${BASE}/api/reservas`, JSON.stringify({
      refugio_id: Math.ceil(Math.random() * 12),
      fecha: '2027-01-15', plazas: 1,
      nombre_titular: 'Fictitious Load',
      email_titular: `carga-${__VU}-${__ITER}@example.com`,
    }), { headers: { 'Content-Type': 'application/json' } });
    // 409 (no places) is correct, not a system error
    check(r, { 'booking 201 or 409': (x) => x.status === 201 || x.status === 409 });
    erroresNegocio.add(r.status >= 500);
  }
  sleep(Math.random() * 2);
}

The detail that makes the test honest: distinguishing between a system error (5xx) and a legitimate business response (409, no places left). A test that counts the 409s as failures will give you a terrifying and false error rate.

5.2 What to measure

Metric Where it is measured What it tells you Reasonable threshold
p50 latency k6 The typical experience <300 ms
p95 latency k6 The bad but common experience <1,000 ms
p99 latency k6 The real worst case <2,000 ms
5xx error rate k6 + Monitoring Reliability <1 %
Active instances Monitoring Saturation and scaling Without reaching the maximum
Cold starts Cloud Run logs Added latency Visible in the p99
DB connections Cloud SQL Insights The usual bottleneck <80 % of the maximum
Cost of the test Billing That it does not ruin you <€1

Do not look only at the average. The average lies: with 99 requests of 100 ms and one of 10 seconds, the average is 200 ms and there is a user who waited ten seconds. Percentiles tell the truth.

5.3 How to interpret and what to adjust

Symptom Likely cause Adjustment
p99 far above the p95 Cold starts min-instances=1 (it costs money) or accept it and document it
Latency rises with load, no errors CPU saturation Raise CPU, or lower concurrency so it scales sooner
5xx errors as load rises DB connections exhausted A smaller connection pool per instance, or a lower max-instances
Throughput plateau with instances maxed out max-instances cap Raise it… and recalculate the cost
Errors from the very first minute A bug, not capacity Fix it before testing again

The connection arithmetic that catches everybody out: Cloud SQL db-f1-micro allows around 25 connections. If your application opens a pool of 10 connections per instance and Cloud Run scales to 5 instances, you ask for 50 connections and half of them fail. The solution is not a bigger database: it is a pool of 2-3 connections per instance, because each instance serves concurrent requests with a single connection most of the time.

  1. Reliability tests

Three drills. None takes more than an hour and all three are pure gold for the presentation.

6.1 Switch off a dependency and see what happens

# DRILL IN DEVELOPMENT, with the time recorded
echo "Drill start: $(date)" | tee -a docs/evidencias/ensayo-bd.log

# Stop the database
gcloud sql instances patch refugio-db --project=refugio-dev \
  --activation-policy=NEVER --quiet

# Observe the behaviour
curl -s -o /dev/null -w "Home page: %{http_code} in %{time_total}s\n"  "${URL}/"
curl -s -o /dev/null -w "API:       %{http_code} in %{time_total}s\n"  "${URL}/api/refugios"
curl -s -w "Health: %{http_code}\n" "${URL}/salud/arranque"
curl -s "${URL}/api/refugios" | head -c 300

# Restore
gcloud sql instances patch refugio-db --project=refugio-dev \
  --activation-policy=ALWAYS --quiet

What behaviour you are looking for:

Aspect ❌ Bad ✅ Good
Status code 500 with a Python traceback 503 with an explanatory JSON body
Message to the user psycopg.OperationalError: could not connect... "The service is temporarily unavailable"
Response time 30 s (waiting for the timeout) <3 s (short timeout in the DB client)
Unaffected parts Everything down Home page and cached content still served
Startup probe 200 (it lies) 503 (Cloud Run stops sending traffic)
Logs Exception traceback with no context Structured ERROR with motivo and no personal data

If your system behaves like the left-hand column, fix it: a short timeout and an exception handler that returns a 503 with a readable message are thirty lines of code and they add up in reliability and in the presentation.

6.2 Restore the backup and time it

INICIO=$(date +%s)

# 1. What backups do I have?
gcloud sql backups list --instance=refugio-db --project=refugio-dev \
  --format="table(id, windowStartTime, status)"

BACKUP_ID=$(gcloud sql backups list --instance=refugio-db --project=refugio-dev \
            --format="value(id)" --limit=1)

# 2. Restore onto a NEW instance (never onto the original in a drill)
gcloud sql instances create refugio-db-restaurada \
  --project=refugio-dev --region=europe-west1 \
  --database-version=POSTGRES_16 --tier=db-f1-micro

gcloud sql backups restore "${BACKUP_ID}" \
  --restore-instance=refugio-db-restaurada \
  --backup-instance=refugio-db --project=refugio-dev --quiet

# 3. VERIFY that the data is there (this is what almost nobody does)
./cloud-sql-proxy --port 55433 "refugio-dev:europe-west1:refugio-db-restaurada" &
psql -h 127.0.0.1 -p 55433 -U app -d reservas -c "
  SELECT (SELECT count(*) FROM reserva) AS reservas,
         (SELECT count(*) FROM refugio) AS refugios,
         (SELECT max(creada_en) FROM reserva) AS ultima;"

FIN=$(date +%s)
echo "Measured RTO: $(( (FIN-INICIO)/60 )) minutes"

# 4. CLEAN UP — do not leave the restored instance running
gcloud sql instances delete refugio-db-restaurada --project=refugio-dev --quiet

Step 3 is what separates a real drill from a fake one. A restore that "completes successfully" but leaves an empty database is a disaster dressed up as a success. And step 4 is not optional: a forgotten restored instance is the most common way of doubling the bill without noticing.

Write down three numbers: how long it took, how much data would have been lost (RPO: the distance between the last backup and the moment of failure) and what problems you found. AlpinaShop found five problems nobody suspected in its 47-minute drill; you will find something similar.

6.3 Trigger the alert

You already did this in 08-03. If not, now is the time. And check all three things:

  1. That the incident opens in Monitoring.
  2. That the email (or whatever channel it is) arrives.
  3. That the content includes your first steps and is actually usable for acting.

  1. The deployment strategy

7.1 Environments and promotion

flowchart LR
    DEV["Development<br/>refugio-dev<br/>Automatic on every push"]
    PRUEBA["Testing<br/>e2e + load<br/>Against dev"]
    PROD["Production<br/>refugio-prod<br/>With approval"]

    DEV --> PRUEBA
    PRUEBA -->|same image<br/>same digest| PROD

    PROD --> CAN["Canary 10%"]
    CAN -->|10 min OK| C50["50%"]
    C50 -->|10 min OK| C100["100%"]
    CAN -.->|error| REV["Rollback<br/>< 60 s"]
    C50 -.->|error| REV

The non-negotiable principle: the same image, identified by its digest.

# Get the EXACT digest of the image tested in dev
DIGEST=$(gcloud run services describe refugio-web --region=europe-west1 \
         --project=refugio-dev --format="value(spec.template.spec.containers[0].image)")
echo "Promoting: ${DIGEST}"

# Deploy to production WITHOUT traffic
gcloud run deploy refugio-web --image="${DIGEST}" \
  --region=europe-west1 --project=refugio-prod \
  --no-traffic --tag=candidata

Referencing by digest (web@sha256:abc...) and not by tag (web:v1.2) removes all ambiguity: tags can be reassigned; a digest is the content.

7.2 The canary, step by step

SERVICIO="refugio-web"; REGION="europe-west1"; PROY="refugio-prod"
NUEVA=$(gcloud run revisions list --service=$SERVICIO --region=$REGION \
        --project=$PROY --limit=1 --format="value(name)")

# Phase 1 — 10% for 10 minutes
gcloud run services update-traffic $SERVICIO --region=$REGION --project=$PROY \
  --to-revisions="${NUEVA}=10"

# Observe THAT SPECIFIC new revision, not the whole service
gcloud logging read \
  "resource.type=cloud_run_revision AND
   resource.labels.revision_name=${NUEVA} AND severity>=ERROR" \
  --project=$PROY --limit=20 --freshness=10m

Objective promotion criteria, decided beforehand and not on the spot:

Phase Traffic Minimum duration Promote if… Roll back if…
1 10 % 10 min 5xx < 0.5 % and p95 < 1.2× the baseline 5xx > 1 % or p95 > 2× the baseline
2 50 % 10 min Same Same
3 100 % 30 min of watching Same Same

And a golden rule: if in doubt, roll back. Rolling back costs a minute; an incident costs an afternoon.

7.3 The rollback, rehearsed

#!/usr/bin/env bash
# scripts/revertir.sh — tested on 2026-10-28, takes 38 seconds
set -euo pipefail
SERVICIO="${1:-refugio-web}"; REGION="europe-west1"; PROY="refugio-prod"

ANTERIOR=$(gcloud run revisions list --service="$SERVICIO" --region="$REGION" \
  --project="$PROY" --format="value(name)" --sort-by="~metadata.creationTimestamp" \
  | sed -n '2p')

echo "Rolling back ${SERVICIO} → ${ANTERIOR}"
gcloud run services update-traffic "$SERVICIO" --region="$REGION" --project="$PROY" \
  --to-revisions="${ANTERIOR}=100"

URL=$(gcloud run services describe "$SERVICIO" --region="$REGION" --project="$PROY" \
      --format='value(status.url)')
sleep 5
curl -s -o /dev/null -w "Check: %{http_code}\n" "${URL}/salud/arranque"
echo "✅ Rolled back to ${ANTERIOR}"

Rehearse it for real, with a stopwatch, at a quiet moment. A rollback procedure that has never been executed is a hypothesis, and the moment to discover that it does not work is not during an incident.

What a traffic rollback does NOT roll back — and you need to know this beforehand:

Change Rolled back with the traffic? What to do
Application code ✅ Yes Nothing else
Environment variables ✅ Yes (they live in the revision) Nothing else
Database migration No Backwards-compatible migrations
Data already written ❌ No Restore a backup (much slower)
Infrastructure changes ❌ No terraform apply of the previous commit

Hence the rule that avoids 90 % of disasters: migrations must be backwards compatible. Add columns, do not remove them. If a column has to go, do it in a later deployment, when no version in circulation uses it.

  1. The pre-launch checklist

You go through it in full, by hand, ticking for real.

Technical

  • [ ] All tests pass in CI (unit, integration, end-to-end)
  • [ ] terraform plan against production says No changes
  • [ ] The rebuild from scratch has been run successfully and is timed
  • [ ] The migrations are applied and are backwards compatible
  • [ ] The image deployed to production is the same one tested in development (identical digest)
  • [ ] The health probes respond correctly
  • [ ] There is no TODO, FIXME or debug print() on the critical path

Security

  • [ ] scripts/auditoria-seguridad.sh exits with 0 failures
  • [ ] The image has no critical vulnerabilities
  • [ ] Valid TLS; TLS 1.0/1.1 rejected; HTTP redirects to HTTPS
  • [ ] Security headers present
  • [ ] Zero JSON keys; zero primitive roles on service accounts
  • [ ] Database with no public IP; no public bucket without justification
  • [ ] The secrets are in Secret Manager and not in the repository or its history
  • [ ] The logs contain no personal data

Cost

  • [ ] The budget is created with alerts at 50/90/100 %
  • [ ] The projected cost fits inside the limit set in 08-01
  • [ ] max-instances is limited on every service (spending cap)
  • [ ] Nothing is left running from the tests (restored instances, test load balancers, load jobs)
  • [ ] The buckets have a lifecycle; Artifact Registry has a clean-up policy
  • [ ] The BigQuery tables have partition expiration

Observability

  • [ ] The dashboard shows the four signals with real data
  • [ ] At least one alert has actually notified you when triggered
  • [ ] The uptime check is active and green
  • [ ] The SLO computes an error budget with a numeric value
  • [ ] The logs are structured and correlatable by trace
  • [ ] The notification channels are verified

Documentation

  • [ ] The README lets another person start the project
  • [ ] docs/arquitectura.md reflects the system as it is today
  • [ ] There are ≥3 ADRs, and the diagrams match the decisions
  • [ ] docs/runbook.md has the operational procedures
  • [ ] The diario.md is up to date
  • [ ] The known technical debt is written down and prioritised

The minimum runbook

# Operations manual — RefugioReserva

## Contact details and access
Owner: <me>. Projects: refugio-dev, refugio-prod, refugio-datos.
Console: https://console.cloud.google.com/home/dashboard?project=refugio-prod

## P1 — The application returns 5xx errors
1. `gcloud logging read 'severity>=ERROR' --project=refugio-prod --limit=20 --freshness=15m`
2. Does it coincide with a deployment? `gcloud run revisions list --service=refugio-web --limit=3`
3. If it does → **roll back**: `./scripts/revertir.sh refugio-web` (38 s)
4. If not → check the DB: `gcloud sql instances describe refugio-db --format="value(state)"`
5. Write down what happened in `docs/diario.md`

## P2 — The application does not respond at all
1. `curl -sI https://refugioreserva.example`
2. Service alive? `gcloud run services describe refugio-web --format="value(status.conditions)"`
3. Does DNS resolve? `dig +short refugioreserva.example`
4. Valid certificate? `openssl s_client -connect refugioreserva.example:443`

## P3 — Restore the database
See `scripts/restaurar.sh`. **Measured RTO: 22 minutes. RPO: up to 24 h.**

## P4 — Runaway cost
1. Billing report → group by service and by the `componente` label
2. Usual suspects: forgotten instances, jobs in a loop, BigQuery queries with no partition filter
3. Immediate measure: lower `max-instances`; as a last resort, `terraform destroy` of dev

## P5 — Deploy a new version
`git push` to `main` → dev automatically → approve in Cloud Build → canary 10/50/100

  1. The launch and the first 24 hours

The launch

Pick a moment when you can be watching the screen for the following hour. Not a Friday night, and not just before going to bed.

# 1. Final check
./scripts/auditoria-seguridad.sh refugio-prod
terraform -chdir=infra/envs/prod plan          # No changes

# 2. Promote the tested image
./scripts/promocionar.sh "$(git rev-parse --short HEAD)"

# 3. Canary 10%, wait, verify
# 4. 50%, wait, verify
# 5. 100%
# 6. Mark the moment
git tag -a v1.0.0 -m "Final project launch"
git push --tags

The first 24 hours: what to watch and in what order

Moment What you watch What would make you roll back
0-15 min 5xx errors in the logs, in real time Any 5xx that did not exist before
15-60 min p95 latency, cold starts, instances p95 at twice the baseline
1-4 h Full dashboard, first alerts, DB connection usage Alerts firing
4-24 h Accumulated cost, error trend, SLO error budget Projected cost above the limit
24 h All of the above + a manual test of the complete flow
# The command you leave running in a terminal for the first hour
gcloud logging tail \
  'resource.type="cloud_run_revision" AND severity>=WARNING' \
  --project=refugio-prod

When to declare success. Not when the deployment finishes, but when the five conditions are met:

  1. 24 hours with no anomalous 5xx errors.
  2. p95 latency inside the SLO target.
  3. No alert fired for a real cause.
  4. Projected daily cost inside the monthly limit.
  5. The complete flow tested by hand by you, in production, once.

Then, and only then: write the date in the README, take the screenshot of the dashboard in green and move on to 08-05.

  1. When something goes wrong: the incident runbook

Adapted from 07-06 to a one-person project.

flowchart TD
    A["🚨 Something is wrong"] --> B["1. STABILISE<br/>Can I roll back? → roll back NOW<br/>Do not investigate first"]
    B --> C{"Is it resolved?"}
    C -- Yes --> D["2. Note the time, the symptom<br/>and what you did"]
    C -- No --> E["3. DIAGNOSE<br/>logs → deployments → dependencies → quotas"]
    E --> F["4. Mitigate<br/>even with a bodge"]
    F --> D
    D --> G["5. POST-MORTEM<br/>in docs/diario.md"]
    G --> H["6. One concrete action<br/>so it does not happen again"]

Rule number one: stabilise before understanding. The natural impulse is to investigate the cause, and it is the wrong order. If you have a rollback that takes 38 seconds, roll back first and understand afterwards, with the system working and no rush.

The one-page post-mortem

# Incident 2026-11-04 — 5xx errors after the v1.1.0 deployment

**Duration:** 19:42 → 19:51 (9 minutes)
**Impact:** ~40 requests with a 500 error. No data loss.
**Detection:** error rate alert (arrived at 19:47, 5 min after the start)

## What happened
The v1.1.0 deployment introduced a query that used a column
(`opinion.magnitud`) created by migration 003, which had not been applied
in production. The application started up fine (the startup probe does not
touch that table) and only failed on requests to `/api/opiniones`.

## Timeline
- 19:42 Canary deployment at 10 %
- 19:44 First 500s in the logs (I did not look at them: I was watching the
        dashboard, which at 10 % of traffic barely moved the needle)
- 19:47 The alert arrives
- 19:49 I run `./scripts/revertir.sh`
- 19:51 Verified: 0 errors

## Root cause
The pipeline deploys the application but **does not run the migrations**.
In development I apply them by hand, so it worked there.

## What worked
- The rollback: 38 seconds, exactly as rehearsed.
- The canary limited the impact to 10 % of the traffic.

## What did not work
- The alert took 5 minutes. At 10 % of traffic, the 5 % threshold over the
  whole service is reached late.
- I was not watching the logs of the canary revision specifically.

## Actions
1. ✅ Add a migrations step to the pipeline, before the deployment.
2. ✅ Additional alert per revision, not only per service.
3. ⬜ Specific smoke test on `/api/opiniones` after deploying.

A post-mortem in the repository is worth gold in an interview. It shows that you have had a real incident, that you resolved it with a procedure and that you learned something concrete. A project with no incident recorded means one of two things: that you have not used it, or that you did not notice.

  1. The worked example of RefugioReserva

Load test results

Run against development on 2026-10-27, with max-instances=5 and db-f1-micro:

     ✓ query 200
     ✓ booking 201 or 409

     checks.........................: 99.31% ✓ 24893  ✗ 172
     http_req_duration..............: avg=241ms min=61ms med=178ms max=8.9s
       { expected_response:true }...: avg=228ms
       p(90)=402ms  p(95)=712ms  p(99)=2.41s
     http_req_failed................: 0.68%  ✓ 172    ✗ 25065
     errores_negocio................: 0.68%
     iterations.....................: 25065
     vus_max........................: 100

     ✗ http_req_duration.............: p(99)<2000  → 2.41s  FAILED
     ✓ http_req_failed...............: rate<0.01   → 0.0068 OK

Interpretation, point by point:

Observation Diagnosis Action
p50 = 178 ms The typical experience is good None
p95 = 712 ms Inside the target None
p99 = 2.41 s (threshold 2 s) Cold starts when scaling from 1 to 5 instances See below
0.68 % of 5xx 172 errors, all between minutes 3 and 4 Investigated ↓
max = 8.9 s One specific request The first cold start

The 172 errors: the investigation.

$ gcloud logging read 'severity>=ERROR' --project=refugio-dev --limit=5 \
    --format="value(jsonPayload.message)"
FATAL: remaining connection slots are reserved for non-replication superuser connections

Connections exhausted. The arithmetic: a pool of 10 connections per instance × 5 instances = 50 connections requested against a limit of about 25 on db-f1-micro.

The fix, and why that one and not another:

# Before
pool = ConnectionPool(dsn, min_size=5, max_size=10)
# After
pool = ConnectionPool(dsn, min_size=1, max_size=3, timeout=5)

The database was not made bigger (it would have cost money and it was not the problem): the pool was adjusted. Each Cloud Run instance serves up to 80 concurrent requests but spends most of its time waiting; 3 connections per instance × 5 instances = 15, comfortably inside the limit.

Second run after the adjustment:

     http_req_duration.....: p(95)=634ms  p(99)=1.82s   ✓
     http_req_failed.......: 0.00%  ✓ 0  ✗ 26102        ✓
     Cost of the test: €0.38

Documented conclusion: the system sustains 100 concurrent users with a p99 under 2 seconds and no errors. The practical limit is max-instances=5, which is a deliberate cost cap, not a technical limitation. With max-instances=20 it would take about four times as much, and it would cost up to four times as much during a peak.

On the residual p99 of 1.82 s: those are cold starts. min-instances=1 would remove them, but it would cost about €12/month — the entire budget of the project. Decision: accepted and documented. The latency SLO was set on the p95 precisely because of this.

The two rollbacks

Rollback 1 — 2026-11-04, 19:49. The one in the post-mortem in section 10: migration 003 not applied in production. Detected by an alert after 5 minutes, rolled back in 38 seconds, impact limited to 10 % of the traffic by the canary. Corrective action: a migrations step in the pipeline.

Rollback 2 — 2026-11-11, 22:14. More interesting, because there was no error at all.

Symptom: after deploying v1.2.0 at 10 %, the p95 latency of the canary
revision went from 680 ms to 1,940 ms. Zero 5xx errors. Zero alerts.

Detection: I was watching it myself, comparing the canary revision with the
stable one on the dashboard, because the promotion criterion includes
"p95 < 1.2 × the baseline" and 1,940 does not meet it.

Cause: v1.2.0 added the average sentiment to the /api/refugios response
with a correlated subquery over `opinion`, with no index.

Decision: roll back without investigating further, at 22:14. The fix
(an index and a rewritten query) was deployed two days later.

Why this rollback is worth more than the first one in a presentation: no automated system would have caught it. There were no errors, no alert fired, the availability SLO was still perfect. It was caught because an objective promotion criterion written in advance existed and because somebody compared against it. It is the practical demonstration of why the criteria are written beforehand: in the moment, with the new version deployed and a desire to be done, the temptation to say "1.9 seconds is not that bad either" is enormous.

Summary of evidence

Test Result Evidence
Unit 34 tests, 100 % pass docs/evidencias/pytest.txt
Integration 11 tests, including the concurrency one docs/evidencias/pytest.txt
End to end 4 tests against dev Output in Cloud Build
Rebuild from scratch 14 min 20 s docs/evidencias/reconstruccion.log
Security audit 0 failures, 1 warning (public web service, correct) docs/evidencias/auditoria.txt
Image scan 0 critical, 2 medium (base) docs/evidencias/scan.txt
TLS 1.2/1.3 yes, 1.0/1.1 no, 5 headers docs/evidencias/tls.txt
Load 100 VUs, p95 634 ms, 0 % error, €0.38 docs/evidencias/k6.txt
DB outage 503 with a readable message in 1.2 s docs/evidencias/ensayo-bd.log
Restore 22 minutes, data verified docs/evidencias/restauracion.log
Alert triggered Email in 6 min docs/evidencias/alerta.png
Rollback rehearsed 38 seconds scripts/revertir.sh + log
Real incidents 2, with post-mortem docs/diario.md

Common Mistakes and Tips

Writing tests that never fail. A test that always passes, even with the code broken, is worse than no test at all: it gives false confidence. Check every test by deliberately breaking what it tests.

Load testing production by accident. Get the URL right. And set max-instances low before you start: it is your spending cap.

Mistaking a 409 for an error. If your load test counts legitimate business responses as failures, your numbers mean nothing.

Restoring a backup and not checking the data. It is the most dangerous mistake in this lesson, because it produces a "successful" drill that actually demonstrates the opposite.

Leaving resources running after the tests. The restored instance, the test load balancer, the load-test DB. Set yourself a reminder and run an inventory after every testing session.

Deploying on a Friday night. A classic for a reason: if something goes wrong, either you fix it exhausted or you leave it broken over the weekend.

Rebuilding the image for production. That is deploying something you have never tested. Promote by digest.

Migrations that are not backwards compatible. They turn a 38-second rollback into a 22-minute restore with data loss.

Not having written promotion criteria. In the moment, it always looks like "it is not that bad". Write them beforehand, when you are not in a hurry.

Tip: save the output of everything. A docs/evidencias/ directory with the output of every test is what turns "my system is reliable" into "here are the numbers". In 08-05 you will be enormously grateful for it.

Tip: do the rebuild from scratch halfway through the project. Discovering in week 3 that a resource was missing from the code costs half an hour; discovering it the day before submission costs the submission.

Tip: one documented real incident is worth more than zero incidents. Do not hide the failures: tell them with their post-mortem. It is the most reliable signal that the system is real and that you know how to operate it.

Exercises

Exercise 1 — Build the test pyramid and test your infrastructure

Write for your project: at least 15 unit tests of the business logic, including edge cases; at least 5 integration tests against emulators or an ephemeral containerised database, applying the same migrations as in production and with a concurrency test on the critical point of your domain; and 3 end-to-end tests against development. Integrate them into the pipeline in the right order.

Then run the definitive infrastructure test: destroy the whole development environment, rebuild it from scratch, seed the data, deploy and verify that it works. Time it and document everything that broke along the way.

Exercise 2 — Audit security and measure capacity

Write and run your security audit script with at least eight automated checks (primitive roles, JSON keys, public buckets, public IP on the DB, secrets in the git history, credential files, services without authentication, audit logs). Fix everything that comes out red. Check the TLS, the accepted versions and the security headers. Scan your image.

Then run a staged load test against development with a spending cap, measuring p50/p95/p99 latency, error rate distinguishing system errors from business responses, active instances and the cost of the test itself. Interpret the results, apply at least one concrete adjustment and run it again to demonstrate the improvement.

Exercise 3 — Drill reliability, deploy and launch

Run the three reliability drills: switch off your database and document the exact behaviour (code, message, time, what keeps working), fixing the degradation if it is not graceful; restore a backup onto a new instance, verify the data and time the RTO; and trigger your alert, checking that it arrives.

Implement the canary deployment with objective promotion and rollback criteria written in advance, rehearse the rollback with a stopwatch, go through the entire pre-launch checklist, and launch. Document the first 24 hours.

Solutions

Solution 1 — Tests and rebuild of RefugioReserva

The final pyramid: 34 unit tests, 11 integration tests and 4 end-to-end tests. The test that justifies the whole project is test_no_hay_sobreventa_con_concurrencia from section 2.2, and its story is worth telling: the first implementation did not pass it.

FAILED test_no_hay_sobreventa_con_concurrencia
AssertionError: Overbooking: 7 bookings

Seven out of twenty threads managed to book the single place. The original implementation read the availability, decided and then inserted, with no lock: between the read and the insert, another six threads had read the same thing. It is exactly the failure that caused the fictional federation's seven overbookings a year — the original business problem, reproduced in the code.

With SELECT ... FOR UPDATE on the refuge row, the result was 1 and 19. This test, in the presentation, takes one slide and answers on its own the question "why Cloud SQL and not Firestore?".

The rebuild from scratch, run twice:

Attempt Date Result Findings
1 2026-10-24 Failed 4 problems (below)
2 2026-11-02 ✅ 14 min 20 s None

The four problems from the first attempt, which is where the value of the exercise lies:

  1. The refugio-session-key secret was created empty. I had filled it in by hand in week 2 and had completely forgotten. The app started and failed on signing the first session. Fixed with random_password + secret_version in Terraform.
  2. A depends_on on the peering was missing. Terraform tried to create Cloud SQL before the peering was ready. In the incremental apply runs it was never visible, because the peering had existed since day one.
  3. The BigQuery dataset had been created by hand. Discovered because the function could not find the table. It did not have the gestionado-por label, which was exactly the detector designed for this.
  4. terraform destroy failed on the photos bucket because it had objects and force_destroy = false. Correct in production, awkward in development: it was parameterised as force_destroy = var.entorno == "dev".

None of the four would have been detected any other way. The rebuild from scratch is not just another test: it is the only one that validates that your IaC is real.

Solution 2 — Audit and load test of RefugioReserva

First run of the audit:

=== SECURITY AUDIT: refugio-prod ===
[1] Primitive roles            ✅ No SA with owner/editor
[2] Account keys               ✅ Zero JSON keys
[3] Public buckets             ❌ 1 public bucket: gs://refugio-fotos-8f2a
[4] DB with a public IP        ✅ refugio-db has no public IP
[5] Secrets in git             ✅ No obvious secrets
[6] Credentials in the tree    ✅ No credential files
[7] Cloud Run without auth     ⚠️  refugio-web is public (correct)
[8] Audit logs                 ✅ Audit logs present

=== RESULT: 1 failure ===

The public bucket. I had made it public in week 4 so that the thumbnails would be served directly, "temporarily". Four weeks later it was still like that, and on top of that I had left allUsers on the whole bucket, not just on the thumbnails prefix: the original photos were also accessible to anybody who guessed the name.

Fix: allUsers was removed, public_access_prevention = "enforced" was switched on, and the thumbnails moved to being served through signed URLs with a one-hour expiry, generated by the application.

def url_firmada(nombre_objeto: str, minutos: int = 60) -> str:
    blob = _bucket.blob(nombre_objeto)
    return blob.generate_signed_url(version="v4",
                                    expiration=timedelta(minutes=minutos),
                                    method="GET")

It is exactly mistake 3 from 08-01 — leaving security until the end — showing up in its most typical form: something "temporary" that stays. And it was found by an eight-check script that took twenty seconds to run. Recorded in the 08-05 self-assessment as debt paid, with the date it was introduced and the date it was detected: four weeks of exposure.

TLS and headers, after adding the middleware:

tls1     rejects    ✅
tls1_1   rejects    ✅
tls1_2   accepts    ✅
tls1_3   accepts    ✅
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff
x-frame-options: DENY
content-security-policy: default-src 'self'; img-src 'self' data: https://storage.googleapis.com
referrer-policy: strict-origin-when-cross-origin

Load: the results and their interpretation are those in section 11. The one-line summary, which is what goes into the presentation: 100 concurrent users, a p95 of 634 ms, 0 % errors, €0.38 of test cost, and a bottleneck found and fixed that was not the database but the size of the connection pool.

Solution 3 — Reliability and launch of RefugioReserva

Database outage drill, first run:

Home page: 200 in 0.31s        ✅ still working (static content)
API:       500 in 30.02s       ❌ 30 seconds and a Python traceback
Health:    200                 ❌ IT LIES: it says it is ready and it is not

Two serious problems. The user waited 30 seconds to receive an incomprehensible error, and Cloud Run kept sending traffic to an instance that could not serve it because the startup probe did not check the database.

Fixes: connect_timeout=5 in the DSN, a global handler translating connection failures into a 503 with a JSON body, and the startup probe doing a SELECT 1.

Second run:

Home page: 200 in 0.28s        ✅
API:       503 in 1.24s        ✅ {"error":"servicio_no_disponible",
                                   "mensaje":"We cannot check availability
                                   right now. Try again in a few minutes."}
Health:    503                 ✅ Cloud Run stops sending traffic
Photos:    200                 ✅ the bucket is independent of the DB
Dashboard: 200                 ✅ Looker Studio reads from BigQuery

Restore: 22 minutes, with the data verified (2,000 bookings, 12 refuges, the last booking at 18:42 the previous day). Real RPO: up to 24 hours, because the backup is daily. Documented in the runbook, and noted as technical debt: enabling point-in-time recovery in production would bring the RPO down to minutes for a small additional cost.

Rollback rehearsed: 38 seconds, measured three times with results of 36, 38 and 41 seconds.

The launch, 2026-11-15 at 11:00 on a Saturday:

Time Action Result
10:40 Security audit + terraform plan 0 failures, No changes
10:50 Pre-launch checklist 38/38 boxes
11:02 Canary at 10 % 0 errors in 10 min, p95 611 ms
11:14 50 % 0 errors, p95 598 ms
11:26 100 % 0 errors
11:30 Tag v1.0.0
12:30 First hour watched 0 errors, 47 requests
Day +1 24 hours 0 5xx errors, daily cost €0.34

Success declared on 16 November at 11:30, with the five conditions met: 24 hours with no anomalous errors, p95 inside the SLO, no real alert, a projected cost of €10.20/month against the €12 limit, and the complete flow tested by hand in production.

The two later rollbacks — the one on 4 November and the one on the 11th — are in section 11 with their post-mortem. Neither of them could have been handled in under a minute without the runbook written and rehearsed beforehand.

Conclusion

Your project no longer just works: you know it works, and you have the evidence.

You can tell apart the three levels of "finished" and which one you are at, with nine operational boxes that define the third one unambiguously.

You have the test pyramid sized for a project of this scale — around 30 unit, 10 integration, 4 end-to-end — you know what is worth testing and what is not, and you have the test that is worth all the others put together: the concurrency test on the critical point of your domain, the one that fails with the naive implementation and that justifies your database choice on its own. And you know how to test against free emulators and ephemeral containerised databases, applying the same migrations as in production.

You know how to test the infrastructure: static validation, the plan published in the pull request with a read-only account, tflint and checkov with their exceptions documented instead of chased, and above all the definitive test: destroying the environment and recreating it from scratch. It is the only one that uncovers the secret you filled in by hand, the resource created from the console, the implicit dependency in the wrong order and the migration that only works against a database that already exists.

You have an executable security checklist with eight automated checks that runs in twenty seconds and that in the example found a public bucket that had been exposed for four weeks. And you know how to verify TLS, the accepted versions, the five headers that matter and the effective permissions — including proving that an account does not have a dangerous permission.

You know how to run an honest, cheap load test: against development, with a spending cap, staged, distinguishing system errors from legitimate business responses, measuring percentiles rather than averages, and including the cost of the test itself among the metrics. And you know how to interpret it: a p99 far above the p95 means cold starts; 5xx errors as load rises usually mean database connections, and the solution is not a bigger database but a smaller pool.

You have drilled reliability for real: switching off the database to discover that your error took thirty seconds and showed a Python traceback, restoring a backup verifying the data — the step that turns a real drill into a fake one — and timing it, and triggering the alert to check that it arrives.

You have a deployment strategy with the same image promoted by digest, a canary at 10/50/100 with objective criteria written in advance, and a rollback rehearsed with a stopwatch. And you know what a traffic rollback does not roll back — migrations, data, infrastructure — from which comes the rule that avoids most disasters: backwards-compatible migrations, always.

You have the pre-launch checklist in its five blocks, the runbook with its numbered procedures, the script for the first 24 hours with what to watch and in what order, the five conditions for declaring success, and the incident runbook with its rule number one: stabilise before understanding.

And you have the RefugioReserva example with its real numbers: 14 minutes to rebuild, 22 to restore, 38 seconds to roll back, a p95 of 634 ms with 100 users, €0.38 of test cost, a public bucket found and closed, and two rollbacks — one caught by an alert and the other caught only because an objective written criterion existed, with no error, no alert, and the SLO in green.

In the next lesson, 08-05, the work changes in nature: you stop building and start communicating. You are going to prepare the presentation for three different audiences, structure fifteen minutes slide by slide, put together a timed demo with a recorded plan B, talk about your numbers — because "this costs me €10 a month" is worth more than any adjective — write the deliverable documentation with its templates, assess yourself against the 08-01 rubric with brutal honesty, prepare the answers to the questions you are going to be asked, and close with the final clean-up: a terraform destroy followed by an apply that works again, which is the best possible ending this project can have.

Google Cloud Platform (GCP) Course

Module 1: Introduction to Google Cloud Platform

Module 2: Core GCP Services

Module 3: Networking and Security

Module 4: Data and Analytics

Module 5: Machine Learning and AI

Module 6: DevOps and Monitoring

Module 7: Advanced GCP Topics

Module 8: Final Project

© Copyright 2026. All rights reserved