Your system works. You have seen it work. You have created a booking, it reached BigQuery, the dashboard shows it and the certificate is valid.
And yet it is not finished, because there is an enormous difference between "I have seen it work" and "I know it works". The first statement rests on a one-off observation, made by you, under the best possible conditions: no concurrency, no errors, nothing falling over, with data you put in yourself. The second rests on tests.
This lesson answers one specific question: what does "finished" really mean? And the answer has seven parts, each of which answers a question somebody is going to ask you:
| Question | Answered by |
|---|---|
| "What if two people do the same thing at the same time?" | The code tests |
| "Can you bring this back up from scratch?" | The infrastructure tests |
| "Is anything exposed that should not be?" | The security tests |
| "How much traffic does it take?" | The load test |
| "What happens if the database goes down?" | The reliability tests |
| "How do you deploy without breaking anything?" | The deployment strategy |
| "And if something goes wrong?" | The rollback procedure and the incident runbook |
None of those seven is optional in a project you want to show off. And they can all be done over a long weekend, with free tools and without spending more than a couple of euros.
By the end, your project will not be "done": it will be launched, with evidence for every claim you make about it.
Contents
- What "finished" really means
- The test pyramid applied to this project
- Infrastructure tests
- Security tests
- Load tests
- Reliability tests
- The deployment strategy
- The pre-launch checklist
- The launch and the first 24 hours
- When something goes wrong: the incident runbook
- The worked example of RefugioReserva
- What "finished" really means
There are three levels of "finished", and it is worth knowing which one you are at:
| Level | Claim | Evidence | Typical score |
|---|---|---|---|
| Works on my machine | "I have seen it work" | A screenshot | 40-55 |
| Works deployed | "It is in production and it responds" | A URL | 55-70 |
| I know it works | "Here are the tests, the measurements and what happens when it fails" | Automated tests, measured numbers, drills carried out | 75-95 |
The difference between the second and the third is this lesson. And it is where half the points live that separate an ordinary project from one an interviewer remembers.
The operational definition of "finished"
A deliverable is finished when you can tick the nine boxes:
- [ ] There is at least one automated test that fails if you break the core logic
- [ ] The infrastructure is recreated from scratch and the system starts up
- [ ] The security checklist passes in full
- [ ] You know your capacity limit as a measured number
- [ ] You have deliberately switched off a dependency and you know what happens
- [ ] You have restored a backup and timed it
- [ ] You have triggered the alert and it arrived
- [ ] You have rehearsed the rollback and you know how long it takes
- [ ] The pre-launch checklist is complete
None of these needs paid tools. All of them need actually doing, not just ticking.
- The test pyramid applied to this project
flowchart TB
E2E["End to end — 3-5 tests<br/>Against the real development environment<br/>Slow (min), expensive, brittle"]
INT["Integration — 8-15 tests<br/>Against emulators or an ephemeral DB<br/>Medium (s)"]
UNI["Unit — 20-40 tests<br/>Pure logic, no network or DB<br/>Fast (ms), free"]
UNI --> INT --> E2E
style UNI fill:#e6f4ea
style INT fill:#fef7e0
style E2E fill:#fce8e6
The proportion rule for a project of this size: around 30 unit tests, around 10 integration tests and 3-5 end-to-end tests. You do not need more, and chasing 100 % coverage is time badly invested in a portfolio.
What to test and what not to:
| Do test | Do not waste time testing |
|---|---|
| Business rules (capacity calculation, prices, states) | That FastAPI returns 200 on a trivial route |
| Edge cases (zero, negative, maximum, empty) | That the ORM knows how to do a SELECT |
| Concurrency at the critical point | Getters and setters |
| Input validation | That Terraform knows how to create a bucket |
| The complete flow, once | Every permutation of the interface |
2.1 Unit tests: the logic with no dependencies
# app/tests/test_unitarios.py
import pytest
from datetime import date
from src.dominio import calcular_disponibilidad, validar_reserva, ErrorValidacion
class TestDisponibilidad:
"""The core business rule: never more bookings than capacity."""
def test_refugio_vacio_ofrece_toda_la_capacidad(self):
assert calcular_disponibilidad(capacidad=40, ocupadas=0) == 40
def test_descuenta_las_plazas_ocupadas(self):
assert calcular_disponibilidad(capacidad=40, ocupadas=15) == 25
def test_refugio_lleno_ofrece_cero(self):
assert calcular_disponibilidad(capacidad=40, ocupadas=40) == 0
def test_nunca_devuelve_negativo(self):
"""Edge case: if a bug left overbooking in the data, the function
must return 0, not a negative number that the interface would
display as '-3 places free'."""
assert calcular_disponibilidad(capacidad=40, ocupadas=45) == 0
class TestValidacion:
@pytest.mark.parametrize("plazas", [0, -1, 13, 999])
def test_rechaza_plazas_fuera_de_rango(self, plazas):
with pytest.raises(ErrorValidacion):
validar_reserva(plazas=plazas, fecha=date(2026, 8, 15))
def test_rechaza_fecha_pasada(self):
with pytest.raises(ErrorValidacion, match="pasado"):
validar_reserva(plazas=2, fecha=date(2020, 1, 1))
@pytest.mark.parametrize("correo", ["", "sin-arroba", "a@", "@b.com"])
def test_rechaza_correo_invalido(self, correo):
with pytest.raises(ErrorValidacion):
validar_reserva(plazas=2, fecha=date(2026, 8, 15), email=correo)test_nunca_devuelve_negativo is the kind of test that earns its keep. It does not test the normal case — everyday use tests that — it tests the odd case that, when it happens in production at three in the morning, will produce absurd behaviour in the interface.
2.2 Integration tests: against real services or emulators
Google publishes local emulators for Firestore, Pub/Sub, Bigtable and Datastore. They are free, they start in seconds and they behave like the real service in everything that matters.
# Local emulators, free and without touching the cloud
gcloud emulators firestore start --host-port=localhost:8080
gcloud emulators pubsub start --host-port=localhost:8085
# The client libraries detect them through an environment variable
export FIRESTORE_EMULATOR_HOST=localhost:8080
export PUBSUB_EMULATOR_HOST=localhost:8085There is no emulator for PostgreSQL, but there is something better: an ephemeral container.
# app/tests/conftest.py
import pytest, subprocess, time, os
import psycopg
@pytest.fixture(scope="session")
def bd_efimera():
"""PostgreSQL in a container, created and destroyed by the test session.
Cost: €0. Does not touch the cloud."""
subprocess.run([
"docker", "run", "-d", "--name", "pg-pruebas",
"-e", "POSTGRES_PASSWORD=pruebas",
"-e", "POSTGRES_DB=reservas",
"-p", "55432:5432", "postgres:16-alpine",
], check=True)
dsn = "postgresql://postgres:pruebas@localhost:55432/reservas"
for _ in range(30): # wait until it accepts connections
try:
psycopg.connect(dsn).close()
break
except Exception:
time.sleep(1)
# Apply the SAME migrations as in production
with psycopg.connect(dsn) as con:
for f in sorted(os.listdir("data/migraciones")):
with open(f"data/migraciones/{f}") as fh:
con.execute(fh.read())
con.commit()
yield dsn
subprocess.run(["docker", "rm", "-f", "pg-pruebas"], check=True)Applying the same migrations as in production is what gives the test its value: if a migration is broken, you find out here and not at deployment time.
And the test that really matters in RefugioReserva — concurrency:
# app/tests/test_integracion.py
import pytest, psycopg
from concurrent.futures import ThreadPoolExecutor
from src.repositorio import crear_reserva, ErrorSinPlazas
def test_no_hay_sobreventa_con_concurrencia(bd_efimera):
"""The case that breaks badly built booking systems:
twenty people try to book the last place at the same time."""
with psycopg.connect(bd_efimera) as con:
con.execute("INSERT INTO refugio (nombre, altitud_m, capacidad) "
"VALUES ('Refugio Prueba', 2000, 1)")
con.commit()
def intentar():
try:
with psycopg.connect(bd_efimera) as c:
crear_reserva(c, refugio_id=1, fecha="2026-08-15", plazas=1,
titular="Fictitious", email="[email protected]")
return "ok"
except ErrorSinPlazas:
return "sin_plazas"
with ThreadPoolExecutor(max_workers=20) as ex:
resultados = list(ex.map(lambda _: intentar(), range(20)))
# EXACTLY one must succeed. Not zero, not two.
assert resultados.count("ok") == 1, f"Overbooking: {resultados.count('ok')} bookings"
assert resultados.count("sin_plazas") == 19
with psycopg.connect(bd_efimera) as con:
total = con.execute("SELECT COALESCE(SUM(plazas),0) FROM reserva "
"WHERE refugio_id=1 AND estado='confirmada'").fetchone()[0]
assert total == 1This test on its own justifies ADR-002 (choosing Cloud SQL for its transactions). And if you run it against a naive implementation — read availability, decide, insert, with no lock — it fails, which is exactly what it should do. The correct implementation uses SELECT ... FOR UPDATE on the refuge row:
def crear_reserva(con, refugio_id, fecha, plazas, titular, email):
with con.transaction():
# The row lock serialises the concurrent attempts
cap = con.execute(
"SELECT capacidad FROM refugio WHERE id = %s FOR UPDATE",
(refugio_id,)).fetchone()[0]
ocupadas = con.execute(
"SELECT COALESCE(SUM(plazas),0) FROM reserva "
"WHERE refugio_id=%s AND fecha=%s AND estado='confirmada'",
(refugio_id, fecha)).fetchone()[0]
if ocupadas + plazas > cap:
raise ErrorSinPlazas()
con.execute("INSERT INTO reserva (refugio_id, fecha, plazas, "
"nombre_titular, email_titular) VALUES (%s,%s,%s,%s,%s)",
(refugio_id, fecha, plazas, titular, email))2.3 End-to-end tests
Few, slow and against the real development environment. Three or four are enough:
# app/tests/test_e2e.py
import os, time, uuid, pytest, requests
from google.cloud import bigquery
BASE = os.environ["URL_DEV"]
@pytest.mark.e2e
def test_flujo_completo_reserva_llega_a_analitica():
"""The whole critical path: book → event → BigQuery."""
marca = str(uuid.uuid4())[:8]
r = requests.post(f"{BASE}/api/reservas", timeout=30, json={
"refugio_id": 1, "fecha": "2027-01-15", "plazas": 1,
"nombre_titular": f"E2E {marca}", # FICTITIOUS data
"email_titular": f"e2e-{marca}@example.com",
})
assert r.status_code == 201
reserva_id = r.json()["id"]
# Propagation through Pub/Sub is not instant: poll with a limit
bq = bigquery.Client()
consulta = """
SELECT COUNT(*) AS n
FROM `refugio-datos.refugio_analitica.reservas_eventos`
WHERE reserva_id = @rid AND DATE(ocurrido_en) = CURRENT_DATE()
"""
cfg = bigquery.QueryJobConfig(query_parameters=[
bigquery.ScalarQueryParameter("rid", "STRING", reserva_id)])
for _ in range(12): # up to 2 minutes
n = list(bq.query(consulta, job_config=cfg).result())[0].n
if n == 1:
return
time.sleep(10)
pytest.fail("The event did not reach BigQuery within 2 minutes")
@pytest.mark.e2e
def test_sondas_de_salud_responden():
assert requests.get(f"{BASE}/salud/vivo", timeout=10).status_code == 200
assert requests.get(f"{BASE}/salud/arranque", timeout=10).status_code == 2002.4 Integration into Cloud Build
steps:
- id: unitarias-e-integracion
name: python:3.12-slim
entrypoint: bash
args:
- -c
- |
pip install --no-cache-dir -r app/requirements.txt -r app/requirements-dev.txt
cd app && python -m pytest tests/ -v -m "not e2e" --tb=short \
--junitxml=/workspace/resultados.xml
# ... build, publish, deploy ...
- id: e2e
name: python:3.12-slim
entrypoint: bash
args:
- -c
- |
pip install --no-cache-dir -r app/requirements-dev.txt
export URL_DEV=$(gcloud run services describe refugio-web \
--region=europe-west1 --format='value(status.url)')
cd app && python -m pytest tests/ -v -m e2e
waitFor: [desplegar]The unit and integration tests go before building; the end-to-end tests after deploying, because they need a live system.
- Infrastructure tests
3.1 Static validation
terraform fmt -check -recursive # consistent formatting
terraform validate # syntax and references
terraform plan -detailed-exitcode # 0=no changes, 2=changes, 1=error-detailed-exitcode is useful in CI: it lets you fail the build if the deployed state does not match the code.
3.2 The plan reviewed in the pull request
# .github/workflows/plan.yml
name: Terraform plan
on: pull_request
permissions:
contents: read
id-token: write
pull-requests: write
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.WIF_PROVIDER }}
service_account: ${{ vars.SA_PLAN }} # READ-ONLY SA
- uses: hashicorp/setup-terraform@v3
- run: terraform -chdir=infra/envs/dev init
- id: plan
run: terraform -chdir=infra/envs/dev plan -no-color -out=plan.tfplan
- name: Publish the plan as a comment
uses: actions/github-script@v7
with:
script: |
const salida = `#### Terraform plan
\`\`\`
${{ steps.plan.outputs.stdout }}
\`\`\``;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner, repo: context.repo.repo, body: salida
});Making the plan account read-only (roles/viewer + access to the state bucket) matters: a plan must not be able to change anything, and that way neither can a malicious pull request.
3.3 Policy analysis: tflint and checkov
# tflint: syntax errors, invalid arguments, provider best practices
tflint --init && tflint --recursive
# checkov: security and compliance checks over the code
checkov -d infra/ --framework terraform --compactExamples of what they catch and that may well have slipped past you:
| Tool | Catches |
|---|---|
tflint |
Non-existent machine type, deprecated attribute, unused variable |
checkov |
Bucket without uniform access, DB without enforced SSL, logs disabled, public service with no justification |
Do not chase zero findings. Some will be false positives for your context (for example, Cloud SQL not having high availability is deliberate in a portfolio). The right move is to document the exception:
# checkov:skip=CKV_GCP_79:High availability not applicable; portfolio project
# with a €12/month budget. See docs/adr/ADR-002.Add it to the pipeline as an informational, non-blocking step, right at the start:
- id: analisis-iac
name: bridgecrew/checkov:latest
args: ["-d", "infra/", "--compact", "--soft-fail"]3.4 The definitive test: recreate the environment from scratch
This is the test that separates a project with Terraform from a project with real infrastructure as code. It is worth 4 points of the rubric and it is the one most people take for granted without ever having done it.
#!/usr/bin/env bash
# scripts/prueba-reconstruccion.sh
set -euo pipefail
ENTORNO="dev"
INICIO=$(date +%s)
echo "=== 1. Initial state ==="
terraform -chdir="infra/envs/${ENTORNO}" state list | wc -l
echo "=== 2. DESTROY ==="
terraform -chdir="infra/envs/${ENTORNO}" destroy -auto-approve
echo "=== 3. Check that nothing is left ==="
gcloud run services list --project="refugio-${ENTORNO}" --format="value(name)"
gcloud sql instances list --project="refugio-${ENTORNO}" --format="value(name)"
gcloud compute networks list --project="refugio-${ENTORNO}" --format="value(name)"
echo "=== 4. REBUILD ==="
terraform -chdir="infra/envs/${ENTORNO}" apply -auto-approve
echo "=== 5. Migrations and data ==="
./scripts/migrar.sh "${ENTORNO}"
python data/seed/generar.py --entorno="${ENTORNO}"
echo "=== 6. Deploy the last known image ==="
./scripts/desplegar.sh "${ENTORNO}" "$(git rev-parse --short HEAD)"
echo "=== 7. Check that it works ==="
URL=$(gcloud run services describe refugio-web --region=europe-west1 \
--project="refugio-${ENTORNO}" --format='value(status.url)')
CODIGO=$(curl -s -o /dev/null -w '%{http_code}' "${URL}/salud/arranque")
test "${CODIGO}" = "200" || { echo "FAILURE: the app does not respond"; exit 1; }
echo "=== 8. End-to-end test ==="
URL_DEV="${URL}" python -m pytest app/tests/ -m e2e -q
FIN=$(date +%s)
echo "✅ FULL REBUILD in $(( (FIN-INICIO)/60 )) minutes"What this test uncovers, and no other one does:
| Typical finding | Why you cannot see it any other way |
|---|---|
| A resource created by hand in week 2 | In an environment that already exists, you never miss it |
| Implicit dependencies in the wrong order | In an incremental apply, the earlier resource was already there |
| A secret you filled in by hand | The code creates it empty and nobody notices |
| Migrations that only work against the existing DB | They are never run from scratch |
| A permission granted from the console | The app works and you do not know why |
Do it at least twice: once halfway through the project (to find the gaps while they are still cheap to fix) and once before the launch. And time it: the rebuild time is a number that looks very good in the presentation.
- Security tests
4.1 Image scanning
Artifact Registry includes vulnerability analysis:
gcloud artifacts docker images scan \
"europe-west1-docker.pkg.dev/refugio-dev/refugio-imagenes/web:latest" \
--format="value(response.scan)"
# And to list the findings
gcloud artifacts docker images list-vulnerabilities <SCAN_RESULT> \
--format="table(vulnerability.effectiveSeverity, vulnerability.shortDescription)"Realistic criterion: zero critical vulnerabilities and zero exploitable high ones in your code. Those in the base image are fixed by updating the base (python:3.12-slim to its latest version) and rebuilding. Medium and low ones in system dependencies you do not use are documented and accepted.
4.2 The executable checklist
This script is a deliverable in its own right. Save it as scripts/auditoria-seguridad.sh, run it, and save the output in docs/evidencias/:
#!/usr/bin/env bash
# scripts/auditoria-seguridad.sh
set -uo pipefail
PROY="${1:?Usage: $0 <project>}"
FALLOS=0
ok() { echo " ✅ $1"; }
falla(){ echo " ❌ $1"; FALLOS=$((FALLOS+1)); }
avisa(){ echo " ⚠️ $1"; }
echo "=== SECURITY AUDIT: ${PROY} ==="
echo "[1] Primitive roles on service accounts"
N=$(gcloud projects get-iam-policy "$PROY" --format=json |
jq '[.bindings[] | select(.role|test("roles/(owner|editor)")) |
.members[] | select(startswith("serviceAccount:"))] | length')
[ "$N" -eq 0 ] && ok "No SA with owner/editor" || falla "$N accounts with a primitive role"
echo "[2] User-managed service account keys"
TOTAL=0
for SA in $(gcloud iam service-accounts list --project="$PROY" --format="value(email)"); do
K=$(gcloud iam service-accounts keys list --iam-account="$SA" --managed-by=user \
--format="value(name)" 2>/dev/null | wc -l)
TOTAL=$((TOTAL+K))
[ "$K" -gt 0 ] && echo " · $SA has $K key(s)"
done
[ "$TOTAL" -eq 0 ] && ok "Zero JSON keys" || falla "$TOTAL downloadable keys"
echo "[3] Publicly accessible buckets"
PUB=0
for B in $(gcloud storage buckets list --project="$PROY" --format="value(name)"); do
if gcloud storage buckets get-iam-policy "gs://$B" --format=json |
jq -e '.bindings[]?.members[]? | select(. == "allUsers" or . == "allAuthenticatedUsers")' >/dev/null 2>&1; then
echo " · gs://$B is PUBLIC"; PUB=$((PUB+1))
fi
done
[ "$PUB" -eq 0 ] && ok "No public bucket" || falla "$PUB public bucket(s)"
echo "[4] Databases with a public IP"
for I in $(gcloud sql instances list --project="$PROY" --format="value(name)"); do
IPV4=$(gcloud sql instances describe "$I" --project="$PROY" \
--format="value(settings.ipConfiguration.ipv4Enabled)")
[ "$IPV4" = "False" ] && ok "$I has no public IP" || falla "$I HAS A PUBLIC IP"
done
echo "[5] Secrets in the repository"
if git log -p --all 2>/dev/null | grep -Ei '(password|api[_-]?key|secret|BEGIN (RSA|PRIVATE))' \
| grep -v 'secret_id\|secretAccessor\|SecretManager\|secret_key_ref\|refugio-db-password' \
| head -5 | grep -q .; then
falla "Possible secrets in the git history — REVIEW"
else
ok "No obvious secrets in the history"
fi
echo "[6] Credential files in the tree"
if find . -name "*.json" -not -path "./node_modules/*" -not -path "./.git/*" \
-exec grep -l '"type": *"service_account"' {} \; 2>/dev/null | grep -q .; then
falla "There are service account key files in the repository"
else
ok "No credential files"
fi
echo "[7] Cloud Run services without authentication"
for S in $(gcloud run services list --project="$PROY" --region=europe-west1 --format="value(name)"); do
if gcloud run services get-iam-policy "$S" --project="$PROY" --region=europe-west1 \
--format=json | jq -e '.bindings[]?.members[]? | select(. == "allUsers")' >/dev/null 2>&1; then
avisa "$S is public (correct if it is the web app; review otherwise)"
else
ok "$S requires authentication"
fi
done
echo "[8] Admin activity audit logs"
gcloud logging read 'logName:"cloudaudit.googleapis.com%2Factivity"' \
--project="$PROY" --limit=1 --format="value(timestamp)" | grep -q . \
&& ok "Audit logs present" || avisa "No recent activity recorded"
echo
echo "=== RESULT: ${FALLOS} failure(s) ==="
exit $((FALLOS > 0))4.3 Checking TLS and headers
DOMINIO="refugioreserva.example"
# Certificate: issuer and validity
echo | openssl s_client -connect "${DOMINIO}:443" -servername "${DOMINIO}" 2>/dev/null |
openssl x509 -noout -subject -issuer -dates
# TLS versions: 1.2 and 1.3 yes; 1.0 and 1.1 must fail
for V in tls1 tls1_1 tls1_2 tls1_3; do
printf "%-8s " "$V"
echo | openssl s_client -"$V" -connect "${DOMINIO}:443" 2>/dev/null | \
grep -q "Verify return code: 0" && echo "accepts" || echo "rejects"
done
# Security headers
curl -sI "https://${DOMINIO}" | grep -iE \
'strict-transport|x-content-type|x-frame|content-security|referrer-policy'
# HTTP must redirect
curl -sI "http://${DOMINIO}" | head -1| Header | Recommended value | What it prevents |
|---|---|---|
Strict-Transport-Security |
max-age=31536000; includeSubDomains |
Downgrade to HTTP |
X-Content-Type-Options |
nosniff |
Wrong type interpretation |
X-Frame-Options |
DENY |
Clickjacking |
Content-Security-Policy |
At least default-src 'self' |
Script injection |
Referrer-Policy |
strict-origin-when-cross-origin |
URL leakage to third parties |
These are added in an application middleware with ten lines of code, and they are worth 2 points.
4.4 Reviewing effective permissions
# What can my application service account actually do?
gcloud projects get-iam-policy refugio-prod --format=json |
jq -r --arg sa "serviceAccount:[email protected]" \
'.bindings[] | select(.members[]? == $sa) | .role'
# Check a specific permission it should NOT have
gcloud policy-troubleshoot iam \
"//cloudresourcemanager.googleapis.com/projects/refugio-prod" \
--principal-email="[email protected]" \
--permission="resourcemanager.projects.setIamPolicy"
# Expected: NOT_GRANTEDProving that it does not have a dangerous permission is as valuable as proving that it has the ones it needs.
- Load tests
5.1 How to run an honest, cheap test
Three rules, in order of importance:
- Against development, never against production, unless it is the launch itself and you have planned it.
- With a spending cap:
--max-instanceslimited and a duration budget. A load test with no cap can scale to a hundred instances and eat the month's budget in twenty minutes. - Start small and go up. 10 users, 50, 100. Stop when something breaks: that is the number you are after.
Free tools: hey, k6, locust, wrk. For this project, k6 is the most complete and hey the quickest to use.
// pruebas/carga.js — run with: k6 run pruebas/carga.js
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';
const erroresNegocio = new Rate('errores_negocio');
export const options = {
stages: [
{ duration: '1m', target: 10 }, // warm-up
{ duration: '2m', target: 50 }, // expected load
{ duration: '2m', target: 100 }, // double that
{ duration: '1m', target: 0 }, // cool-down
],
thresholds: {
http_req_duration: ['p(95)<1000', 'p(99)<2000'],
http_req_failed: ['rate<0.01'],
errores_negocio: ['rate<0.05'],
},
};
const BASE = __ENV.URL;
export default function () {
// 80% queries, 20% bookings: a realistic profile, not all writes
if (Math.random() < 0.8) {
const r = http.get(`${BASE}/api/disponibilidad?refugio_id=1&fecha=2027-01-15`);
check(r, { 'query 200': (x) => x.status === 200 });
erroresNegocio.add(r.status >= 500);
} else {
const r = http.post(`${BASE}/api/reservas`, JSON.stringify({
refugio_id: Math.ceil(Math.random() * 12),
fecha: '2027-01-15', plazas: 1,
nombre_titular: 'Fictitious Load',
email_titular: `carga-${__VU}-${__ITER}@example.com`,
}), { headers: { 'Content-Type': 'application/json' } });
// 409 (no places) is correct, not a system error
check(r, { 'booking 201 or 409': (x) => x.status === 201 || x.status === 409 });
erroresNegocio.add(r.status >= 500);
}
sleep(Math.random() * 2);
}The detail that makes the test honest: distinguishing between a system error (5xx) and a legitimate business response (409, no places left). A test that counts the 409s as failures will give you a terrifying and false error rate.
5.2 What to measure
| Metric | Where it is measured | What it tells you | Reasonable threshold |
|---|---|---|---|
| p50 latency | k6 | The typical experience | <300 ms |
| p95 latency | k6 | The bad but common experience | <1,000 ms |
| p99 latency | k6 | The real worst case | <2,000 ms |
| 5xx error rate | k6 + Monitoring | Reliability | <1 % |
| Active instances | Monitoring | Saturation and scaling | Without reaching the maximum |
| Cold starts | Cloud Run logs | Added latency | Visible in the p99 |
| DB connections | Cloud SQL Insights | The usual bottleneck | <80 % of the maximum |
| Cost of the test | Billing | That it does not ruin you | <€1 |
Do not look only at the average. The average lies: with 99 requests of 100 ms and one of 10 seconds, the average is 200 ms and there is a user who waited ten seconds. Percentiles tell the truth.
5.3 How to interpret and what to adjust
| Symptom | Likely cause | Adjustment |
|---|---|---|
| p99 far above the p95 | Cold starts | min-instances=1 (it costs money) or accept it and document it |
| Latency rises with load, no errors | CPU saturation | Raise CPU, or lower concurrency so it scales sooner |
| 5xx errors as load rises | DB connections exhausted | A smaller connection pool per instance, or a lower max-instances |
| Throughput plateau with instances maxed out | max-instances cap |
Raise it… and recalculate the cost |
| Errors from the very first minute | A bug, not capacity | Fix it before testing again |
The connection arithmetic that catches everybody out: Cloud SQL db-f1-micro allows around 25 connections. If your application opens a pool of 10 connections per instance and Cloud Run scales to 5 instances, you ask for 50 connections and half of them fail. The solution is not a bigger database: it is a pool of 2-3 connections per instance, because each instance serves concurrent requests with a single connection most of the time.
- Reliability tests
Three drills. None takes more than an hour and all three are pure gold for the presentation.
6.1 Switch off a dependency and see what happens
# DRILL IN DEVELOPMENT, with the time recorded
echo "Drill start: $(date)" | tee -a docs/evidencias/ensayo-bd.log
# Stop the database
gcloud sql instances patch refugio-db --project=refugio-dev \
--activation-policy=NEVER --quiet
# Observe the behaviour
curl -s -o /dev/null -w "Home page: %{http_code} in %{time_total}s\n" "${URL}/"
curl -s -o /dev/null -w "API: %{http_code} in %{time_total}s\n" "${URL}/api/refugios"
curl -s -w "Health: %{http_code}\n" "${URL}/salud/arranque"
curl -s "${URL}/api/refugios" | head -c 300
# Restore
gcloud sql instances patch refugio-db --project=refugio-dev \
--activation-policy=ALWAYS --quietWhat behaviour you are looking for:
| Aspect | ❌ Bad | ✅ Good |
|---|---|---|
| Status code | 500 with a Python traceback | 503 with an explanatory JSON body |
| Message to the user | psycopg.OperationalError: could not connect... |
"The service is temporarily unavailable" |
| Response time | 30 s (waiting for the timeout) | <3 s (short timeout in the DB client) |
| Unaffected parts | Everything down | Home page and cached content still served |
| Startup probe | 200 (it lies) | 503 (Cloud Run stops sending traffic) |
| Logs | Exception traceback with no context | Structured ERROR with motivo and no personal data |
If your system behaves like the left-hand column, fix it: a short timeout and an exception handler that returns a 503 with a readable message are thirty lines of code and they add up in reliability and in the presentation.
6.2 Restore the backup and time it
INICIO=$(date +%s)
# 1. What backups do I have?
gcloud sql backups list --instance=refugio-db --project=refugio-dev \
--format="table(id, windowStartTime, status)"
BACKUP_ID=$(gcloud sql backups list --instance=refugio-db --project=refugio-dev \
--format="value(id)" --limit=1)
# 2. Restore onto a NEW instance (never onto the original in a drill)
gcloud sql instances create refugio-db-restaurada \
--project=refugio-dev --region=europe-west1 \
--database-version=POSTGRES_16 --tier=db-f1-micro
gcloud sql backups restore "${BACKUP_ID}" \
--restore-instance=refugio-db-restaurada \
--backup-instance=refugio-db --project=refugio-dev --quiet
# 3. VERIFY that the data is there (this is what almost nobody does)
./cloud-sql-proxy --port 55433 "refugio-dev:europe-west1:refugio-db-restaurada" &
psql -h 127.0.0.1 -p 55433 -U app -d reservas -c "
SELECT (SELECT count(*) FROM reserva) AS reservas,
(SELECT count(*) FROM refugio) AS refugios,
(SELECT max(creada_en) FROM reserva) AS ultima;"
FIN=$(date +%s)
echo "Measured RTO: $(( (FIN-INICIO)/60 )) minutes"
# 4. CLEAN UP — do not leave the restored instance running
gcloud sql instances delete refugio-db-restaurada --project=refugio-dev --quietStep 3 is what separates a real drill from a fake one. A restore that "completes successfully" but leaves an empty database is a disaster dressed up as a success. And step 4 is not optional: a forgotten restored instance is the most common way of doubling the bill without noticing.
Write down three numbers: how long it took, how much data would have been lost (RPO: the distance between the last backup and the moment of failure) and what problems you found. AlpinaShop found five problems nobody suspected in its 47-minute drill; you will find something similar.
6.3 Trigger the alert
You already did this in 08-03. If not, now is the time. And check all three things:
- That the incident opens in Monitoring.
- That the email (or whatever channel it is) arrives.
- That the content includes your first steps and is actually usable for acting.
- The deployment strategy
7.1 Environments and promotion
flowchart LR
DEV["Development<br/>refugio-dev<br/>Automatic on every push"]
PRUEBA["Testing<br/>e2e + load<br/>Against dev"]
PROD["Production<br/>refugio-prod<br/>With approval"]
DEV --> PRUEBA
PRUEBA -->|same image<br/>same digest| PROD
PROD --> CAN["Canary 10%"]
CAN -->|10 min OK| C50["50%"]
C50 -->|10 min OK| C100["100%"]
CAN -.->|error| REV["Rollback<br/>< 60 s"]
C50 -.->|error| REV
The non-negotiable principle: the same image, identified by its digest.
# Get the EXACT digest of the image tested in dev
DIGEST=$(gcloud run services describe refugio-web --region=europe-west1 \
--project=refugio-dev --format="value(spec.template.spec.containers[0].image)")
echo "Promoting: ${DIGEST}"
# Deploy to production WITHOUT traffic
gcloud run deploy refugio-web --image="${DIGEST}" \
--region=europe-west1 --project=refugio-prod \
--no-traffic --tag=candidataReferencing by digest (web@sha256:abc...) and not by tag (web:v1.2) removes all ambiguity: tags can be reassigned; a digest is the content.
7.2 The canary, step by step
SERVICIO="refugio-web"; REGION="europe-west1"; PROY="refugio-prod"
NUEVA=$(gcloud run revisions list --service=$SERVICIO --region=$REGION \
--project=$PROY --limit=1 --format="value(name)")
# Phase 1 — 10% for 10 minutes
gcloud run services update-traffic $SERVICIO --region=$REGION --project=$PROY \
--to-revisions="${NUEVA}=10"
# Observe THAT SPECIFIC new revision, not the whole service
gcloud logging read \
"resource.type=cloud_run_revision AND
resource.labels.revision_name=${NUEVA} AND severity>=ERROR" \
--project=$PROY --limit=20 --freshness=10mObjective promotion criteria, decided beforehand and not on the spot:
| Phase | Traffic | Minimum duration | Promote if… | Roll back if… |
|---|---|---|---|---|
| 1 | 10 % | 10 min | 5xx < 0.5 % and p95 < 1.2× the baseline | 5xx > 1 % or p95 > 2× the baseline |
| 2 | 50 % | 10 min | Same | Same |
| 3 | 100 % | 30 min of watching | Same | Same |
And a golden rule: if in doubt, roll back. Rolling back costs a minute; an incident costs an afternoon.
7.3 The rollback, rehearsed
#!/usr/bin/env bash
# scripts/revertir.sh — tested on 2026-10-28, takes 38 seconds
set -euo pipefail
SERVICIO="${1:-refugio-web}"; REGION="europe-west1"; PROY="refugio-prod"
ANTERIOR=$(gcloud run revisions list --service="$SERVICIO" --region="$REGION" \
--project="$PROY" --format="value(name)" --sort-by="~metadata.creationTimestamp" \
| sed -n '2p')
echo "Rolling back ${SERVICIO} → ${ANTERIOR}"
gcloud run services update-traffic "$SERVICIO" --region="$REGION" --project="$PROY" \
--to-revisions="${ANTERIOR}=100"
URL=$(gcloud run services describe "$SERVICIO" --region="$REGION" --project="$PROY" \
--format='value(status.url)')
sleep 5
curl -s -o /dev/null -w "Check: %{http_code}\n" "${URL}/salud/arranque"
echo "✅ Rolled back to ${ANTERIOR}"Rehearse it for real, with a stopwatch, at a quiet moment. A rollback procedure that has never been executed is a hypothesis, and the moment to discover that it does not work is not during an incident.
What a traffic rollback does NOT roll back — and you need to know this beforehand:
| Change | Rolled back with the traffic? | What to do |
|---|---|---|
| Application code | ✅ Yes | Nothing else |
| Environment variables | ✅ Yes (they live in the revision) | Nothing else |
| Database migration | ❌ No | Backwards-compatible migrations |
| Data already written | ❌ No | Restore a backup (much slower) |
| Infrastructure changes | ❌ No | terraform apply of the previous commit |
Hence the rule that avoids 90 % of disasters: migrations must be backwards compatible. Add columns, do not remove them. If a column has to go, do it in a later deployment, when no version in circulation uses it.
- The pre-launch checklist
You go through it in full, by hand, ticking for real.
Technical
- [ ] All tests pass in CI (unit, integration, end-to-end)
- [ ]
terraform planagainst production saysNo changes - [ ] The rebuild from scratch has been run successfully and is timed
- [ ] The migrations are applied and are backwards compatible
- [ ] The image deployed to production is the same one tested in development (identical digest)
- [ ] The health probes respond correctly
- [ ] There is no
TODO,FIXMEor debugprint()on the critical path
Security
- [ ]
scripts/auditoria-seguridad.shexits with 0 failures - [ ] The image has no critical vulnerabilities
- [ ] Valid TLS; TLS 1.0/1.1 rejected; HTTP redirects to HTTPS
- [ ] Security headers present
- [ ] Zero JSON keys; zero primitive roles on service accounts
- [ ] Database with no public IP; no public bucket without justification
- [ ] The secrets are in Secret Manager and not in the repository or its history
- [ ] The logs contain no personal data
Cost
- [ ] The budget is created with alerts at 50/90/100 %
- [ ] The projected cost fits inside the limit set in 08-01
- [ ]
max-instancesis limited on every service (spending cap) - [ ] Nothing is left running from the tests (restored instances, test load balancers, load jobs)
- [ ] The buckets have a lifecycle; Artifact Registry has a clean-up policy
- [ ] The BigQuery tables have partition expiration
Observability
- [ ] The dashboard shows the four signals with real data
- [ ] At least one alert has actually notified you when triggered
- [ ] The uptime check is active and green
- [ ] The SLO computes an error budget with a numeric value
- [ ] The logs are structured and correlatable by trace
- [ ] The notification channels are verified
Documentation
- [ ] The
READMElets another person start the project - [ ]
docs/arquitectura.mdreflects the system as it is today - [ ] There are ≥3 ADRs, and the diagrams match the decisions
- [ ]
docs/runbook.mdhas the operational procedures - [ ] The
diario.mdis up to date - [ ] The known technical debt is written down and prioritised
The minimum runbook
# Operations manual — RefugioReserva
## Contact details and access
Owner: <me>. Projects: refugio-dev, refugio-prod, refugio-datos.
Console: https://console.cloud.google.com/home/dashboard?project=refugio-prod
## P1 — The application returns 5xx errors
1. `gcloud logging read 'severity>=ERROR' --project=refugio-prod --limit=20 --freshness=15m`
2. Does it coincide with a deployment? `gcloud run revisions list --service=refugio-web --limit=3`
3. If it does → **roll back**: `./scripts/revertir.sh refugio-web` (38 s)
4. If not → check the DB: `gcloud sql instances describe refugio-db --format="value(state)"`
5. Write down what happened in `docs/diario.md`
## P2 — The application does not respond at all
1. `curl -sI https://refugioreserva.example`
2. Service alive? `gcloud run services describe refugio-web --format="value(status.conditions)"`
3. Does DNS resolve? `dig +short refugioreserva.example`
4. Valid certificate? `openssl s_client -connect refugioreserva.example:443`
## P3 — Restore the database
See `scripts/restaurar.sh`. **Measured RTO: 22 minutes. RPO: up to 24 h.**
## P4 — Runaway cost
1. Billing report → group by service and by the `componente` label
2. Usual suspects: forgotten instances, jobs in a loop, BigQuery queries with no partition filter
3. Immediate measure: lower `max-instances`; as a last resort, `terraform destroy` of dev
## P5 — Deploy a new version
`git push` to `main` → dev automatically → approve in Cloud Build → canary 10/50/100
- The launch and the first 24 hours
The launch
Pick a moment when you can be watching the screen for the following hour. Not a Friday night, and not just before going to bed.
# 1. Final check
./scripts/auditoria-seguridad.sh refugio-prod
terraform -chdir=infra/envs/prod plan # No changes
# 2. Promote the tested image
./scripts/promocionar.sh "$(git rev-parse --short HEAD)"
# 3. Canary 10%, wait, verify
# 4. 50%, wait, verify
# 5. 100%
# 6. Mark the moment
git tag -a v1.0.0 -m "Final project launch"
git push --tagsThe first 24 hours: what to watch and in what order
| Moment | What you watch | What would make you roll back |
|---|---|---|
| 0-15 min | 5xx errors in the logs, in real time | Any 5xx that did not exist before |
| 15-60 min | p95 latency, cold starts, instances | p95 at twice the baseline |
| 1-4 h | Full dashboard, first alerts, DB connection usage | Alerts firing |
| 4-24 h | Accumulated cost, error trend, SLO error budget | Projected cost above the limit |
| 24 h | All of the above + a manual test of the complete flow | — |
# The command you leave running in a terminal for the first hour
gcloud logging tail \
'resource.type="cloud_run_revision" AND severity>=WARNING' \
--project=refugio-prodWhen to declare success. Not when the deployment finishes, but when the five conditions are met:
- 24 hours with no anomalous 5xx errors.
- p95 latency inside the SLO target.
- No alert fired for a real cause.
- Projected daily cost inside the monthly limit.
- The complete flow tested by hand by you, in production, once.
Then, and only then: write the date in the README, take the screenshot of the dashboard in green and move on to 08-05.
- When something goes wrong: the incident runbook
Adapted from 07-06 to a one-person project.
flowchart TD
A["🚨 Something is wrong"] --> B["1. STABILISE<br/>Can I roll back? → roll back NOW<br/>Do not investigate first"]
B --> C{"Is it resolved?"}
C -- Yes --> D["2. Note the time, the symptom<br/>and what you did"]
C -- No --> E["3. DIAGNOSE<br/>logs → deployments → dependencies → quotas"]
E --> F["4. Mitigate<br/>even with a bodge"]
F --> D
D --> G["5. POST-MORTEM<br/>in docs/diario.md"]
G --> H["6. One concrete action<br/>so it does not happen again"]
Rule number one: stabilise before understanding. The natural impulse is to investigate the cause, and it is the wrong order. If you have a rollback that takes 38 seconds, roll back first and understand afterwards, with the system working and no rush.
The one-page post-mortem
# Incident 2026-11-04 — 5xx errors after the v1.1.0 deployment
**Duration:** 19:42 → 19:51 (9 minutes)
**Impact:** ~40 requests with a 500 error. No data loss.
**Detection:** error rate alert (arrived at 19:47, 5 min after the start)
## What happened
The v1.1.0 deployment introduced a query that used a column
(`opinion.magnitud`) created by migration 003, which had not been applied
in production. The application started up fine (the startup probe does not
touch that table) and only failed on requests to `/api/opiniones`.
## Timeline
- 19:42 Canary deployment at 10 %
- 19:44 First 500s in the logs (I did not look at them: I was watching the
dashboard, which at 10 % of traffic barely moved the needle)
- 19:47 The alert arrives
- 19:49 I run `./scripts/revertir.sh`
- 19:51 Verified: 0 errors
## Root cause
The pipeline deploys the application but **does not run the migrations**.
In development I apply them by hand, so it worked there.
## What worked
- The rollback: 38 seconds, exactly as rehearsed.
- The canary limited the impact to 10 % of the traffic.
## What did not work
- The alert took 5 minutes. At 10 % of traffic, the 5 % threshold over the
whole service is reached late.
- I was not watching the logs of the canary revision specifically.
## Actions
1. ✅ Add a migrations step to the pipeline, before the deployment.
2. ✅ Additional alert per revision, not only per service.
3. ⬜ Specific smoke test on `/api/opiniones` after deploying.A post-mortem in the repository is worth gold in an interview. It shows that you have had a real incident, that you resolved it with a procedure and that you learned something concrete. A project with no incident recorded means one of two things: that you have not used it, or that you did not notice.
- The worked example of RefugioReserva
Load test results
Run against development on 2026-10-27, with max-instances=5 and db-f1-micro:
✓ query 200
✓ booking 201 or 409
checks.........................: 99.31% ✓ 24893 ✗ 172
http_req_duration..............: avg=241ms min=61ms med=178ms max=8.9s
{ expected_response:true }...: avg=228ms
p(90)=402ms p(95)=712ms p(99)=2.41s
http_req_failed................: 0.68% ✓ 172 ✗ 25065
errores_negocio................: 0.68%
iterations.....................: 25065
vus_max........................: 100
✗ http_req_duration.............: p(99)<2000 → 2.41s FAILED
✓ http_req_failed...............: rate<0.01 → 0.0068 OKInterpretation, point by point:
| Observation | Diagnosis | Action |
|---|---|---|
| p50 = 178 ms | The typical experience is good | None |
| p95 = 712 ms | Inside the target | None |
| p99 = 2.41 s (threshold 2 s) | Cold starts when scaling from 1 to 5 instances | See below |
| 0.68 % of 5xx | 172 errors, all between minutes 3 and 4 | Investigated ↓ |
| max = 8.9 s | One specific request | The first cold start |
The 172 errors: the investigation.
$ gcloud logging read 'severity>=ERROR' --project=refugio-dev --limit=5 \
--format="value(jsonPayload.message)"
FATAL: remaining connection slots are reserved for non-replication superuser connectionsConnections exhausted. The arithmetic: a pool of 10 connections per instance × 5 instances = 50 connections requested against a limit of about 25 on db-f1-micro.
The fix, and why that one and not another:
# Before
pool = ConnectionPool(dsn, min_size=5, max_size=10)
# After
pool = ConnectionPool(dsn, min_size=1, max_size=3, timeout=5)The database was not made bigger (it would have cost money and it was not the problem): the pool was adjusted. Each Cloud Run instance serves up to 80 concurrent requests but spends most of its time waiting; 3 connections per instance × 5 instances = 15, comfortably inside the limit.
Second run after the adjustment:
http_req_duration.....: p(95)=634ms p(99)=1.82s ✓
http_req_failed.......: 0.00% ✓ 0 ✗ 26102 ✓
Cost of the test: €0.38Documented conclusion: the system sustains 100 concurrent users with a p99 under 2 seconds and no errors. The practical limit is max-instances=5, which is a deliberate cost cap, not a technical limitation. With max-instances=20 it would take about four times as much, and it would cost up to four times as much during a peak.
On the residual p99 of 1.82 s: those are cold starts. min-instances=1 would remove them, but it would cost about €12/month — the entire budget of the project. Decision: accepted and documented. The latency SLO was set on the p95 precisely because of this.
The two rollbacks
Rollback 1 — 2026-11-04, 19:49. The one in the post-mortem in section 10: migration 003 not applied in production. Detected by an alert after 5 minutes, rolled back in 38 seconds, impact limited to 10 % of the traffic by the canary. Corrective action: a migrations step in the pipeline.
Rollback 2 — 2026-11-11, 22:14. More interesting, because there was no error at all.
Symptom: after deploying v1.2.0 at 10 %, the p95 latency of the canary
revision went from 680 ms to 1,940 ms. Zero 5xx errors. Zero alerts.
Detection: I was watching it myself, comparing the canary revision with the
stable one on the dashboard, because the promotion criterion includes
"p95 < 1.2 × the baseline" and 1,940 does not meet it.
Cause: v1.2.0 added the average sentiment to the /api/refugios response
with a correlated subquery over `opinion`, with no index.
Decision: roll back without investigating further, at 22:14. The fix
(an index and a rewritten query) was deployed two days later.Why this rollback is worth more than the first one in a presentation: no automated system would have caught it. There were no errors, no alert fired, the availability SLO was still perfect. It was caught because an objective promotion criterion written in advance existed and because somebody compared against it. It is the practical demonstration of why the criteria are written beforehand: in the moment, with the new version deployed and a desire to be done, the temptation to say "1.9 seconds is not that bad either" is enormous.
Summary of evidence
| Test | Result | Evidence |
|---|---|---|
| Unit | 34 tests, 100 % pass | docs/evidencias/pytest.txt |
| Integration | 11 tests, including the concurrency one | docs/evidencias/pytest.txt |
| End to end | 4 tests against dev | Output in Cloud Build |
| Rebuild from scratch | ✅ 14 min 20 s | docs/evidencias/reconstruccion.log |
| Security audit | 0 failures, 1 warning (public web service, correct) | docs/evidencias/auditoria.txt |
| Image scan | 0 critical, 2 medium (base) | docs/evidencias/scan.txt |
| TLS | 1.2/1.3 yes, 1.0/1.1 no, 5 headers | docs/evidencias/tls.txt |
| Load | 100 VUs, p95 634 ms, 0 % error, €0.38 | docs/evidencias/k6.txt |
| DB outage | 503 with a readable message in 1.2 s | docs/evidencias/ensayo-bd.log |
| Restore | ✅ 22 minutes, data verified | docs/evidencias/restauracion.log |
| Alert triggered | Email in 6 min | docs/evidencias/alerta.png |
| Rollback rehearsed | 38 seconds | scripts/revertir.sh + log |
| Real incidents | 2, with post-mortem | docs/diario.md |
Common Mistakes and Tips
Writing tests that never fail. A test that always passes, even with the code broken, is worse than no test at all: it gives false confidence. Check every test by deliberately breaking what it tests.
Load testing production by accident. Get the URL right. And set max-instances low before you start: it is your spending cap.
Mistaking a 409 for an error. If your load test counts legitimate business responses as failures, your numbers mean nothing.
Restoring a backup and not checking the data. It is the most dangerous mistake in this lesson, because it produces a "successful" drill that actually demonstrates the opposite.
Leaving resources running after the tests. The restored instance, the test load balancer, the load-test DB. Set yourself a reminder and run an inventory after every testing session.
Deploying on a Friday night. A classic for a reason: if something goes wrong, either you fix it exhausted or you leave it broken over the weekend.
Rebuilding the image for production. That is deploying something you have never tested. Promote by digest.
Migrations that are not backwards compatible. They turn a 38-second rollback into a 22-minute restore with data loss.
Not having written promotion criteria. In the moment, it always looks like "it is not that bad". Write them beforehand, when you are not in a hurry.
Tip: save the output of everything. A docs/evidencias/ directory with the output of every test is what turns "my system is reliable" into "here are the numbers". In 08-05 you will be enormously grateful for it.
Tip: do the rebuild from scratch halfway through the project. Discovering in week 3 that a resource was missing from the code costs half an hour; discovering it the day before submission costs the submission.
Tip: one documented real incident is worth more than zero incidents. Do not hide the failures: tell them with their post-mortem. It is the most reliable signal that the system is real and that you know how to operate it.
Exercises
Exercise 1 — Build the test pyramid and test your infrastructure
Write for your project: at least 15 unit tests of the business logic, including edge cases; at least 5 integration tests against emulators or an ephemeral containerised database, applying the same migrations as in production and with a concurrency test on the critical point of your domain; and 3 end-to-end tests against development. Integrate them into the pipeline in the right order.
Then run the definitive infrastructure test: destroy the whole development environment, rebuild it from scratch, seed the data, deploy and verify that it works. Time it and document everything that broke along the way.
Exercise 2 — Audit security and measure capacity
Write and run your security audit script with at least eight automated checks (primitive roles, JSON keys, public buckets, public IP on the DB, secrets in the git history, credential files, services without authentication, audit logs). Fix everything that comes out red. Check the TLS, the accepted versions and the security headers. Scan your image.
Then run a staged load test against development with a spending cap, measuring p50/p95/p99 latency, error rate distinguishing system errors from business responses, active instances and the cost of the test itself. Interpret the results, apply at least one concrete adjustment and run it again to demonstrate the improvement.
Exercise 3 — Drill reliability, deploy and launch
Run the three reliability drills: switch off your database and document the exact behaviour (code, message, time, what keeps working), fixing the degradation if it is not graceful; restore a backup onto a new instance, verify the data and time the RTO; and trigger your alert, checking that it arrives.
Implement the canary deployment with objective promotion and rollback criteria written in advance, rehearse the rollback with a stopwatch, go through the entire pre-launch checklist, and launch. Document the first 24 hours.
Solutions
Solution 1 — Tests and rebuild of RefugioReserva
The final pyramid: 34 unit tests, 11 integration tests and 4 end-to-end tests. The test that justifies the whole project is test_no_hay_sobreventa_con_concurrencia from section 2.2, and its story is worth telling: the first implementation did not pass it.
Seven out of twenty threads managed to book the single place. The original implementation read the availability, decided and then inserted, with no lock: between the read and the insert, another six threads had read the same thing. It is exactly the failure that caused the fictional federation's seven overbookings a year — the original business problem, reproduced in the code.
With SELECT ... FOR UPDATE on the refuge row, the result was 1 and 19. This test, in the presentation, takes one slide and answers on its own the question "why Cloud SQL and not Firestore?".
The rebuild from scratch, run twice:
| Attempt | Date | Result | Findings |
|---|---|---|---|
| 1 | 2026-10-24 | ❌ Failed | 4 problems (below) |
| 2 | 2026-11-02 | ✅ 14 min 20 s | None |
The four problems from the first attempt, which is where the value of the exercise lies:
- The
refugio-session-keysecret was created empty. I had filled it in by hand in week 2 and had completely forgotten. The app started and failed on signing the first session. Fixed withrandom_password+secret_versionin Terraform. - A
depends_onon the peering was missing. Terraform tried to create Cloud SQL before the peering was ready. In the incrementalapplyruns it was never visible, because the peering had existed since day one. - The BigQuery dataset had been created by hand. Discovered because the function could not find the table. It did not have the
gestionado-porlabel, which was exactly the detector designed for this. terraform destroyfailed on the photos bucket because it had objects andforce_destroy = false. Correct in production, awkward in development: it was parameterised asforce_destroy = var.entorno == "dev".
None of the four would have been detected any other way. The rebuild from scratch is not just another test: it is the only one that validates that your IaC is real.
Solution 2 — Audit and load test of RefugioReserva
First run of the audit:
=== SECURITY AUDIT: refugio-prod ===
[1] Primitive roles ✅ No SA with owner/editor
[2] Account keys ✅ Zero JSON keys
[3] Public buckets ❌ 1 public bucket: gs://refugio-fotos-8f2a
[4] DB with a public IP ✅ refugio-db has no public IP
[5] Secrets in git ✅ No obvious secrets
[6] Credentials in the tree ✅ No credential files
[7] Cloud Run without auth ⚠️ refugio-web is public (correct)
[8] Audit logs ✅ Audit logs present
=== RESULT: 1 failure ===The public bucket. I had made it public in week 4 so that the thumbnails would be served directly, "temporarily". Four weeks later it was still like that, and on top of that I had left allUsers on the whole bucket, not just on the thumbnails prefix: the original photos were also accessible to anybody who guessed the name.
Fix: allUsers was removed, public_access_prevention = "enforced" was switched on, and the thumbnails moved to being served through signed URLs with a one-hour expiry, generated by the application.
def url_firmada(nombre_objeto: str, minutos: int = 60) -> str:
blob = _bucket.blob(nombre_objeto)
return blob.generate_signed_url(version="v4",
expiration=timedelta(minutes=minutos),
method="GET")It is exactly mistake 3 from 08-01 — leaving security until the end — showing up in its most typical form: something "temporary" that stays. And it was found by an eight-check script that took twenty seconds to run. Recorded in the 08-05 self-assessment as debt paid, with the date it was introduced and the date it was detected: four weeks of exposure.
TLS and headers, after adding the middleware:
tls1 rejects ✅
tls1_1 rejects ✅
tls1_2 accepts ✅
tls1_3 accepts ✅
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff
x-frame-options: DENY
content-security-policy: default-src 'self'; img-src 'self' data: https://storage.googleapis.com
referrer-policy: strict-origin-when-cross-originLoad: the results and their interpretation are those in section 11. The one-line summary, which is what goes into the presentation: 100 concurrent users, a p95 of 634 ms, 0 % errors, €0.38 of test cost, and a bottleneck found and fixed that was not the database but the size of the connection pool.
Solution 3 — Reliability and launch of RefugioReserva
Database outage drill, first run:
Home page: 200 in 0.31s ✅ still working (static content)
API: 500 in 30.02s ❌ 30 seconds and a Python traceback
Health: 200 ❌ IT LIES: it says it is ready and it is notTwo serious problems. The user waited 30 seconds to receive an incomprehensible error, and Cloud Run kept sending traffic to an instance that could not serve it because the startup probe did not check the database.
Fixes: connect_timeout=5 in the DSN, a global handler translating connection failures into a 503 with a JSON body, and the startup probe doing a SELECT 1.
Second run:
Home page: 200 in 0.28s ✅
API: 503 in 1.24s ✅ {"error":"servicio_no_disponible",
"mensaje":"We cannot check availability
right now. Try again in a few minutes."}
Health: 503 ✅ Cloud Run stops sending traffic
Photos: 200 ✅ the bucket is independent of the DB
Dashboard: 200 ✅ Looker Studio reads from BigQueryRestore: 22 minutes, with the data verified (2,000 bookings, 12 refuges, the last booking at 18:42 the previous day). Real RPO: up to 24 hours, because the backup is daily. Documented in the runbook, and noted as technical debt: enabling point-in-time recovery in production would bring the RPO down to minutes for a small additional cost.
Rollback rehearsed: 38 seconds, measured three times with results of 36, 38 and 41 seconds.
The launch, 2026-11-15 at 11:00 on a Saturday:
| Time | Action | Result |
|---|---|---|
| 10:40 | Security audit + terraform plan |
0 failures, No changes |
| 10:50 | Pre-launch checklist | 38/38 boxes |
| 11:02 | Canary at 10 % | 0 errors in 10 min, p95 611 ms |
| 11:14 | 50 % | 0 errors, p95 598 ms |
| 11:26 | 100 % | 0 errors |
| 11:30 | Tag v1.0.0 |
— |
| 12:30 | First hour watched | 0 errors, 47 requests |
| Day +1 | 24 hours | 0 5xx errors, daily cost €0.34 |
Success declared on 16 November at 11:30, with the five conditions met: 24 hours with no anomalous errors, p95 inside the SLO, no real alert, a projected cost of €10.20/month against the €12 limit, and the complete flow tested by hand in production.
The two later rollbacks — the one on 4 November and the one on the 11th — are in section 11 with their post-mortem. Neither of them could have been handled in under a minute without the runbook written and rehearsed beforehand.
Conclusion
Your project no longer just works: you know it works, and you have the evidence.
You can tell apart the three levels of "finished" and which one you are at, with nine operational boxes that define the third one unambiguously.
You have the test pyramid sized for a project of this scale — around 30 unit, 10 integration, 4 end-to-end — you know what is worth testing and what is not, and you have the test that is worth all the others put together: the concurrency test on the critical point of your domain, the one that fails with the naive implementation and that justifies your database choice on its own. And you know how to test against free emulators and ephemeral containerised databases, applying the same migrations as in production.
You know how to test the infrastructure: static validation, the plan published in the pull request with a read-only account, tflint and checkov with their exceptions documented instead of chased, and above all the definitive test: destroying the environment and recreating it from scratch. It is the only one that uncovers the secret you filled in by hand, the resource created from the console, the implicit dependency in the wrong order and the migration that only works against a database that already exists.
You have an executable security checklist with eight automated checks that runs in twenty seconds and that in the example found a public bucket that had been exposed for four weeks. And you know how to verify TLS, the accepted versions, the five headers that matter and the effective permissions — including proving that an account does not have a dangerous permission.
You know how to run an honest, cheap load test: against development, with a spending cap, staged, distinguishing system errors from legitimate business responses, measuring percentiles rather than averages, and including the cost of the test itself among the metrics. And you know how to interpret it: a p99 far above the p95 means cold starts; 5xx errors as load rises usually mean database connections, and the solution is not a bigger database but a smaller pool.
You have drilled reliability for real: switching off the database to discover that your error took thirty seconds and showed a Python traceback, restoring a backup verifying the data — the step that turns a real drill into a fake one — and timing it, and triggering the alert to check that it arrives.
You have a deployment strategy with the same image promoted by digest, a canary at 10/50/100 with objective criteria written in advance, and a rollback rehearsed with a stopwatch. And you know what a traffic rollback does not roll back — migrations, data, infrastructure — from which comes the rule that avoids most disasters: backwards-compatible migrations, always.
You have the pre-launch checklist in its five blocks, the runbook with its numbered procedures, the script for the first 24 hours with what to watch and in what order, the five conditions for declaring success, and the incident runbook with its rule number one: stabilise before understanding.
And you have the RefugioReserva example with its real numbers: 14 minutes to rebuild, 22 to restore, 38 seconds to roll back, a p95 of 634 ms with 100 users, €0.38 of test cost, a public bucket found and closed, and two rollbacks — one caught by an alert and the other caught only because an objective written criterion existed, with no error, no alert, and the SLO in green.
In the next lesson, 08-05, the work changes in nature: you stop building and start communicating. You are going to prepare the presentation for three different audiences, structure fifteen minutes slide by slide, put together a timed demo with a recorded plan B, talk about your numbers — because "this costs me €10 a month" is worth more than any adjective — write the deliverable documentation with its templates, assess yourself against the 08-01 rubric with brutal honesty, prepare the answers to the questions you are going to be asked, and close with the final clean-up: a terraform destroy followed by an apply that works again, which is the best possible ending this project can have.
Google Cloud Platform (GCP) Course
Module 1: Introduction to Google Cloud Platform
- What is Google Cloud Platform?
- Setting Up Your GCP Account
- A Tour of the GCP Console
- Projects, Resource Hierarchy and Billing
- Regions, Zones and the Shared Responsibility Model
- Cloud Shell and the gcloud CLI
Module 2: Core GCP Services
- Compute Engine: Virtual Machines on Google Cloud
- Cloud Storage: Object Storage
- Cloud SQL: Managed Relational Databases
- App Engine: Platform as a Service
- Google Kubernetes Engine (GKE)
- NoSQL Databases: Firestore, Bigtable and Spanner
- How to Choose the Right Compute Service
Module 3: Networking and Security
- VPC Networks
- Cloud Load Balancing
- Cloud CDN
- Identity and Access Management (IAM)
- Cloud Armor
- Secrets and Encryption: Secret Manager and Cloud KMS
- Cloud DNS, TLS Certificates and Publishing Services Securely
Module 4: Data and Analytics
- BigQuery: The Analytical Data Warehouse
- Cloud Dataflow: Batch and Streaming Data Processing
- Cloud Dataproc: Managed Spark and Hadoop
- Cloud Pub/Sub: Asynchronous Messaging
- Cloud Data Fusion: Code-Free Data Integration
- Orchestrating Pipelines with Cloud Composer and Workflows
- Data Governance and Dashboards with Dataplex and Looker Studio
Module 5: Machine Learning and AI
- Vertex AI: The Machine Learning Platform on GCP
- AutoML: Custom Models Without Writing Code
- TensorFlow on GCP: Training and Serving Models
- Natural Language API
- Vision API
- Generative AI on Vertex AI: Gemini Models and Embeddings
- MLOps: From Model to Product with Vertex AI Pipelines
Module 6: DevOps and Monitoring
- Cloud Build: Continuous Integration on GCP
- Cloud Source Repositories and Source Code Management
- Cloud Functions: Serverless Functions
- Cloud Monitoring (formerly Stackdriver): Metrics, Dashboards and Alerts
- Cloud Deployment Manager and Native Infrastructure as Code
- Cloud Logging and Cloud Trace: Logs, Traces and Diagnostics
- Terraform on GCP: Infrastructure as Code in Practice
Module 7: Advanced GCP Topics
- Hybrid and Multicloud with Anthos
- Serverless Computing with Cloud Run
- Advanced Networking: Shared VPC, Peering and Hybrid Connectivity
- Security Best Practices
- Cost Management and Optimization
- Reliability: SLOs, High Availability and Disaster Recovery
- Governance at Scale: Organization, Policies and Auditing
