MercadoFresco's code now lives in a repository with rules, short branches and pull requests that somebody
reviews. But reviewing a PR is a person reading a diff, and there are questions nobody can answer by
reading: do the 214 tests pass? does the project start on a clean machine, or only on Luis's laptop where
there is a psycopg2 installed by hand in 2024? does any of the 87 dependencies have a vulnerability
published this week? has a key slipped into a configuration file?
AWS CodeBuild answers all of that automatically. It is a managed, serverless build service: it starts
a clean container, clones your code, runs whatever you tell it in a buildspec.yml, saves the results and
shuts down, charging per minute of execution and not for existing. This is where "it works on my laptop"
stops being an argument, because from now on the only thing that counts is whether it works in an empty
container nobody has touched.
Cost warning. CodeBuild charges per minute according to the compute size: in
eu-west-1, aBUILD_GENERAL1_SMALLaround 0.005 USD/min and aMEDIUMaround 0.010, with 100 free minutes a month of the small type. With 20 daily builds of 4 minutes on MEDIUM, MercadoFresco will pay around 24 USD a month. That is not the expensive part: it is a 22-minute build nobody optimises, or building inside the VPC without being aware of the NAT cost, which we will see at the end. Fictitious data.
Contents
- What continuous integration is and what problem it solves
- Anatomy of a CodeBuild project
- The service role and its permissions
- MercadoFresco's
buildspec.yml, commented - The phases, one by one
- Environment variables and secrets without leaks in the logs
- Unit tests and test and coverage reports
- What to do when a test fails
- Integration tests: mocking or the Aurora clone
- Static analysis, dependencies and leaked secrets
- Quality as a gate, not as a report
- Artefacts, cache and parallel builds
- CodeBuild inside the VPC, with its cost warning
- Debugging a build that fails
- Metrics, alarms and real cost
- Common mistakes and tips
- Exercises
- Conclusion
What continuous integration is and what problem it solves
Continuous integration means that every change is integrated into the shared branch and verified automatically, several times a day. The important part is the second half: it is verified automatically. Without verification, "continuous integration" is just pushing code often.
| Symptom | Real cause | What eliminates it |
|---|---|---|
| "It works on my laptop" | The laptop has accumulated state | A clean container on every build |
| "I forgot to run the tests" | Running them is voluntary | Automatic execution on every push |
| "That test has been failing for months" | Nobody looks at the result | A red build blocks the PR |
| "It worked before your change" | Nobody knows when it broke | Build history per commit |
| "I do not know what version is deployed" | A folder is deployed, not an artefact | Versioned, immutable artefact |
Notice the pattern: in every case the fix is taking away from people the responsibility of remembering. A process that depends on individual discipline fails on the day there is a rush, which is exactly the day it is needed most. The flow we are going to build:
flowchart LR
A[Push or PR] --> B[Clean container]
B --> C[install] --> D[pre_build<br/>secrets, lint, scans]
D --> E[build<br/>tests + packaging] --> F[post_build]
F --> G{All green?}
G -->|Yes| H[Artefact in<br/>mercadofresco-artefactos] --> J[Deployment: 08-03]
G -->|No| I[Red: alertas-mercadofresco] --> K[The PR is not merged]
Anatomy of a CodeBuild project
A project is the definition of how to build something. Its seven pieces:
| Piece | What it defines | MercadoFresco's decision |
|---|---|---|
| Source | Where the code comes from | GitHub via conn-mercadofresco-github |
| Environment | Image, compute size, privileges | amazonlinux2-x86_64-standard:5.0, MEDIUM |
| Service role | What the build can do in AWS | rol-codebuild-mercadofresco-tienda |
| Buildspec | The commands to run | buildspec.yml at the root of the repository |
| Artefacts | What is saved and where | ZIP in mercadofresco-artefactos |
| Cache | What is reused between builds | Local (custom + docker layer) |
| Logs | Where the output goes | /aws/codebuild/build-mercadofresco-tienda |
About the environment image. The AWS managed ones come with Python, Node, Java, Go, the CLI and Docker preinstalled, and they update themselves. Your own image in ECR makes sense when installing system dependencies takes more than two or three minutes on every build: you move that time into an occasional image build. MercadoFresco starts with the managed one.
About the compute size. It is the decision with most impact on cost and duration, and intuition lies:
| Size | vCPU / RAM | USD/min | Duration at MercadoFresco | Cost per build |
|---|---|---|---|---|
SMALL |
2 / 3 GB | 0.005 | 9 min 40 s | 0.048 USD |
MEDIUM |
4 / 7 GB | 0.010 | 4 min 10 s | 0.042 USD |
LARGE |
8 / 15 GB | 0.020 | 3 min 30 s | 0.070 USD |
MEDIUM is cheaper than SMALL here, because double the price per minute is offset by more than double
the speed: the tests run with pytest -n auto and take advantage of the 4 vCPUs. From MEDIUM to LARGE, by
contrast, the time drops little and the cost rises, because the work is no longer parallelisable. You
have to measure, not assume: the best size depends on whether your build can use the cores you give it.
About privilegedMode. It is only needed for building Docker images, because the daemon needs
privileges. Enable it only when you need it: a privileged container running code from a repository has a
larger attack surface.
aws codebuild create-project \
--name build-mercadofresco-tienda \
--source '{"type": "GITHUB",
"location": "https://github.com/mercadofresco/mercadofresco-tienda.git",
"buildspec": "buildspec.yml", "gitCloneDepth": 1, "reportBuildStatus": true}' \
--environment '{"type": "LINUX_CONTAINER",
"image": "aws/codebuild/amazonlinux2-x86_64-standard:5.0",
"computeType": "BUILD_GENERAL1_MEDIUM", "privilegedMode": false,
"environmentVariables": [
{"name": "BUCKET_ARTEFACTOS", "value": "mercadofresco-artefactos", "type": "PLAINTEXT"}]}' \
--artifacts '{"type": "S3", "location": "mercadofresco-artefactos",
"packaging": "ZIP", "namespaceType": "BUILD_ID"}' \
--cache '{"type": "LOCAL", "modes": ["LOCAL_CUSTOM_CACHE", "LOCAL_DOCKER_LAYER_CACHE"]}' \
--service-role arn:aws:iam::111122223333:role/rol-codebuild-mercadofresco-tienda \
--timeout-in-minutes 20 --queued-timeout-in-minutes 30 \
--tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
Key=Componente,Value=cicd Key=Propietario,Value=luis Key=CentroCoste,Value=plataforma \
--profile mercadofresco-dev --region eu-west-1Two parameters deserve attention. gitCloneDepth: 1 clones only the last commit: Luis's 1,847
commits add 40 seconds to every build without contributing anything — unless you need the history for
git describe, in which case set 0. And timeout-in-minutes 20 is an emergency brake: a hung build
would consume the default 60-minute timeout, and those are billed minutes.
The service role and its permissions
The build acts in AWS with the service role, and here the least privilege of 04-01 matters a lot: this
role is used by code that changes on every commit. With PowerUserAccess, anybody who can merge a PR
could do anything in the account.
{
"Version": "2012-10-17",
"Statement": [
{ "Sid": "Registros", "Effect": "Allow",
"Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
"Resource": "arn:aws:logs:eu-west-1:111122223333:log-group:/aws/codebuild/build-mercadofresco-*:*" },
{ "Sid": "ArtefactosSoloEnSuPrefijo", "Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject", "s3:GetObjectVersion"],
"Resource": "arn:aws:s3:::mercadofresco-artefactos/tienda/*" },
{ "Sid": "ConfiguracionNoSensible", "Effect": "Allow",
"Action": ["ssm:GetParameters", "ssm:GetParameter"],
"Resource": "arn:aws:ssm:eu-west-1:111122223333:parameter/mercadofresco/construccion/*" },
{ "Sid": "SoloElSecretoDePruebas", "Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:mercadofresco/pruebas/*" },
{ "Sid": "InformesDePruebas", "Effect": "Allow",
"Action": ["codebuild:CreateReportGroup", "codebuild:CreateReport",
"codebuild:UpdateReport", "codebuild:BatchPutTestCases",
"codebuild:BatchPutCodeCoverages"],
"Resource": "arn:aws:codebuild:eu-west-1:111122223333:report-group/build-mercadofresco-*" }
]
}What matters is in the Resources, not in the actions. The role cannot read
mercadofresco/produccion/rds/mfadmin: only secrets under mercadofresco/pruebas/. That is the
difference between a compromised build that spoils a test environment and one that holds the production
password. A build project never needs production credentials, and if you think it does, it means you
are deploying from the build instead of from CodeDeploy (08-03).
MercadoFresco's buildspec.yml, commented
This file lives at the root of mercadofresco-tienda, versioned alongside the code, and it is the heart
of the lesson:
version: 0.2
# Variables. The ones under 'variables' are public and appear in the logs.
# Those under 'parameter-store' and 'secrets-manager' are resolved at run time
# and CodeBuild masks them in the output.
env:
variables:
BUCKET_ARTEFACTOS: "mercadofresco-artefactos"
COBERTURA_MINIMA: "75"
parameter-store:
URL_API_PRUEBAS: "/mercadofresco/construccion/url_api_pruebas"
HOST_AURORA_PRUEBAS: "/mercadofresco/construccion/host_aurora_pruebas"
secrets-manager:
CLAVE_PASARELA_PRUEBAS: "mercadofresco/pruebas/pasarela:clave_api"
exported-variables:
- VERSION_APP
- COMMIT_CORTO
phases:
install:
runtime-versions:
python: 3.12
commands:
- pip install --upgrade pip
- pip install -r requirements.txt -r requirements-dev.txt
- pip install --quiet bandit pip-audit ruff
pre_build:
commands:
- export COMMIT_CORTO=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c1-7)
- export VERSION_APP=$(cat VERSION)-${COMMIT_CORTO}
- echo "Building version ${VERSION_APP}"
# Gates ordered from fastest to slowest
- ruff check app/ tests/ && ruff format --check app/ tests/
- git secrets --scan || (echo "SECRET DETECTED IN THE CODE"; exit 1)
- pip-audit --strict --desc
- bandit -r app/ -ll -f json -o informe-bandit.json
build:
commands:
# Unit tests in parallel, with a JUnit report and coverage.
- |
pytest tests/unitarias -n auto \
--junitxml=informes/junit-unitarias.xml \
--cov=app --cov-report=xml:informes/cobertura.xml \
--cov-fail-under=${COBERTURA_MINIMA}
# Integration tests against mocked services (moto).
- pytest tests/integracion --junitxml=informes/junit-integracion.xml
# Packaging: only what is needed in production.
- mkdir -p paquete
- pip install -r requirements.txt --target paquete/ --quiet
- cp -r app/ scripts/ appspec.yml paquete/
- echo "${VERSION_APP}" > paquete/VERSION
- cd paquete && zip -rq ../mercadofresco-tienda-${VERSION_APP}.zip . && cd ..
post_build:
# finally runs EVEN IF the previous commands fail.
commands:
- |
if [ "${CODEBUILD_BUILD_SUCCEEDING}" = "1" ]; then
aws s3 cp mercadofresco-tienda-${VERSION_APP}.zip \
s3://${BUCKET_ARTEFACTOS}/tienda/${VERSION_APP}/ \
--metadata commit=${CODEBUILD_RESOLVED_SOURCE_VERSION},version=${VERSION_APP}
echo "Artefact published: ${VERSION_APP}"
else
echo "Build failed: no artefact is published"
fi
finally:
- echo "Total duration: $((SECONDS / 60))m $((SECONDS % 60))s"
# Reports are registered in CodeBuild and seen in the console with their history.
reports:
pruebas-unitarias:
files: ["informes/junit-unitarias.xml"]
file-format: JUNITXML
cobertura:
files: ["informes/cobertura.xml"]
file-format: COBERTURAXML
artifacts:
files: [mercadofresco-tienda-*.zip]
discard-paths: yes
cache:
paths: ["/root/.cache/pip/**/*", ".ruff_cache/**/*"]The phases, one by one
| Phase | What it is for | If it fails | Duration |
|---|---|---|---|
install |
Runtimes and tools | Aborts; only finally runs |
20-90 s |
pre_build |
Credentials, variables, quick checks | Aborts before spending on tests | 30-60 s |
build |
Compile, test, package | Aborts, but post_build does run |
2-4 min |
post_build |
Publish, notify, clean up | Marks the build as failed | 10-30 s |
Two behaviours confuse everybody the first time. post_build runs even if build fails, and it is
deliberate: you want to publish the test reports precisely when they have failed. That is why the
post_build above checks CODEBUILD_BUILD_SUCCEEDING; without that check you would publish to S3 the
artefact of a build whose tests failed, the most dangerous mistake in this lesson. And the order
inside pre_build is not accidental: ruff takes 2 seconds, the secret scan 5, bandit 15 and
pip-audit 20, so a formatting error fails in 2 seconds instead of 40. Always order your gates by
increasing cost.
Environment variables and secrets without leaks in the logs
The logs go to CloudWatch Logs and are read by anybody with permission on that log group. Any value you
print there is visible, and a set -x or a debugging echo can print a secret without anybody realising
it.
| Type | Where it is stored | Masked? | What for |
|---|---|---|---|
PLAINTEXT |
Project definition | No | Public configuration |
PARAMETER_STORE |
Parameter Store | Yes (if SecureString) |
Per-environment configuration |
SECRETS_MANAGER |
Secrets Manager | Yes | Passwords, API keys |
The syntax of the secrets-manager block — VARIABLE_NAME: secret-name:json-key:label — is the one
people get wrong most often. The colons separate the secret's name from the specific key inside the
JSON: without :clave_api, the variable would receive the whole JSON — {"clave_api": "...", "url": "..."} — and your code would fail with a confusing error. And if the secret has the rotation from 04-03,
AWSCURRENT guarantees that you are using the current version.
How CodeBuild masks. It replaces with *** the values resolved from Parameter Store and Secrets
Manager. But it is string matching, not magic, and it escapes in three cases: if you transform the
value (echo $CLAVE | base64), what is printed no longer matches and comes out in the clear; if the
secret ends up in a file that you then print with cat; and if a tool includes it in an error
message, such as a full connection string in a driver's exception.
Three practical rules: never set -x in a buildspec handling secrets; redirect to /dev/null
whatever might carry credentials when you do not need the output; and never put secrets in PLAINTEXT
variables, not even "just to test", because they stay in the project definition and in CloudTrail.
Unit tests and test and coverage reports
MercadoFresco's tests live in tests/ and test what matters. A real example — the idempotency of the
consumer we built in 07-05, exactly the kind of property nobody verifies by hand:
# tests/unitarias/test_idempotencia.py
import pytest
from app.pedidos import process_order_message
def test_same_message_twice_charges_only_once(idempotency_table, fake_gateway):
"""The module 7 scenario: SQS delivers the same order twice."""
message = {"id_pedido": "PED-2026-00841", "importe_eur": 48.20,
"clave_idempotencia": "PED-2026-00841:cobro"}
first = process_order_message(message)
second = process_order_message(message) # the duplicate
assert first["estado"] == "cobrado"
assert second["desde_cache"] is True
assert fake_gateway.number_of_charges == 1 # THE assertion that matters
assert fake_gateway.total_charged == pytest.approx(48.20)
def test_invalid_slot_rejected_without_touching_aurora(fake_aurora):
with pytest.raises(ValueError, match="franja_entrega"):
process_order_message({"id_pedido": "X", "franja_entrega": "noche"})
assert fake_aurora.writes == 0 # rejected BEFORE writingNotice what is being asserted. It does not check "that the function returns something": it checks that the gateway was called exactly once and that an invalid piece of data never got written to Aurora. A test that cannot fail when the behaviour breaks is not a test, it is decoration.
The reports block turns the JUnit XML into something CodeBuild understands: in the console you will
see, per build, how many tests passed, which ones failed with their trace, and the historical trend.
Without it you would have to hunt for the failure in 2,000 lines of log.
About --cov-fail-under=75. Coverage measures which lines the tests executed, not whether they did so
well: you can have 95 % without a single useful assertion. Even so it is useful for spotting code that
nobody runs, and a threshold that fails the build stops it dropping without anybody noticing. Marta's
criterion is sensible: the threshold starts at the current value and can only go up, never lowered to
"unblock" a PR. Lowering it once is lowering it forever.
What to do when a test fails
It is the shortest and most important section of the module, because everything else depends on it.
| Situation | Correct reaction | Reaction that ruins the system |
|---|---|---|
| It fails because of a badly written test | Fix the test in the same PR | Mark it @pytest.mark.skip |
| It fails intermittently | Investigate today: usually a real race | Retry until it passes |
| It fails because an external service is down | Mock it; a unit test does not go to the network | Disable the test |
| It fails and there is a rush to deploy | Revert the commit and deploy the previous one | Skip the build |
Intermittent tests are the biggest danger, more so than the ones that always fail. One that always fails gets fixed; one that fails one time in fifteen teaches the team to press "retry", and from then on a real failure gets retried too until it passes. They usually point at something true: dependence on execution order, a fixed wait instead of a condition, or a race condition that also exists in production. Treat it as a high-priority failure or delete it, but do not leave it flickering. And the rule that holds everything up: a red build blocks the merge; without it, the team learns within two weeks that red is optional. That gate is closed completely in 08-04.
Integration tests: mocking or the Aurora clone
Unit tests do not touch the network. Integration tests do, and there are two strategies with very different trade-offs:
| Strategy | Fidelity | Speed | Cost | When |
|---|---|---|---|---|
| Mocked services (moto, LocalStack) | Medium | Seconds | 0 | Most cases |
| Real container (Postgres in Docker) | High for SQL | 30-60 s | 0 | Queries and schema |
aurora-mf-pruebas-luis clone |
Total | Minutes | ENI + NAT | Migrations and performance |
With moto, calls to AWS are intercepted in memory:
# tests/integracion/test_flujo_pedido.py
import boto3, json
from moto import mock_aws
from app.pedidos import confirm_order
@mock_aws
def test_confirmed_order_enqueues_and_publishes_event():
sqs = boto3.client("sqs", region_name="eu-west-1")
sns = boto3.client("sns", region_name="eu-west-1")
queue_url = sqs.create_queue(QueueName="cola-mercadofresco-pedidos")["QueueUrl"]
topic = sns.create_topic(Name="mercadofresco-pedido-confirmado")["TopicArn"]
result = confirm_order(
{"id_cliente": "CLI-4471", "lineas": [{"sku": "FRUT-0012", "uds": 3}],
"franja_entrega": "tarde"}, queue_url=queue_url, topic_arn=topic)
messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
body = json.loads(messages["Messages"][0]["Body"])
assert body["id_pedido"] == result["id_pedido"]
assert body["franja_entrega"] == "tarde"
assert body["version_evento"] == "2" # see 08-05This verifies the complete flow — confirm, enqueue, publish — in less than a second and at no cost, and for 90 % of cases it is enough. The Aurora clone is reserved for what mocking cannot tell you: whether a migration takes 4 seconds or 40 minutes over 340 real GB, whether a new index is actually used, whether a query degrades at production volume. That requires a VPC, and the VPC has a cost.
Static analysis, dependencies and leaked secrets
| Tool | What it looks for | Blocking | Cost |
|---|---|---|---|
ruff |
Style errors, unused imports, obvious bugs | Yes | 0 |
git-secrets |
Credential patterns in the code | Yes, always | 0 |
pip-audit |
Known CVEs in dependencies | Yes, with nuances | 0 |
bandit |
Insecure code patterns | Report | 0 |
| Amazon Inspector | Vulnerabilities in ECR images and Lambda | Configurable | Per resource |
git-secrets is the net that was missing in 08-01. The .gitignore protects you from carelessness;
this also protects you from git add -f:
- git secrets --register-aws # AWS patterns: AKIA, secret keys
- git secrets --add 'sk_live_[0-9a-zA-Z]{24}' # payment gateway keys
- git secrets --add 'postgres://[^:]+:[^@]+@' # connection strings with password
- git secrets --add --allowed 'AKIAIOSFODNN7EXAMPLE' # the documentation one, allowedWhether it should block is not up for discussion: a false positive costs adding an --allowed exception;
a false negative costs rotating credentials in production.
pip-audit with nuances. It fails if any dependency has a known CVE, which is correct but has a
failure mode you have to anticipate: a CVE is published in a popular library on a Tuesday afternoon and
all your builds go red, including the one for an urgent fix that has nothing to do with it. Avoid the
easy way out of --ignore-vuln, which turns into a list that only grows; the right thing is to fail and
alert through alertas-mercadofresco with the build identifier, so that there is a procedure and not a
permanent exception.
Amazon CodeGuru Reviewer deserves a mention: it analyses code with models trained on Amazon's own code and detects what a linter does not see — resource leaks, race conditions, incorrect use of the AWS APIs. It is paid for and its recommendations are suggestions, not gates: for three people it is a dispensable luxury; for fifty it may pay off.
Quality as a gate, not as a report
Here is the criterion that separates a useful pipeline from quality theatre. A check is either a gate — it fails the build and blocks the merge — or a report nobody is going to read. There is no middle ground: a report that blocks nothing is ignored by the third day.
| Check | Gate? | Why |
|---|---|---|
| Unit tests | Yes | A red test is broken behaviour |
| Coverage < 75 % | Yes | Stops it dropping without anybody noticing |
| Secrets detected | Yes | The cost of a false negative is rotating in production |
ruff |
Yes | Fast and objective; arguing about style in PRs is wasted time |
| CVEs in dependencies | Yes, with an alert | With a procedure for Tuesday afternoon |
bandit |
No, report | Many false positives; reviewed weekly |
| Cyclomatic complexity | No, report | Arbitrary threshold; generates sterile arguments |
The rule for deciding: if you are not willing to stop a deployment because of it, it is not a gate.
And if it is not, be honest about the fact that almost nobody will read it: give it an explicit moment of
review — "on Mondays we look at the bandit report" — or take it out of the buildspec.
Artefacts: the application package and the images
The result of the build is an immutable, versioned artefact, and this is the most important
conceptual change compared with Luis's scp: you no longer deploy "whatever is in the folder", you deploy
mercadofresco-tienda-1.5.0-a3f9c21.zip, a file that exists, has an identifier, can be downloaded six
months later and is exactly what was tested.
The mercadofresco-artefactos bucket is created with three non-negotiable properties: versioning,
which CodePipeline requires and which protects against accidental overwriting; encryption with
alias/mercadofresco-datos and BucketKeyEnabled to reduce the KMS calls, which are billed separately;
and a lifecycle rule that expires the artefacts.
aws s3api put-bucket-versioning --bucket mercadofresco-artefactos \
--versioning-configuration Status=Enabled --profile mercadofresco-dev
aws s3api put-bucket-lifecycle-configuration --bucket mercadofresco-artefactos \
--lifecycle-configuration '{"Rules":[{
"ID":"caducar-artefactos-antiguos","Status":"Enabled",
"Filter":{"Prefix":"tienda/"},
"Expiration":{"Days":90},
"NoncurrentVersionExpiration":{"NoncurrentDays":30}}]}' \
--profile mercadofresco-devWithout a lifecycle rule, twenty daily artefacts of 45 MB are 27 GB a year of ZIP that nobody will ever deploy; with 90 days you have plenty of room to go back and the cost stays in cents.
Container images. If instead of a ZIP you build an image, you need privilegedMode: true, a
docker login against ECR in pre_build with aws ecr get-login-password, a docker build tagging with
${VERSION_APP} and a docker push in post_build. On this, only the essentials: ECR, layers, image
scanning and container deployment are module 10. Here it is enough to know that CodeBuild builds images
just as well as ZIPs and that LOCAL_DOCKER_LAYER_CACHE reuses the layers.
Dependency cache and its real effect
Installing 87 packages from PyPI takes 45-70 seconds on every build: over 20 daily builds, that is 20 minutes a day downloading the same thing.
| Cache type | Where it lives | Measured saving | Limitation |
|---|---|---|---|
Local (LOCAL_CUSTOM_CACHE) |
The build host | 40-55 s | Only if it lands on the same host |
| S3 | The bucket you specify | 25-40 s | You pay to upload and download |
| Docker layers | The build host | 1-3 min | Only with privilegedMode |
The real numbers: with no cache, 4 min 10 s; on a hit, 3 min 20 s; on a miss, 4 min 12 s. The saving is real but modest, and it is worth saying so because expectations are exaggerated: the local cache depends on the build landing on a host that has it, and with 20 a day the hit rate is around 60 %.
Two warnings. A poisoned cache is worse than no cache: if you store site-packages instead of the
pip cache, you drag old versions along and debugging it takes hours — always cache ~/.cache/pip. And
if you suspect the cache, invalidate it: aws codebuild invalidate-project-cache is the first command
to try when a build fails inexplicably.
Batch builds
When the suite grows, the batch block of the buildspec defines a graph of builds: several —
pruebas_unitarias, pruebas_integracion, escaneos — run at the same time, and an empaquetado
stage with depend-on waits for the three; fast-fail: true cancels the rest as soon as one fails. A
sequential build of 12 minutes can come down to 6, but the total cost in minutes does not drop — you
pay the same, just in parallel —; what you buy is developer waiting time, usually worth far more than
0.04 USD. Start simple: do not parallelise a 4-minute build.
CodeBuild inside the VPC, with its cost warning
By default CodeBuild runs outside your VPC, with internet access and without access to your private
subnets: it cannot talk to aurora-mf-pruebas-luis, which is in snet-mercadofresco-datos-a.
aws codebuild update-project --name build-mercadofresco-tienda \
--vpc-config '{"vpcId": "vpc-mercadofresco",
"subnets": ["snet-mercadofresco-app-a", "snet-mercadofresco-app-b"],
"securityGroupIds": ["sg-mercadofresco-construccion"]}' \
--profile mercadofresco-dev --region eu-west-1Three things to get right. Private subnets, always: in a public one with no public IP the build will
have no internet access and pip install will hang until the timeout, and it is the most common and most
baffling failure. Egress through NAT to reach PyPI, or VPC endpoints for the AWS APIs. And a
security group sg-mercadofresco-construccion with an inbound rule on sg-mercadofresco-basedatos
allowing 5432 from it: never open the database by CIDR.
Important cost warning. Putting CodeBuild in the VPC has two costs that come as a surprise on the bill. The ENIs: every build creates and destroys a network interface, which adds 30-60 seconds of start-up to each execution — with 20 a day, 20 minutes a day billed for doing nothing. And the NAT gateway: 0.048 USD/hour plus 0.048 USD per GB; downloading 300 MB of dependencies 20 times a day is around 9 GB a month in transit alone. The practical recommendation is to have two projects:
build-mercadofresco-tiendaoutside the VPC for 90 % of the builds with mocked services, andbuild-mercadofresco-integracioninside, only on PRs intomainor nightly. And Gateway-type VPC endpoints for S3 and DynamoDB, which are free and keep that traffic off the NAT.
Debugging a build that fails
1. The CloudWatch logs, in /aws/codebuild/build-mercadofresco-tienda. To get straight to the point
without reading 2,000 lines, aws codebuild batch-get-builds --ids <id> with
--query 'builds[0].phases[?phaseStatus==FAILED].[phaseType,contexts[0].message]' gives you the phase
that failed and its message.
2. Local reproduction with the official image. The fastest way to iterate, and it costs nothing:
./codebuild_build.sh -i public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:5.0 \
-a /tmp/salida -s ~/proyectos/tienda -b buildspec.ymlThe script is in the aws/aws-codebuild-docker-images repository and runs the same buildspec on the
same image, on your machine: if it fails the same way, you have a debugging loop of seconds instead of
minutes. Watch out: secrets are not resolved locally unless you pass credentials, so export test
values.
3. The interactive debugging session. When the failure only happens in CodeBuild — and it does — put
codebuild-breakpoint as the command just before the one that fails: the build stops there and waits.
Launch it with --debug-session-enabled and connect with Session Manager from the console. You find
yourself inside the container, at the exact point, with the code cloned and the variables resolved;
codebuild-resume continues. It requires SSM permissions on the service role, and the build keeps
billing while you wait: do not leave a session open over lunch.
4. The variables CodeBuild gives you, which usually settle the question before anything else:
CODEBUILD_RESOLVED_SOURCE_VERSION (the exact SHA built), CODEBUILD_WEBHOOK_TRIGGER (what triggered it:
branch/main, pr/42, tag/v1.5.0), CODEBUILD_BUILD_SUCCEEDING (essential in post_build),
CODEBUILD_BUILD_ID and CODEBUILD_SRC_DIR.
Metrics, alarms and real cost
CodeBuild publishes metrics in AWS/CodeBuild. The three that matter are FailedBuilds (alarm: ≥ 3 in
1 hour, or a single one if it is main), Duration (p90 > 8 min) and SucceededBuilds for the hit rate.
aws cloudwatch put-metric-alarm --alarm-name mercadofresco-construcciones-fallidas \
--namespace AWS/CodeBuild --metric-name FailedBuilds --statistic Sum \
--dimensions Name=ProjectName,Value=build-mercadofresco-tienda \
--period 3600 --evaluation-periods 1 --threshold 3 \
--comparison-operator GreaterThanOrEqualToThreshold --treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
--profile mercadofresco-dev --region eu-west-1--treat-missing-data notBreaching is important: with no builds there is no data, and you do not want an
alarm going off every night. It is the same lesson as 05-01.
MercadoFresco's real monthly cost:
| Item | Calculation | Cost |
|---|---|---|
| PR and branch builds | 20/day × 4.2 min × 22 days × 0.010 | 18.48 USD |
| Nightly integration (in the VPC) + NAT transit | 1/day × 9 min × 30 × 0.010 + 9 GB | 3.13 USD |
| Artefacts in S3 with a 90-day expiry | ~4 GB | 0.09 USD |
| CloudWatch logs | ~2 GB ingested | 1.15 USD |
| Total | ≈ 22.85 USD/month |
Twenty-three dollars a month to remove "it works on my laptop" from the team's vocabulary. Compare it with a broken deployment on a Friday at 19:00 with 900 orders/hour at stake.
Cleanup: delete-project, delete-report-group --delete-reports and delete-log-group. Log groups
are not deleted when you delete the project and keep billing for storage; give them retention with
aws logs put-retention-policy --retention-in-days 30 from day one.
Common Mistakes and Tips
Publishing the artefact without checking CODEBUILD_BUILD_SUCCEEDING. post_build runs even if
build fails, so you upload to S3 the package of a build whose tests failed. It is the most dangerous
mistake in the lesson.
Putting secrets in PLAINTEXT variables, which stay in the project definition and in CloudTrail; or
forgetting :key in the Secrets Manager reference, so the variable receives the whole JSON and the
application fails with an error that does not point at the cause. And using set -x with secrets:
masking is string matching and as soon as you transform the value it comes out in the clear.
CodeBuild in a public subnet. With no public IP there is no internet access and pip install hangs
until the timeout. Private subnets with NAT, always. And do not put every build in the VPC "just in
case": you pay 30-60 s of ENI on each one plus the NAT transit.
Lowering the coverage threshold to unblock a PR. It gets lowered once and never goes back up. And retrying an intermittent test until it passes teaches the team that red means nothing, so on the day it is real it will be retried too.
Caching site-packages instead of ~/.cache/pip. You drag old versions along and debugging it takes
hours. And not setting retention on log groups: they bill forever, even after deleting the project.
Tip: order the gates from fastest to slowest. A formatting error should fail in 2 seconds, not after 4 minutes of tests.
Tip: make the buildspec runnable locally with codebuild_build.sh and the official image; if it
only works inside CodeBuild, every debugging iteration costs minutes. Export the version with
exported-variables so that CodePipeline (08-04) uses it in the following stages, and set a tight
timeout-in-minutes: it is the brake that stops you paying 60 minutes for a hung build.
Exercises
Exercise 1: the artefact that should not exist
On Friday at 18:40, Luis sees in mercadofresco-artefactos the file
tienda/1.5.2-c7a91f3/mercadofresco-tienda-1.5.2-c7a91f3.zip, published at 18:32. But in CodeBuild the
c7a91f3 build shows as red: 3 tests in tests/unitarias/test_pagos.py failed.
Answer: (a) how it is possible for the artefact to exist if the build failed; (b) what allowed it and how it is fixed; (c) why it is dangerous beyond this case, bearing in mind what CodePipeline will do in 08-04; (d) what check you would put in place so that an artefact like that cannot be deployed; (e) what alarm would have warned Marta at 18:35.
Exercise 2: the 22-minute build
The build takes 22 minutes: install 4 min (140 packages and psycopg2 compiled from source),
pre_build 3 min (pip-audit, bandit, ruff and git-secrets in series), build 14 min (620 unit
tests in series, 40 integration ones against the Aurora clone and the packaging), post_build 1 min. The
project is in the VPC, on SMALL, with no cache. Luis has started skipping the local build.
Propose a plan that brings the time below 6 minutes. For each measure state the estimated saving, the effect on cost and what is lost. Say explicitly which one you would apply first and why.
Exercise 3: the secret that appeared in the logs
Sara, reviewing the logs of a build, finds this line in
/aws/codebuild/build-mercadofresco-tienda:
The secret is correctly declared in env.secrets-manager and CodeBuild masks it in direct output. In
pre_build there are these two lines:
- export CADENA_CONEXION="postgresql://mfadmin:${PASS_AURORA}@${HOST_AURORA_PRUEBAS}:5432/mf"
- echo "Connecting to ${CADENA_CONEXION}"Answer: (a) why the masking did not work; (b) what you do in the first 30 minutes, in order;
(c) how you fix the buildspec; (d) the logs have had indefinite retention for 6 months and that
line appears in 340 builds: how do you deal with it; (e) what automatic control you would add to
detect it next time.
Solutions
Solution 1
(a) Because post_build runs even if build fails. It is deliberate — it lets you publish
reports for failed tests — but it means the publishing commands you put there run anyway: the tests failed
in build, the phase aborted, and post_build uploaded the ZIP quite happily.
(b) The CODEBUILD_BUILD_SUCCEEDING guard is missing: the guilty buildspec has an unconditional
aws s3 cp in post_build. The fix is the one that does appear in the lesson's buildspec:
post_build:
commands:
- |
if [ "${CODEBUILD_BUILD_SUCCEEDING}" != "1" ]; then
echo "Build failed: no artefact is published"; exit 1
fi
- aws s3 cp mercadofresco-tienda-*.zip s3://${BUCKET_ARTEFACTOS}/tienda/${VERSION_APP}/(c) Because it breaks the guarantee everything else depends on: that a published artefact is an artefact that passed the tests. In 08-04 the pipeline will take the artefact from one stage and pass it to the next; and if on top of that somebody deploys by file name from S3, code with three broken payment tests would get deployed. And it would be silent: the ZIP exists, it looks fine and its name says nothing about where it came from. Trust in the system is lost entirely with a single case like this.
(d) Two cheap layers. Metadata on the object — --metadata estado=exitosa,commit=... — and a
check at deployment time that rejects anything without it. And, more robustly, separating
responsibilities: have the build publish nothing and let the native mechanism (artifacts:) hand the
package to CodePipeline, which only advances if the previous stage finished well. The general rule: if
publishing is a side effect of a script, one day it will run when it should not; if it is a pipeline
state transition, it cannot happen out of order.
(e) The mercadofresco-construcciones-fallidas alarm on FailedBuilds, with a low threshold — 1 in
5 minutes for main — and alertas-mercadofresco as the destination: the threshold of 3 in 1 hour is
useful for spotting a trend, but on main a single failure deserves an immediate warning. In addition, an
EventBridge rule on CodeBuild Build State Change with build-status: FAILED warns within seconds with a
direct link to the logs.
Solution 2
| # | Measure | Saving | Cost | What is lost |
|---|---|---|---|---|
| 1 | Take the project out of the VPC; a separate nightly build-mercadofresco-integracion |
~7 min | Down: less NAT and ENI | Immediate feedback against real Aurora |
| 2 | pytest -n auto with MEDIUM |
~7 min | Price/min up, total down | Nothing, if the tests are isolated |
| 3 | pip cache + psycopg2-binary instead of compiling |
~3 min | None | The prebuilt wheel, acceptable |
| 4 | Scans in parallel | ~2 min | Same in total minutes | Logs harder to read |
| 5 | gitCloneDepth: 1 |
~30 s | None | git describe for the version |
| 6 | pytest-split across 3 batch builds |
~2 min | Same minutes, in parallel | Configuration complexity |
First, measure 1, and the justification is the key to the exercise: it is the one that saves the
most time and the only one that also reduces the cost. The 40 tests against the Aurora clone add value,
but not on every push: they add it before merging into main and in the nightly run. With 90 % of the
builds outside the VPC, the 30-60 s of ENI, the NAT transit and much of the network slowness disappear. It
is also the lowest-risk one: it does not change a single test.
Second, measure 2, counter-intuitive and very profitable: going from SMALL to MEDIUM doubles the
price per minute but reduces the total, because 620 tests with -n auto on 4 vCPUs almost divide the
suite's time by four. The requirement is that they be isolated: if they share a SQLite or a fixed temporary
directory, parallelisation will make them fail intermittently — precisely the problem we were talking
about — and then the fix is to isolate them, not to go back to running in series.
Result: install 4 → 1 min, pre_build 3 → 1, build 14 → 3, post_build 1. About 6 minutes,
with almost the same cost per build and a smaller NAT bill. And the consequence that is not in the table
and matters most: at 6 minutes, Luis stops skipping the build.
Solution 3
(a) Because the masking is a literal substitution of the secret's string in the output, and not a
semantic analysis. The buildspec interpolates ${PASS_AURORA} inside a larger string; the result is a
new value that CodeBuild knows nothing about, so when the echo runs it finds no match to mask and
prints the whole thing in the clear. It is the first of the three cases: transforming the value breaks
the masking.
(b) The first 30 minutes, in order. Rotate the test mfadmin password, because you always start
by cutting short the validity of what leaked. Work out who could have read it, with CloudTrail looking
for GetLogEvents and FilterLogEvents on that group over the last 6 months; the scope depends on how
many people have read access to CloudWatch Logs, which is usually more than you think. Check whether it
affects production — and here you see the value of the role's prefix separation: CodeBuild cannot
read mercadofresco/produccion/rds/mfadmin, so the incident stays confined to testing. And fix the
buildspec before the next build, or the log will keep being written.
(c) Never print the full string: if you need tracing, trace only the non-sensitive part
(echo "Connecting to ${HOST_AURORA_PRUEBAS}:5432/mf as mfadmin"). The deeper improvement is a different
one: if the application reads the secret from Secrets Manager with its own role, the value never passes
through a shell environment variable and this whole class of leak disappears. The environment variable is
convenient, but it is also the surface through which secrets escape.
(d) Treat the historical logs as compromised data: export the group to S3 if it has to be kept for auditing, delete the affected streams — there is no partial editing in CloudWatch Logs — and set 30-day retention. Since the password has already been rotated in (b), the residual exposure gives access to nothing: deleting them is hygiene and compliance, not containment. If the secret had been a production one, you would also have to weigh up the internal notification we saw in 08-01.
(e) A CloudWatch Logs metric filter over /aws/codebuild/* counting matches of patterns such as
postgresql://*:*@, AKIA or sk_live_, with an alarm towards alertas-mercadofresco. It is better than
a check inside the buildspec because it works even if nobody remembers to maintain it and it covers
every project, present and future. It is the idea from 05-01: if a piece of data should never appear in a
log, put a metric that counts it and alarm as soon as it goes above zero.
Conclusion
"It works on my laptop" no longer means anything at MercadoFresco. Every push starts a clean container
that knows nothing about the psycopg2 Luis installed by hand in 2024: it installs the declared
dependencies, runs 214 tests, checks that coverage does not drop below 75 %, scans the 87 dependencies for
CVEs, looks for leaked credentials and produces a versioned, immutable artefact in
mercadofresco-artefactos. What gets deployed will be that ZIP, with its commit and its metadata.
You know the anatomy of a project — source, environment, service role, buildspec, artefacts, cache and
logs — and you know that the compute size has to be measured and not assumed, because MEDIUM can work
out cheaper than SMALL when the build knows how to use the four cores. You know that the service role is
used by code that changes on every commit, and that this is why its scope is limited to the
mercadofresco/pruebas/ prefix: a build never needs production credentials. You have mastered the
buildspec.yml, the order of the phases and the two traps that catch everybody: that post_build runs
even if build fails — hence the CODEBUILD_BUILD_SUCCEEDING guard, without which you publish artefacts
from broken builds — and that gates are ordered from fastest to slowest.
You know how to inject secrets with the secret:key:label syntax and — more importantly — why the
masking breaks as soon as you transform the value or interpolate it into a larger string. You have tests
that verify what really matters, such as the gateway being called exactly once in the face of a
duplicated message, and the reports that turn a JUnit XML into a queryable history. And you have the
discipline that holds all of this up: an intermittent test is investigated or deleted, never retried; the
coverage threshold only ever goes up; and a check you are not willing to have stop a deployment is not a
gate, it is decoration.
You know when moto is enough — 90 % of cases, in a second and at no cost — and when the
aurora-mf-pruebas-luis clone is needed, with the warning that comes with it: CodeBuild in the VPC costs
30-60 seconds of ENI per build plus the NAT transit, and that is why MercadoFresco has two projects, a
fast one outside and an integration one inside. And you know how to debug without guessing: the logs, local
reproduction with the official image and codebuild-breakpoint to get inside the container at the exact
point. All for about 23 dollars a month.
But there is something this lesson has not changed. The artefact is there, verified, waiting in a
bucket. And to get it onto the instances of the ASG asg-mercadofresco-tienda, Luis still does exactly
what he did on day one: SSH in, copy the ZIP, unzip it, restart the service and look at the site with his
fingers crossed. For the forty seconds it takes, the instance serves errors; if there are two and he does
them one after the other, half the traffic sees one version and the other half another; and if the ZIP
turns out to be broken, going back means finding the previous one and repeating the ritual with the shop
down. The artefact is good; the way of putting it into production is still a manual ritual with no net.
In 08-03, "AWS CodeDeploy", that gets solved. We will see the three targets — EC2, Lambda and ECS —
and which strategies each one supports; the concepts of application, deployment group, revision and
deployment configuration, with the agent installed on the ASG's instances; the appspec.yml and its
lifecycle hooks one by one, with MercadoFresco's real scripts to stop the service, warm the cache and
check /salud; the in-place strategies with AllAtOnce, HalfAtATime and OneAtATime, the
blue/green one with the ALB's target groups, and the canary and linear ones for deploying the
mercadofresco-cobrar-pago Lambda; the warning about schema migrations and why they must never go in the
same step as the code; and above all the piece that closes the course's fourth problem: automatic
rollback when mercadofresco-alb-latencia-alta fires or a hook fails, with nobody watching.
AWS Course
Module 1: Introduction to AWS
- What Is AWS?
- Setting Up Your AWS Account
- AWS Global Infrastructure
- The AWS Management Console
- AWS CLI and SDKs
Module 2: Core AWS Services
Module 3: Networking and Content Delivery
Module 4: Security and Identity
- AWS Identity and Access Management (IAM)
- AWS Key Management Service (KMS)
- Secrets Manager and Parameter Store
- AWS Shield
- AWS WAF
Module 5: Monitoring and Management
Module 6: Databases
Module 7: Application Integration
- Amazon SQS
- Amazon SNS
- Amazon EventBridge
- AWS Step Functions
- Integration Patterns: Idempotency, Retries and Dead-Letter Queues
