In mercadofresco-artefactos there is a verified ZIP: it passed 214 tests, carries its commit in the metadata and has the version in its name. And to get it into production, Luis still does what he has always done: ssh into the first instance of the ASG, scp the ZIP, unzip, systemctl restart, look at the site, repeat on the second. Forty seconds per instance serving errors, two versions coexisting while it lasts, and if something goes wrong, hunt for the previous ZIP and repeat the ritual with the shop down.

AWS CodeDeploy turns that ritual into a governed operation. It knows how to take an instance out of the target group before touching it, run your scripts in a defined order, check that the application responds before giving it traffic back, advance across the fleet a bit at a time and — what really matters — roll back on its own when a CloudWatch alarm fires during the deployment. It is the lesson that closes the fourth problem of the course: risky deployments.

Cost warning. CodeDeploy is free for EC2, Lambda and ECS; it only charges 0.02 USD per on-premises instance update. What costs money is everything around it: in blue/green you temporarily double the ASG instances — around 0.09 USD for a 30-minute deployment with two t3.medium — and Lambda versions do not expire on their own. The cleanup section comes at the end. Fictitious data.

Contents

  1. What CodeDeploy solves and what it does not
  2. The three targets and the strategies each one supports
  3. Concepts: application, deployment group, revision and configuration
  4. The agent on the ASG instances
  5. The appspec.yml for EC2 and the lifecycle hooks
  6. MercadoFresco's real scripts
  7. In-place deployment: AllAtOnce, HalfAtATime and OneAtATime
  8. Blue/green with the ASG and the target groups
  9. The Lambda and ECS appspec
  10. Canary and linear: deploying mercadofresco-cobrar-pago
  11. Automatic rollback: the piece that solves problem 4
  12. Comparison with manual rollback and with weighted Route 53
  13. Schema migrations: expand and contract
  14. Deployment observability and diagnosing a failure
  15. Cost and cleanup
  16. Common mistakes and tips
  17. Exercises
  18. Conclusion

What CodeDeploy solves and what it does not

CodeDeploy does not build, does not decide when to deploy and does not define infrastructure. It does one thing: it takes a revision — a ZIP in S3 or an image — and installs it on a set of targets following a plan, with hooks where you put your scripts and with stop conditions.

Problem with manual deployment What CodeDeploy brings
The instance serves errors while it is updated It takes it out of the target group before touching it
Two versions coexisting with no control It controls how many instances are on each version
"Did the service restart properly?" ValidateService checks it and fails if not
Going back means repeating the ritual by hand Automatic rollback in minutes
Nobody knows what was deployed or when History with revision, author and result
The deployment is the same in dev as in prod Deployment groups with different configuration

And what it does not bring: it does not coordinate the order between different services — that is the job of the 08-04 pipeline — it does not migrate your database and it does not make up for an application that cannot coexist with itself in two versions. That last limitation is the subject of the schema migration section.

The three targets and the strategies each one supports

Target In place Blue/green Canary Linear Agent
EC2 / ASG Yes Yes No No Yes
On-premises servers Yes No No No Yes
AWS Lambda No Yes (by alias) Yes Yes No
Amazon ECS No Yes Yes Yes No

Three observations that avoid frequent confusion. On EC2 there is no canary and no linear: percentage traffic shifting belongs to Lambda and ECS, where traffic is split by alias or target group weight; on EC2 the closest thing is OneAtATime, which is instance granularity, not a percentage of requests. On Lambda and ECS there is no in-place deployment: the new version is always created alongside the old one and the traffic is shifted, which is safer by construction. And the agent is only needed on EC2 and on-premises servers; on Lambda and ECS, CodeDeploy talks to the service API.

Concepts: application, deployment group, revision and configuration

Four objects, and confusing them causes half of the initial doubts:

Object What it is In MercadoFresco
Application Logical container: name and platform app-mercadofresco-tienda, app-mercadofresco-cobrar-pago
Deployment group Where and how: targets, strategy, alarms, rollback dg-mercadofresco-tienda-desarrollo / -produccion
Revision What is deployed: the ZIP with its appspec.yml s3://mercadofresco-artefactos/tienda/1.5.0-a3f9c21/
Deployment configuration The pace and how much must stay healthy CodeDeployDefault.HalfAtATime, or your own

The important one is the deployment group: the same application and the same revision behave very differently depending on the group. In development, AllAtOnce with no alarms so that it is fast; in production, blue/green with alarms and automatic rollback. That is the mechanism that separates environments.

aws deploy create-application --application-name app-mercadofresco-tienda \
  --compute-platform Server --profile mercadofresco-dev --region eu-west-1

aws deploy create-deployment-group \
  --application-name app-mercadofresco-tienda \
  --deployment-group-name dg-mercadofresco-tienda-produccion \
  --service-role-arn arn:aws:iam::111122223333:role/rol-codedeploy-mercadofresco \
  --auto-scaling-groups asg-mercadofresco-tienda \
  --deployment-config-name CodeDeployDefault.HalfAtATime \
  --load-balancer-info '{"targetGroupInfoList":[{"name":"tg-mercadofresco-tienda"}]}' \
  --alarm-configuration '{"enabled": true, "ignorePollAlarmFailure": false,
    "alarms": [{"name":"mercadofresco-alb-latencia-alta"},
               {"name":"mercadofresco-pedidos-fallidos"}]}' \
  --auto-rollback-configuration '{"enabled": true,
    "events": ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM"]}' \
  --profile mercadofresco-dev --region eu-west-1

Notice ignorePollAlarmFailure: false: if CodeDeploy cannot query an alarm, the deployment fails instead of carrying on blind. It is the right behaviour — if the safety mechanism does not answer, do not carry on — but it produces baffling failures when the role is missing cloudwatch:DescribeAlarms.

The agent on the ASG instances

The agent is a process that polls CodeDeploy, downloads the revision and runs the hooks. Without it, the instance stays in Pending until the timeout and the deployment fails with an unhelpful message.

It can be installed baked into the AMI, from the user data of lt-mercadofresco-tienda, or — the recommended option — as a Systems Manager association with automatic updates:

aws ssm create-association --name AWSCodeDeployAgentUpdate \
  --targets Key=tag:Proyecto,Values=mercadofresco \
  --schedule-expression "rate(14 days)" \
  --profile mercadofresco-dev --region eu-west-1

That last one is what MercadoFresco uses because it solves the real problem: an out-of-date agent stops working without warning, and updating it by hand on instances the ASG creates and destroys is impossible.

Two requirements that get forgotten. The instance role needs to be able to read the artefact:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::mercadofresco-artefactos",
                   "arn:aws:s3:::mercadofresco-artefactos/tienda/*"] },
    { "Effect": "Allow", "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/*",
      "Condition": {"StringEquals": {"kms:ViaService": "s3.eu-west-1.amazonaws.com"}} }
  ]
}

The second block is needed because the bucket is encrypted with alias/mercadofresco-datos: without KMS permission, the agent downloads the object and fails to decrypt it, with an access denied error that looks like an S3 problem. The second requirement is internet egress or VPC endpoints: the agent talks to the CodeDeploy API and to S3, and a private subnet with no NAT and no endpoints leaves the deployment hanging. To diagnose, systemctl status codedeploy-agent and the log at /var/log/aws/codedeploy-agent/codedeploy-agent.log.

The appspec.yml for EC2

The appspec.yml goes in the root of the ZIP, not in a subdirectory, and it tells the agent what to copy and what to run:

version: 0.0
os: linux

files:
  - source: /app
    destination: /opt/mercadofresco/app
  - source: /scripts
    destination: /opt/mercadofresco/scripts
  - source: /VERSION
    destination: /opt/mercadofresco

# Without this, a file that already exists at the destination fails the whole deployment
file_exists_behavior: OVERWRITE

permissions:
  - object: /opt/mercadofresco/app
    owner: mercadofresco
    mode: 640
    type: [file]

hooks:
  ApplicationStop:
    - { location: scripts/parar_servicio.sh, timeout: 60, runas: root }
  BeforeInstall:
    - { location: scripts/comprobar_espacio.sh, timeout: 30, runas: root }
  AfterInstall:
    - { location: scripts/instalar_dependencias.sh, timeout: 300, runas: mercadofresco }
    - { location: scripts/cargar_configuracion.sh, timeout: 60, runas: mercadofresco }
  ApplicationStart:
    - { location: scripts/arrancar_servicio.sh, timeout: 120, runas: root }
  ValidateService:
    - { location: scripts/comprobar_salud.sh, timeout: 180, runas: mercadofresco }
    - { location: scripts/calentar_cache.sh, timeout: 120, runas: mercadofresco }

file_exists_behavior: OVERWRITE deserves a warning: without it, the default value is DISALLOW and the deployment fails if a file already exists at the destination. Since the destination almost always has the previous version, it is the first failure everybody runs into. RETAIN exists for files the application generates and that must not be overwritten.

The lifecycle hooks, one by one

Hook When Revision What it is for
ApplicationStop Before anything The PREVIOUS one Stopping the service cleanly
BeforeInstall Before copying The new one Preliminary checks, backups
AfterInstall After copying The new one Dependencies, permissions, configuration
ApplicationStart After installing The new one Starting the service
ValidateService Last The new one Checking that it really works
BeforeAllowTraffic Blue/green only The new one Warming up before receiving traffic
AfterAllowTraffic Blue/green only The new one Validating with real traffic

Three things to understand properly. ApplicationStop runs from the PREVIOUS revision, and it is the source of a classic problem: if the previous deployment left a broken parar_servicio.sh, the new deployment fails on the very first hook even if the new code is perfect — and on an instance's first deployment it does not run at all, because there is no previous revision. Write it bulletproof and make it finish successfully even if it cannot find the service. ValidateService is the hook that justifies everything else: without it, "deployed" means "the files are copied and systemctl did not complain", which is not the same as "the shop works". And timeouts are per hook, with a maximum of one hour: an AfterInstall with no cache can go over 300 seconds, and a ValidateService with 10 seconds always fails on the first instance, which is the slowest to warm up.

MercadoFresco's real scripts

Stopping the service without failing if it is not there:

#!/bin/bash
# scripts/parar_servicio.sh  -> ApplicationStop
# CAREFUL: it runs from the PREVIOUS revision. It has to be bulletproof.
set -u    # We do NOT use 'set -e': we want to control the failures ourselves

systemctl list-unit-files | grep -q '^mercadofresco.service' || {
  echo "The service does not exist yet (first deployment)."; exit 0; }

systemctl stop mercadofresco
for i in $(seq 1 30); do
  systemctl is-active --quiet mercadofresco || {
    echo "Stopped cleanly in ${i}s"; exit 0; }
  sleep 1
done

echo "It did not stop in 30s. Forcing."
systemctl kill -s SIGKILL mercadofresco || true
exit 0      # NEVER fail here: it would block every future deployment

The final exit 0 is deliberate: an ApplicationStop that fails leaves the instance in a state you can only get out of by recreating the deployment group.

Checking health for real:

#!/bin/bash
# scripts/comprobar_salud.sh  -> ValidateService
set -euo pipefail

for i in $(seq 1 30); do
  RESPONSE=$(curl -sf -m 5 http://localhost:8080/salud 2>/dev/null || echo '{}')
  if [ "$(echo "$RESPONSE" | jq -r '.estado // "ko"')" = "ok" ]; then
    # A 200 is not enough: we check the critical dependencies one by one
    for dep in aurora redis sqs; do
      HEALTH=$(echo "$RESPONSE" | jq -r ".dependencias.${dep} // \"ko\"")
      [ "$HEALTH" = "ok" ] || { echo "Dependency ${dep} in state ${HEALTH}"; exit 1; }
    done
    # And that the live version is the one we have just installed
    LIVE_VERSION=$(echo "$RESPONSE" | jq -r '.version')
    [ "$LIVE_VERSION" = "$(cat /opt/mercadofresco/VERSION)" ] || {
      echo "Serving ${LIVE_VERSION}, not the expected one"; exit 1; }
    echo "Health OK, version ${LIVE_VERSION}"; exit 0
  fi
  echo "Attempt ${i}/30"; sleep 5
done

echo "Did not pass the health check in 150s"
exit 1

This script is the heart of deployment safety, and it makes three checks that a simple curl -f /salud does not: that the critical dependencies respond — it is no use for the process to be alive if it cannot reach Aurora — that the version answering is the one you have just installed — which catches the case where the service never actually restarted — and that all of it happens within a deadline. Its exit 1 is what triggers the rollback.

Warming the cache before receiving traffic:

#!/bin/bash
# scripts/calentar_cache.sh  -> ValidateService (or BeforeAllowTraffic in blue/green)
set -euo pipefail
curl -sf -m 30 -X POST http://localhost:8080/interno/precargar-catalogo
for sku in FRUT-0012 VERD-0034 PESC-0007 CARN-0021; do
  curl -sf -m 5 "http://localhost:8080/productos/${sku}" > /dev/null
done
echo "Cache warm"

Without this step, the first wave of traffic finds an empty cache, goes to Aurora, and TiempoConfirmacionPedido shoots up for two minutes. At the Friday peak with 900 orders/hour, that is exactly what makes mercadofresco-alb-latencia-alta fire and roll back a deployment that was perfectly fine.

In-place deployment: AllAtOnce, HalfAtATime and OneAtATime

Configuration At a time Minimum capacity Duration (4 instances) When
AllAtOnce All of them 0 ~3 min Development. Never in production
HalfAtATime 50 % 50 % ~6 min Production with spare capacity
OneAtATime 1 N-1 ~12 min Maximum caution; small fleets

AllAtOnce in production is an outage, not a deployment. Every instance out at the same time is exactly what you are trying to avoid; and if it fails, no instance is left with the good version to go back to.

You can define your own configuration with aws deploy create-deployment-config and --minimum-healthy-hosts type=FLEET_PERCENT,value=90. But be careful: with that 90 % and a fleet of 2 instances, the healthy minimum is 2 (it rounds up) and no deployment can start; with small fleets use HOST_COUNT instead of percentages.

Blue/green with the ASG and the target groups

In blue/green no existing instance is touched: new ones are created, they are checked, they are given the traffic and only then are the old ones terminated.

flowchart TB
    ALB[alb-mercadofresco-tienda] -->|100% before| TGA[tg-mercadofresco-tienda]
    ALB -.->|100% after| TGV[tg-mercadofresco-tienda-verde]
    TGA --> AZUL[Blue: v1.4.3<br/>i-aaa1, i-aaa2]
    TGV --> VERDE[Green: v1.5.0<br/>i-vvv1, i-vvv2]
    VERDE --> H[BeforeAllowTraffic:<br/>warm the cache]
    H --> S{State OK?}
    S -->|No| F[Terminate green<br/>Blue keeps serving]
    S -->|Yes| C[Switch the ALB listener]
    C --> W[Wait 30 min<br/>watching alarms]
    W -->|Alarm| R[ROLLBACK:<br/>back to blue in 90 s]
    W -->|All good| T[Terminate blue]
aws deploy update-deployment-group \
  --application-name app-mercadofresco-tienda \
  --current-deployment-group-name dg-mercadofresco-tienda-produccion \
  --deployment-style '{"deploymentType":"BLUE_GREEN",
                       "deploymentOption":"WITH_TRAFFIC_CONTROL"}' \
  --blue-green-deployment-configuration '{
    "terminateBlueInstancesOnDeploymentSuccess": {
      "action": "TERMINATE", "terminationWaitTimeInMinutes": 30 },
    "deploymentReadyOption": { "actionOnTimeout": "CONTINUE_DEPLOYMENT" },
    "greenFleetProvisioningOption": { "action": "COPY_AUTO_SCALING_GROUP" }}' \
  --profile mercadofresco-dev --region eu-west-1

terminationWaitTimeInMinutes: 30 is the window in which the blue environment still exists, shut down but intact, after the traffic switch: it is what makes the rollback take ninety seconds, because all you have to do is send the listener back to the blue target group — at 0 you save pennies and going back becomes a full ten-minute deployment. actionOnTimeout: CONTINUE_DEPLOYMENT makes the traffic switch automatic; the alternative, STOP_DEPLOYMENT, waits for somebody to press "reroute traffic", useful on day one and unsustainable as a practice. And COPY_AUTO_SCALING_GROUP creates a new ASG by copying lt-mercadofresco-tienda, as opposed to DISCOVER_EXISTING with already tagged instances.

Aspect In place Blue/green
Cost during the deployment None Doubled capacity
Rollback time Full deployment (~10 min) ~90 seconds
Risk to the stable version It is overwritten Untouched until the end
Total duration 6-12 min 15-40 min
Local state on disk Preserved Lost: new instances

MercadoFresco uses in place with HalfAtATime in development and blue/green in production. Doubling two t3.medium for half an hour is about nine cents per deployment; ninety seconds of rollback are worth quite a lot more than that.

The Lambda and ECS appspec

For Lambda and ECS the appspec is a different thing: it does not copy files, it declares which version should receive the traffic. It accepts YAML or JSON.

# appspec-lambda.yml
version: 0.0
Resources:
  - mercadofresco-cobrar-pago:
      Type: AWS::Lambda::Function
      Properties:
        Name: mercadofresco-cobrar-pago
        Alias: produccion
        CurrentVersion: "7"      # the one serving now
        TargetVersion: "8"       # the new one
Hooks:
  - BeforeAllowTraffic: mercadofresco-validar-antes-de-trafico
  - AfterAllowTraffic: mercadofresco-validar-despues-de-trafico

An important difference: in Lambda the hooks are Lambda functions, not scripts, and they have an obligation that is always forgotten: reporting their result with PutLifecycleEventHookExecutionStatus, or CodeDeploy waits until the timeout and fails.

import boto3, json, os
cd = boto3.client("codedeploy")
lam = boto3.client("lambda")

def handler(event, context):
    deployment_id = event["DeploymentId"]
    status = "Failed"
    try:
        # We invoke the NEW version directly, before it receives real traffic
        r = lam.invoke(
            FunctionName=f"mercadofresco-cobrar-pago:{os.environ['VERSION_NUEVA']}",
            Payload=json.dumps({"modo": "prueba_humo", "importe_eur": 1.00,
                                "clave_idempotencia": f"humo-{deployment_id}"}))
        body = json.loads(r["Payload"].read())
        if r["StatusCode"] == 200 and body.get("estado") == "cobrado":
            status = "Succeeded"
    except Exception as e:
        print(f"Smoke test failed: {e}")

    # WITHOUT this call, CodeDeploy waits until the timeout and fails the deployment
    cd.put_lifecycle_event_hook_execution_status(
        deploymentId=deployment_id,
        lifecycleEventHookExecutionId=event["LifecycleEventHookExecutionId"],
        status=status)
    return {"status": status}

Notice the clave_idempotencia derived from the deployment identifier: the smoke test goes down the same path as a real charge, and without a unique key every deployment would reuse the previous one. It is the idempotency of 07-05 applied to the deployment itself. In ECS the appspec references the task definition and the container; the ECS detail is module 10.

Canary and linear: deploying mercadofresco-cobrar-pago

Here there is percentage traffic shifting, the best safety net CodeDeploy offers.

Configuration How it shifts Duration When
AllAtOnce 100 % in one go Seconds Development
Canary10Percent5Minutes 10 %, wait 5 min, the rest ~5 min Charges: few bad requests
Canary10Percent30Minutes 10 %, wait 30 min, the rest ~30 min High-risk changes
Linear10PercentEvery1Minute +10 % every minute ~10 min Gradual degradation

Canary or linear, with judgement. The canary keeps a small percentage for a while and then jumps to 100 %: if the failure is obvious, only 10 % suffer it and only for a short time. The linear one goes up bit by bit and is better at spotting degradations that depend on load — a memory leak, a pool that runs out — but it exposes 50 % before the deployment is halfway through. For mercadofresco-cobrar-pago, Marta chooses Canary10Percent5Minutes: a failure in charging is obvious within seconds, not gradual, and with 900 orders/hour at peak, five minutes at 10 % means about 7 orders exposed.

The mechanism relies on versions and aliases, which we saw in 02-05: mercadofresco-cobrar-pago:produccion points to version 7, and CodeDeploy adds a weight towards version 8, raising it according to the configuration. Everything that invokes the function uses the alias and notices nothing.

It is launched with aws deploy create-deployment and --deployment-config-name CodeDeployDefault.LambdaCanary10Percent5Minutes, passing the appspec in --revision. The alarm that watches it goes on the errors of the alias, not of the whole function:

aws cloudwatch put-metric-alarm \
  --alarm-name mercadofresco-cobrar-pago-errores-canario \
  --namespace AWS/Lambda --metric-name Errors --statistic Sum \
  --dimensions Name=FunctionName,Value=mercadofresco-cobrar-pago \
               Name=Resource,Value=mercadofresco-cobrar-pago:produccion \
  --period 60 --evaluation-periods 1 --threshold 3 \
  --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
  --profile mercadofresco-dev --region eu-west-1

The 60-second period and a single evaluation period are deliberate: with a 5-minute window for the canary, an alarm of 3 periods of 5 minutes would fire when the deployment is already at 100 %. The alarm has to be faster than the deployment it is watching.

Automatic rollback: the piece that solves problem 4

This is the section the lesson exists for. CodeDeploy rolls back on its own in three situations, declared in autoRollbackConfiguration: DEPLOYMENT_FAILURE when a hook returns non-zero or times out, DEPLOYMENT_STOP_ON_ALARM when an alarm from alarmConfiguration goes to ALARM during the deployment, and DEPLOYMENT_STOP_ON_REQUEST when somebody stops it by hand.

What matters is understanding what "rolling back" means in each mode, because they are not the same. In place, CodeDeploy launches a new deployment with the previous revision: a full one, with its hooks and its minutes. In blue/green within the wait window, it sends the ALB listener back to the blue target group: ninety seconds, and the blue instances never even noticed. In Lambda or ECS, it returns the alias weight to the previous version: seconds.

Alarm What it detects Why it is here
mercadofresco-alb-latencia-alta p95 > 1,500 ms, 2 periods of 60 s A slow version is a broken version
mercadofresco-pedidos-fallidos DLQ > 5 messages in 5 min The new consumer is not processing properly
mercadofresco-tienda-degradada Composite (05-01) The overall picture

Choosing the alarms is the design decision of this section, and they must meet three conditions. Detecting what a bad deployment causes: a cost alarm has no business here; a latency or 5xx one does. Being faster than the deployment, because one of 3 periods of 5 minutes does not protect a 10-minute deployment. And not firing for unrelated reasons: if mercadofresco-alb-latencia-alta fires every Friday at 19:00 because of the normal peak, it will roll back correct deployments and the team will end up disabling the rollback. Watch them for a month before wiring them up.

The initial state of the alarm matters. If it is already in ALARM when you start, CodeDeploy will not begin: correct — do not deploy onto a system that is already broken — but baffling the first time.

Comparison with manual rollback and with weighted Route 53

Mechanism Rollback time Automatic Granularity
Manual over SSH (today's way) 10-30 min, with nerves No Instance
CodeDeploy in place ~10 min Yes Instance
CodeDeploy blue/green ~90 s Yes Whole environment
CodeDeploy canary (Lambda) Seconds Yes % of invocations
Weighted Route 53 (03-05) 60 s + TTL Not out of the box % of DNS resolutions

It is worth comparing with weighted Route 53, which in 03-05 also split traffic and might look like an alternative. It is not, for two reasons. The DNS TTL: even if you set the weight to 0 instantly, resolvers and browsers keep using the cached answer for as long as the TTL lasts, so "rolling back" takes minutes and is never complete. And the fact that the split is of resolutions, not of requests: a client that resolves once and keeps the connection stays where it landed. Weighted Route 53 is excellent for moving traffic between regions; CodeDeploy is the right mechanism for deploying a version, because it operates on the load balancer and is immediate and complete.

Schema migrations: expand and contract

Here is the most important warning in the lesson, and it is not a technical detail but a process rule: the code deployment and the schema change must not go in the same step.

The reasoning is simple. During any gradual deployment two versions of the code coexist against a single database: if the AfterInstall of the new one runs ALTER TABLE pedidos DROP COLUMN direccion_antigua, the old one — which is still serving half the traffic — starts to fail. And if the deployment rolls back, the rollback does not undo the ALTER TABLE: you are left with the old code and the new schema, the worst possible state. The answer is expand and contract, in three deployments:

Phase What is done Compatible with When
1. Expand Add the new without removing anything: NULLABLE column Both versions Before the code
2. Migrate Code that writes to both and reads the new one with a fallback Both versions Normal deployment
3. Contract Remove the old Only the new one Days or weeks later

Applied to the delivery slot Luis is about to add:

-- PHASE 1 (expand): BEFORE deploying the code. No NOT NULL, no DEFAULT and no
-- index: it neither rewrites the table nor locks it. 1.4.3 does not know the column and ignores it.
ALTER TABLE pedidos ADD COLUMN franja_entrega VARCHAR(10) NULL;

-- PHASE 3 (contract): weeks later, when NO instance is serving 1.4.3 any more
ALTER TABLE pedidos ALTER COLUMN franja_entrega SET NOT NULL;
CREATE INDEX CONCURRENTLY idx_pedidos_franja ON pedidos(franja_entrega);

Four practical rules follow from this. Never run migrations in a CodeDeploy hook: the hook runs on every instance, so with four instances you get four simultaneous ALTER TABLEs, and if it fails halfway the deployment rolls back but the schema does not. The migration is a pipeline step of its own, before the application deployment; that is 08-04. Every expansion migration must be reversible or harmless: a NULLABLE column is redundant but harmless if the code rolls back. And CREATE INDEX CONCURRENTLY so as not to lock the table, outside the deployment. The full walkthrough, with the queue consumer and the event contract, is lesson 08-05.

Deployment observability and diagnosing a failure

CodeDeploy publishes every state change to EventBridge, so everything from 07-03 applies, with a pattern on source: ["aws.codedeploy"], detail-type: ["CodeDeploy Deployment State-change Notification"] and detail.state: ["FAILURE", "STOP"].

With alertas-mercadofresco as the target and an input transformation that leaves a readable message: which application, which version, which state and the link to the console. A deployment that rolls back on its own at 19:15 on a Friday is good news, but only if somebody hears about it. Diagnosing a failure:

# Which deployment failed and why; then, which instance and which specific hook
aws deploy get-deployment --deployment-id d-A1B2C3D4E \
  --query 'deploymentInfo.[status,errorInformation.code,errorInformation.message]' --output table
aws deploy get-deployment-target --deployment-id d-A1B2C3D4E --target-id i-0abc123def456 \
  --query 'deploymentTarget.instanceTarget.lifecycleEvents[?status==`Failed`]'
Error code What it means Where to look
HEALTH_CONSTRAINTS Fewer healthy instances than required A percentage with a small fleet?
SCRIPT_FAILED A hook returned non-zero The agent log on that instance
SCRIPT_TIMED_OUT A hook went over its timeout Raise the timeout or speed the script up
NO_INSTANCES No instance matches Tags or the name of the ASG
AGENT_ISSUE_* The agent is not responding Is it alive? Is there network egress?
ALARM_ACTIVE An alarm was in ALARM at the start Fix the system before deploying

Cost and cleanup

Item Cost
CodeDeploy on EC2, Lambda and ECS 0 USD (0.02 USD/instance on on-premises servers)
Doubled capacity in blue/green ~0.09 USD per deployment (2 × t3.medium, 30 min)
Storage of Lambda versions 75 GB quota per region
MercadoFresco total ~2 USD/month with 20 deployments
# Old Lambda versions are NOT deleted on their own and they eat into the quota
aws lambda list-versions-by-function --function-name mercadofresco-cobrar-pago \
  --query 'Versions[?Version!=`$LATEST`].[Version,LastModified]' --output table
aws lambda delete-function --function-name mercadofresco-cobrar-pago:3

# ASGs orphaned by a failed blue/green: instances running and billing
aws autoscaling describe-auto-scaling-groups \
  --query "AutoScalingGroups[?starts_with(AutoScalingGroupName,'CodeDeploy_')].[AutoScalingGroupName,DesiredCapacity]"

Orphaned ASGs are the hidden cost of this lesson. A blue/green interrupted halfway can leave an ASG with the CodeDeploy_ prefix and its instances alive, serving nobody and billing. Check for them after every failed deployment.

Common Mistakes and Tips

An ApplicationStop that can fail. It runs from the previous revision and blocks every future deployment of that instance. Always finish with exit 0.

Forgetting file_exists_behavior: OVERWRITE. The default DISALLOW fails the deployment as soon as a file already exists at the destination, which is always from the second one onwards.

A ValidateService that only checks that the process is alive. One that cannot reach Aurora passes the check and receives traffic: validate dependencies and version.

Running database migrations in a hook. It runs on every instance, it is not transactional with respect to the deployment, and the rollback does not undo it. It is the most expensive mistake in the module.

AllAtOnce in production, which is not a deployment but a scheduled outage; and healthy fleet percentages with small fleets, because FLEET_PERCENT=90 with 2 instances demands 2 healthy and no deployment can start. Use HOST_COUNT.

terminationWaitTimeInMinutes: 0 in blue/green. You save pennies and lose the 90-second rollback, which is the reason you chose blue/green in the first place.

Alarms slower than the deployment, which fire when everything is already deployed; or alarms with false positives wired to the rollback, which will roll back good deployments until the team disables the rollback altogether. Watch them for a month before wiring them up.

Forgetting PutLifecycleEventHookExecutionStatus in a Lambda hook, which makes it wait until the timeout with a message that does not mention the missing call; or missing KMS permissions on the instance role, which stops the ZIP being decrypted with an error that looks like an S3 one.

Tip: test the rollback on purpose. Deploy a version with a ValidateService that returns 1 and time it: a rollback nobody has rehearsed is a hypothesis. Have a /salud that tells the truth, with the state of every dependency and the version being served, because it is the piece everything else rests on. And deploy in development first: broken hooks are discovered just as well there.

Exercises

Exercise 1: the deployment that got stuck

Luis deploys 1.5.0 to dg-mercadofresco-tienda-produccion (in place, HalfAtATime, 4 instances). The first half goes well; the second fails with SCRIPT_FAILED on ApplicationStop. The rollback is launched and also fails on ApplicationStop. Result: 2 instances on 1.5.0, 2 on 1.4.3 and no deployment possible. Investigating, he sees that the parar_servicio.sh of 1.4.3 does set -e and systemctl stop mercadofresco && rm /var/run/mercadofresco.pid, and that this PID file has not existed since 1.4.2.

Answer: (a) why the rollback fails in the same place; (b) why the first two instances did work; (c) how you unblock the situation right now, with the commands; (d) how you stop it happening again; (e) what would have been different if the group were blue/green.

Exercise 2: choosing a strategy for three components

For each component state the target, the deployment configuration, the alarms wired to the rollback and the justification. (a) The web shop on asg-mercadofresco-tienda (2-4 instances), with a peak on Fridays from 17:00 to 21:00 and 900 orders/hour, where a 500 error is a lost sale. (b) The Lambda mercadofresco-cobrar-pago, around 900 invocations/hour at peak, already idempotent (07-05), where a failure can charge twice or not charge at all. (c) The workers on asg-mercadofresco-trabajadores, which consume cola-mercadofresco-pedidos, receive no ALB traffic and, if they stop for 5 minutes, the queue grows and drains afterwards.

Exercise 3: the migration that broke the rollback

On Tuesday at 11:00, Luis deploys 1.6.0 in blue/green. 1.6.0 renames the column direccion to direccion_entrega, and the AfterInstall runs ALTER TABLE pedidos RENAME COLUMN direccion TO direccion_entrega;. Green starts up fine, passes ValidateService and receives 100 % of the traffic. Eight minutes later, mercadofresco-pedidos-fallidos fires: the queue consumer, still on 1.5.2, fails on every message. CodeDeploy rolls back to blue in 90 seconds. And then the whole shop stops working.

Answer: (a) why the rollback made things worse; (b) how many times the ALTER TABLE ran and what happened on the second one; (c) the immediate recovery plan; (d) how the change should have been made, with the phases and what is deployed in each; (e) what automatic control would have stopped this reaching production.

Solutions

Solution 1

(a) Because ApplicationStop always runs from the previous revision installed on the instance, and on instances 3 and 4 the previous revision is 1.4.3, whose script is broken. The rollback is a new deployment of 1.4.3, and its first hook is again the ApplicationStop of whatever is installed — which is still the broken 1.4.3. The same faulty script runs and fails in exactly the same way. That loop is precisely why this hook has to be bulletproof.

(b) By chance, and it is worth understanding: on instances 1 and 2 the PID file existed — they were older and were dragging one along from 1.4.1 — so rm returned 0 and the script finished fine. On 3 and 4, recreated by the ASG with a clean 1.4.3, there was no file, rm returned 1 and set -e aborted. A deployment that works on half the fleet and fails on the other half almost always points to divergent state between instances, and that is a problem in itself: the instances of an ASG should be indistinguishable.

(c) Unblocking. The quick route, if the shop is serving badly, is to fix the script by hand on the affected instances through Session Manager, editing the deployment-root/<id>/deployment-archive/scripts/parar_servicio.sh, and relaunch. The clean one is to skip the hook just once: in the console you can launch a deployment ticking the option to omit ApplicationStop, BeforeBlockTraffic and AfterBlockTraffic, designed for exactly this case. The equivalent alternative through the CLI is deleting and recreating the deployment group, because that removes the reference to the last known revision and ApplicationStop stops running:

aws deploy delete-deployment-group --application-name app-mercadofresco-tienda \
  --deployment-group-name dg-mercadofresco-tienda-produccion \
  --profile mercadofresco-dev --region eu-west-1
# ...recreate it, and deploy 1.4.3 again with create-deployment.

(d) Three measures. In the script, remove set -e, put an unconditional exit 0 at the end and use rm -f, which does not fail if the file is not there. In the process, a test of the hooks in CodeBuild: a container that runs each script against a clean system and checks that they return 0 in the cases "service does not exist", "service stopped" and "service running". And always deploy in development first, where this failure would have shown up with no consequences.

(e) With blue/green nothing would have happened: the green instances are new and have no previous revision, so ApplicationStop does not even run. If the deployment failed for another reason, blue — untouched, with 1.4.3 working — would keep serving 100 % and the rollback would be sending the ALB listener back. It is an argument in favour of blue/green that does not appear in the tables: it eliminates a whole class of failures at the root, the ones inherited from the previous deployment.

Solution 2

(a) The web shop. EC2 with the ASG, blue/green, terminationWaitTimeInMinutes: 30 and actionOnTimeout: CONTINUE_DEPLOYMENT. Alarms: mercadofresco-alb-latencia-alta and mercadofresco-tienda-degradada. Justification: with 2-4 instances, an in-place deployment takes half the capacity out just when it is needed most; blue/green keeps 100 % throughout the whole process, and the 90-second rollback is what lets you deploy without fear. Added operational rule: no deployment of the shop between 16:00 and 22:00 on a Friday, not out of distrust of the mechanism but because getting it wrong at the peak is disproportionately expensive. Hooks: BeforeAllowTraffic warms the cache — crucial, or the first wave sends latency up and rolls back a good deployment — and AfterAllowTraffic runs a smoke test with real traffic.

(b) The charging Lambda. Canary10Percent5Minutes, with mercadofresco-cobrar-pago-errores-canario (period 60 s, 1 evaluation, threshold 3) and one on Duration p99. Justification: a charging failure is obvious within seconds, not gradual, so the canary detects it sooner and exposes less than the linear one. With 900 invocations/hour, 5 minutes at 10 % means about 7 orders exposed, and the idempotency of 07-05 stops a retry after the rollback duplicating the charge. Plus a BeforeAllowTraffic that invokes the new version with a €1.00 test payment. The detail not to forget: the alarm must carry the Resource dimension of the alias, or it would measure the errors of every version together and be contaminated by invocations of the old one.

(c) The workers. In place with AllAtOnce, with no latency alarms, with mercadofresco-pedidos-fallidos wired to the rollback. Justification: here the reasoning changes completely, and that is what makes this part interesting. They receive no ALB traffic, so there are no requests that can fail: if they stop for five minutes, the queue grows and drains afterwards, which is exactly the buffering we were after in 07-01. Blue/green would be complexity and cost with no benefit. What you do have to guarantee is a clean ApplicationStop, one that lets the message in flight finish and does not delete it halfway, or you will have in-flight messages reappearing when the visibility expires. And watch ApproximateAgeOfOldestMessage after the deployment: if it does not come down within ten minutes, the new consumer is not processing even though the process is alive.

Solution 3

(a) Because the rollback returned the code to 1.5.2 but did not undo the schema. 1.5.2 queries direccion, which no longer exists. Before the rollback at least the shop worked with 1.6.0 and only the consumer was failing; afterwards everything fails, because 100 % of the traffic is served by code incompatible with the schema. It is the trap the lesson warns about: the rollback rolls artefacts back, not databases, and a destructive schema change turns your safety net into an accelerator of the incident.

(b) It ran once per green instance, that is, twice. The first one succeeded; the second failed with ERROR: column "direccion" does not exist, because the rename had already been done. Whether the deployment stopped there depends on whether the script swallowed the error. This behaviour is the demonstration of why migrations do not go in hooks: they are not idempotent and they run as many times as there are instances.

(c) Immediate recovery, with the shop down, so quickly and without elegance. Converge forwards, not backwards: the schema is already in the new state and renaming it back is another migration with risk, so redeploy 1.6.0, which is compatible, and get the shop back. Then, deploy the compatible version of the consumer, which is still on 1.5.2 and still failing; meanwhile the messages pile up in cola-mercadofresco-pedidos and will end up in the DLQ, but they are not lost. With the shop serving, apply the DLQ runbook from 07-05 — contain, classify, reprocess. And a post-mortem with no hunt for culprits, with the process rule written down.

(d) How it should have been done, in three deployments separated by days. Deployment 1 (expand), Monday: schema only, with no new code — ALTER TABLE pedidos ADD COLUMN direccion_entrega VARCHAR(255) NULL; plus an UPDATE copying the values — 1.5.2 keeps working because it does not know the new column and is not bothered by it. Deployment 2 (migrate), Tuesday: the 1.6.0 code writes to both columns and reads from direccion_entrega with a fallback to direccion, which makes it compatible in both directions — it coexists with 1.5.2 during the gradual deployment and works if it has to be rolled back — and here the queue consumer is deployed before the shop, so that it knows how to read the new field before it starts arriving, which is the producer/consumer order that closes 08-05. Deployment 3 (contract), the following week, when no instance is serving 1.5.2 any more: ALTER TABLE pedidos DROP COLUMN direccion;. And each schema phase as a pipeline step of its own, run once from a dedicated task.

(e) Three controls, from cheapest to most expensive. An analysis of the migrations in CodeBuild that fails the build if it finds DROP COLUMN, RENAME COLUMN, ALTER COLUMN ... NOT NULL or DROP TABLE without an explicit "approved contraction" tag: that is thirty lines of script and would have stopped this dead. A backwards compatibility test that starts the previous version of the code against the new schema and runs the smoke suite — it simulates the rollback directly, and it is surprisingly rare in practice. And mandatory review of every change under migraciones/ through a code owners rule, so that no schema change gets in on a distracted approval.

Conclusion

Luis's ritual is over. There is no more ssh, no scp, no forty seconds of errors per instance, no two versions coexisting with no control. There is an application, app-mercadofresco-tienda, two deployment groups that make the same revision behave differently in development and in production, an artefact identified in mercadofresco-artefactos and a plan CodeDeploy executes the same way every time. And above all, something that did not exist: a way back that does not depend on somebody watching.

You know the three targets and what each one supports — on EC2 there is no canary and no linear; on Lambda and ECS there is no in-place deployment — and you know the agent is only needed on EC2, installed as a Systems Manager association so that it updates itself. You have mastered the appspec.yml with its files, permissions and hooks, including file_exists_behavior: OVERWRITE, the first failure everybody hits. And you know the hooks one by one, with the two truths that prevent the most incidents: that ApplicationStop runs from the previous revision — and that is why it must always finish with exit 0 — and that ValidateService is the hook that justifies everything else, checking dependencies and version instead of settling for a live process.

You know how to choose the pace: AllAtOnce only in development, HalfAtATime with spare capacity, OneAtATime when caution matters more than the clock, and why a high FLEET_PERCENT with two instances stops any deployment starting. You have blue/green with its terminationWaitTimeInMinutes of 30 minutes — literally the price of the 90-second rollback — and its hidden advantage: new instances eliminate at the root the failures inherited from the previous deployment. And the canary of mercadofresco-cobrar-pago with Canary10Percent5Minutes, its alarm on the Resource dimension of the alias and periods shorter than the deployment itself, because an alarm slower than what it watches protects nothing.

The centrepiece is the automatic rollback, and now you know what it means in each case: a full deployment in place, ninety seconds in blue/green within the window, seconds in Lambda. You know that choosing the alarms is a design decision with three conditions — that they detect what a bad deployment causes, that they are faster than the deployment and that they have no false positives, because a noisy alarm ends with the team disabling the rollback altogether. And why weighted Route 53 is not the tool for this: the DNS TTL makes rolling back slow and incomplete.

And you take away the most important warning of the module, which is not technical but a matter of process: the application deployment and the schema change do not go in the same step. Because two versions coexist against a single database, because a hook runs once per instance, and above all because the rollback rolls artefacts back, not databases — a badly placed RENAME COLUMN turns your safety net into an accelerator of the incident. The answer is expand and contract in three separate deployments.

And there is what is still missing. Every piece works, but nobody has chained them together. Luis still has to remember to launch the build when he merges, copy the artefact path, write aws deploy create-deployment with the right bucket and key, wait, look at whether it went well and repeat it for the next environment. The schema step is run by hand. Nobody checks that what is deployed in production is exactly what was validated in pre-production, nor is there a moment when Marta formally approves that a change goes out. Excellent tools and zero orchestration: we still depend on one person remembering the sequence on a Friday afternoon.

In 08-04, "AWS CodePipeline", everything gets chained together. We will see the difference between continuous delivery and continuous deployment and why MercadoFresco chooses the first, with manual approval before production; the structure of pipelines, stages, actions and the artefacts that flow between them; the V1 and V2 types, the triggers by branch, tag or file path, and pipeline variables; the manual approval with the exact information Marta needs to decide in thirty seconds; the development, pre-production and production stages; the quality gates with a Lambda action that queries the MercadoFresco/Tienda metrics and stops the pipeline if anything degrades; retries, per-action permissions and cross-account access; and the notifications towards alertas-mercadofresco and Slack.

© Copyright 2026. All rights reserved