The Project | About Us | Contribute | Donations | License

HOME

The table we closed 05-04 with —what runs on save, in the pull request, on deployment and in production— is a list of good intentions for as long as somebody has to remember to run it. And nobody remembers at seven on a Friday evening with an urgent fix in hand.

This lesson turns that list into machinery. We are going to package the Aroma Store API into a Docker image that starts anywhere, bring up the complete environment with Redis in one command, and write the GitHub Actions pipeline that runs, in order, linting, Spectral, the tests with coverage, npm audit, oasdiff, the image build and Newman against staging. Then we will look at what really separates a team that deploys calmly from one that deploys in fear: backwards-compatible migrations, zero-downtime deployment strategies, and a rollback plan that works.

Everything built in module 4 reappears here with an operational role: /health and /health/ready decide when traffic comes in, the metrics and SLOs from 04-07 decide whether the deployment continues or is reverted, and the graceful shutdown from 03-07 is what makes it possible to deploy without cutting requests in half.

Continuous integration, continuous delivery and continuous deployment
Why the main branch must always be deployable
Packaging with Docker: the multi-stage Dockerfile
.dockerignore, layers and image size
The complete environment with docker-compose.yml
Per-environment configuration and secrets outside the image
The CI pipeline: .github/workflows/ci.yml
Version matrix and dependency caching
What makes the pipeline fail and why the gates are inflexible
Building and publishing the image
Migrations: expand, migrate, contract
Deployment strategies
The role of liveness and readiness
Rollback and feature flags
Where to deploy
Artefact versioning and traceability
After the deployment: smoke tests and watching
Secrets in CI and least privilege

Continuous integration, continuous delivery and continuous deployment

Three terms used as synonyms that are not:

	Continuous integration (CI)	Continuous delivery (CD)	Continuous deployment
What it automates	Building and testing every change	Always producing a deployable artefact	Deploying to production with no intervention
Frequency	Every push	Every merge into `main`	Every merge into `main`
Is there a human button?	Not applicable	Yes: somebody decides when	No
Prerequisite	Reliable automated tests	Solid CI + reproducible environments	CD + observability + automatic rollback
Risk of not having it	Branches that diverge for weeks	Manual, fragile deployments	None: it is a legitimate choice

Where Aroma Store stands. Full continuous integration, automatic continuous delivery to staging, and deployment to production with a button. It is the right configuration for the point we are at, and it deserves justification: continuous deployment to production requires that rollback be automatic and that observability detect a degradation within minutes. We have had the second since 04-07; the first will arrive once the canary in section 12 is set up. Adopting continuous deployment before having those two pieces is not maturity, it is recklessness.

The word that matters in all three terms is continuous: small and frequent. A deployment of ten small changes has ten separate opportunities to fail and each failure is trivial to locate. A quarterly deployment with two hundred changes fails once and nobody knows which of the two hundred it was.

Why the main branch must always be deployable

The rule is easy to state and hard to sustain: any commit on main must be deployable to production right now. Several practices follow from it:

Short branches. A three-week branch guarantees a painful conflict and a review that cannot be done properly. One or two days is reasonable.
Gates before merging, not after. If the pipeline runs after the merge, main is broken while somebody fixes it, and the whole team is blocked.
Incomplete features behind a flag. When something is not ready, it is merged switched off with a feature flag instead of living on a separate branch. It is the only real alternative to long-lived branches.
Fixing the pipeline is the absolute priority. A red pipeline that nobody fixes within an hour stops being a signal and becomes noise; from then on people merge on red and the whole system loses its value.

And a less obvious consequence: branch protection must be technical, not cultural. On GitHub, that means branch protection rules with the checks marked as required. A verbal agreement breaks on the day of an emergency; a configured rule does not.

Packaging with Docker: the multi-stage `Dockerfile`

A container solves the oldest problem in the trade: making the application behave the same on your laptop, in CI and in production. It packages the code, its dependencies and the runtime into an immutable image.

New file Dockerfile at the root of the project:

# syntax=docker/dockerfile:1.7

# ============================================================================
# STAGE 1 — build dependencies
# Installs ALL the dependencies (development ones included) because
# better-sqlite3 is a native module and has to be compiled.
# ============================================================================
FROM node:20-bookworm-slim AS dependencies

# Build tools for the native modules. They only live in this
# stage: they never reach the final image, which is the whole point of multi-stage.
RUN apt-get update \
 && apt-get install -y --no-install-recommends python3 make g++ \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# We copy ONLY the manifests before the source code.
# Docker caches by layer: as long as package*.json does not change, the npm ci
# below is reused even if you have touched a hundred files in src/.
COPY package.json package-lock.json ./

# npm ci (not npm install): installs EXACTLY what the lock says, it is reproducible
# and it fails if the lock and package.json disagree. That is what you want in CI.
RUN npm ci

# ============================================================================
# STAGE 2 — production dependencies
# Reinstalls only what is needed to run. It shrinks the final image a lot
# and, above all, the attack surface: less code, fewer CVEs.
# ============================================================================
FROM dependencies AS production-dependencies
RUN npm ci --omit=dev

# ============================================================================
# STAGE 3 — tests (optional, invoked with --target tests)
# Lets you run the suite inside the same image that will be deployed,
# eliminating "it worked on my machine".
# ============================================================================
FROM dependencies AS tests
COPY . .
RUN npm run lint && npm test

# ============================================================================
# STAGE 4 — final runtime image
# ============================================================================
FROM node:20-bookworm-slim AS production

# NODE_ENV=production changes the behaviour of Express (cached views,
# stack traces kept out of responses) and of many libraries. It is mandatory.
ENV NODE_ENV=production \
    PORT=3000 \
    NPM_CONFIG_UPDATE_NOTIFIER=false

# tini is a minimal init: it forwards signals to the child process and reaps
# zombie processes. Without it, a SIGTERM may never reach Node and the
# graceful shutdown from 03-07 never runs.
RUN apt-get update \
 && apt-get install -y --no-install-recommends tini \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Non-root user: the Node image already ships the "node" user (uid 1000).
# Running as root inside the container is an unnecessary risk: if somebody
# achieves code execution, they start with all the container's privileges.
# --chown avoids a later "chown -R", which would duplicate the whole layer.
COPY --chown=node:node --from=production-dependencies /app/node_modules ./node_modules
COPY --chown=node:node package.json ./
COPY --chown=node:node src/ ./src/
COPY --chown=node:node migrations/ ./migrations/
COPY --chown=node:node openapi.yaml ./

USER node

EXPOSE 3000

# HEALTHCHECK uses the LIVENESS endpoint from 04-07, not the readiness one:
# here we ask "is the process alive?", not "can it serve traffic?".
# If we used /health/ready, a Redis outage would restart the container
# in a loop instead of simply taking it out of the load balancer.
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD node -e "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

# tini as PID 1; the Node process is its child and receives signals cleanly.
ENTRYPOINT ["/usr/bin/tini", "--"]

# "exec" form (an array), NEVER shell form. With `CMD node src/server.js` in
# shell form, PID 1 would be /bin/sh and SIGTERM would not reach Node.
CMD ["node", "src/server.js"]

The five decisions in that file that matter most:

Multi-stage. The build tools (python3, make, g++, around 300 MB) stay in stage 1. The final image carries only the runtime and production dependencies.
npm ci --omit=dev. Out go supertest, eslint, prettier, Spectral, autocannon, Prism and Newman. Less size and, above all, less attack surface.
The node user, not root. Combined with readOnlyRootFilesystem in the orchestrator, it closes off a good part of the escalation paths.
tini + exec form. It is what makes SIGTERM reach Node so that the shutdownGracefully() from 03-07 runs. Without this, every deployment cuts requests in half and clients see network errors that appear in no log.
HEALTHCHECK on liveness. The distinction from 04-07 has its practical consequence here: confusing the two endpoints causes cascading restarts when a dependency fails.

Local check:

docker build -t aroma-store-api:local .
docker run --rm -p 3000:3000 --env-file .env.local aroma-store-api:local

# Verify the graceful shutdown: it must exit in under a second,
# not after Docker's forced 10 s timeout.
docker stop $(docker ps -q --filter ancestor=aroma-store-api:local)

If docker stop takes ten seconds, the SIGTERM is not getting through. It is the most useful check and the one almost nobody does.

`.dockerignore`, layers and image size

New file .dockerignore:

# Never enter the image
node_modules
npm-debug.log*
.git
.github
.env
.env.*
*.local.json

# Development and test artefacts
tests/
coverage/
reports/
postman/
docs/
*.md
!README.md

# Local database and temporary files
data/
*.sqlite
*.sqlite-journal
.DS_Store

Three reasons, in order of importance:

Security. Without .dockerignore, a COPY . . puts your .env with the real keys inside an image that may well end up in a shared registry. It is one of the most frequent credential leaks there is.
Correctness. Copying your local node_modules with binaries compiled for macOS into a Linux image produces incomprehensible failures.
Speed and size. .git can weigh hundreds of megabytes and is sent in full to the Docker daemon on every build.

About layers: each instruction creates a layer, and Docker caches them. The order of the Dockerfile is not aesthetic, it is a caching strategy: what changes rarely at the top, what changes often at the bottom. Copying package*.json before src/ means a code change reuses the npm ci, which is the expensive instruction. The other way round, every build would reinstall everything.

Approximate size reference for choosing a base:

Base	Final image size	Notes
`node:20`	~1.1 GB	Full Debian. Only if you need lots of tooling.
`node:20-bookworm-slim`	~250 MB	Ours. A good balance; glibc, no surprises with native modules.
`node:20-alpine`	~180 MB	musl instead of glibc: it can cause problems with native modules such as `better-sqlite3`.
`gcr.io/distroless/nodejs20`	~170 MB	No shell and no package manager: maximum security, awkward debugging.

The recommendation for Aroma Store is bookworm-slim: Alpine saves 70 MB and can cost you an afternoon compiling better-sqlite3 against musl. Distroless is an excellent choice when the team has operational maturity, but stepping into the container to debug stops being an option.

And a security tip that fits with 04-02: scan the image.

# Known vulnerabilities in the built image
docker scout cves aroma-store-api:local
# or
trivy image aroma-store-api:local --severity HIGH,CRITICAL

The complete environment with `docker-compose.yml`

New file docker-compose.yml:

# docker-compose.yml — the complete Aroma Store environment for development and testing
services:
  api:
    build:
      context: .
      target: production
    ports:
      - "3000:3000"
    environment:
      NODE_ENV: development
      PORT: 3000
      # In Compose, services resolve by name: "redis" is a host.
      REDIS_URL: redis://redis:6379
      DATABASE_PATH: /data/aroma.sqlite
      # Secrets do NOT go here in plain text: they come from the local .env file,
      # which is in .gitignore and in .dockerignore.
      JWT_SECRET: ${JWT_SECRET:?JWT_SECRET is missing from .env}
      ALLOWED_ORIGINS: http://localhost:5173,http://localhost:4173
      LOG_LEVEL: debug
    volumes:
      # Named volume so the database survives docker compose down
      - api-data:/data
    depends_on:
      redis:
        condition: service_healthy      # it does not start until Redis answers
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 15s
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    # appendonly yes: the rate limiting from 04-04 and the cache from 04-06 can
    # tolerate losing data, but in development it is convenient for it to survive a restart.
    command: ["redis-server", "--appendonly", "yes", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  # Contract mock (05-04): lets the SPA work without depending on the API.
  mock:
    image: stoplight/prism:5
    command: ["mock", "-h", "0.0.0.0", "-p", "4010", "--errors", "/tmp/openapi.yaml"]
    ports:
      - "4010:4010"
    volumes:
      - ./openapi.yaml:/tmp/openapi.yaml:ro
    profiles: ["development"]     # only starts with --profile development

  # PostgreSQL: mentioned here because it is the natural destination when SQLite
  # falls short. Migrating means changing only src/repositories/*, thanks to the
  # repository pattern from 03-05; the rest of the project never notices.
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: aroma
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-fictional-local-password}
      POSTGRES_DB: aroma
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U aroma"]
      interval: 5s
      retries: 5
    profiles: ["postgres"]

volumes:
  api-data:
  redis-data:
  postgres-data:

Everyday use:

docker compose up -d --wait                 # brings up API and Redis, waits until healthy
docker compose --profile development up -d  # adds the Prism mock
docker compose logs -f api                  # follows the pino logs
docker compose exec api npm run migrate     # runs a migration inside the container
docker compose down -v                      # destroys everything, volumes included

Two details that save hours of debugging:

condition: service_healthy in depends_on. Without it, depends_on only waits for the container to start, not for the service to work, and the API tries to connect to a Redis that is not accepting connections yet.
${JWT_SECRET:?JWT_SECRET is missing ...}. That syntax makes Compose fail with a clear message if the variable is not defined, instead of starting with an empty secret. Starting with an empty secret is worse than not starting.

And an additional file docker-compose.test.yml, the one the end-to-end tests from 05-04 use: the same but with NODE_ENV=test, an ephemeral database (tmpfs, with no persistent volume) and no exposed ports other than the API's.

Per-environment configuration and secrets outside the image

The principle, taken from the Twelve-Factor App: one image, many environments. The same image that passed the tests is the one that goes to staging and then to production, with no rebuild. If you rebuild for each environment, you are not deploying what you tested.

Everything that varies between environments is an environment variable:

Variable	development	test	staging	production
`NODE_ENV`	`development`	`test`	`production`	`production`
`LOG_LEVEL`	`debug`	`silent`	`info`	`info`
`DATABASE_PATH`	local file	`:memory:`	volume	managed
`REDIS_URL`	`redis://redis:6379`	`redis://redis:6379`	internal	managed with TLS
`JWT_SECRET`	fictional in `.env`	fixed fictional	from the secret manager	from the secret manager
`ALLOWED_ORIGINS`	`localhost:5173`	`localhost`	`*.test.aromastore.example`	`aromastore.example`, `panel.…`
`GLOBAL_LIMIT_PER_MINUTE`	`10000`	`10000`	`600`	`600`
`PUBLIC_DOCS`	`true`	`true`	`true`	`false`
`TRACE_SAMPLING`	`1.0`	`0`	`1.0`	`0.05`

Four rules about secrets that admit no exception:

No secret inside the image. Not in the Dockerfile, not with ARG —ARGs stay in the layer history and can be seen with docker history— and not in a copied file.
No secret in the repository. .env in .gitignore; .env.example with the keys and fictional values, versioned, to document what is needed.
Secrets are injected at run time by the orchestrator, from its manager: GitHub Secrets, AWS Secrets Manager, Google Secret Manager, HashiCorp Vault, Kubernetes Secrets.
src/config/environment.js validates at start-up. We already wrote it in 03-01, and here you see why it matters: if JWT_SECRET is missing, the process must die immediately with a clear message, not start and fail on the first authenticated request. Failing fast and loudly at start-up is what stops a misconfigured deployment from ever receiving traffic.

The CI pipeline: `.github/workflows/ci.yml`

New file .github/workflows/ci.yml:

name: CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

# Cancels earlier runs on the same branch: if you push three times in a row,
# only the last one runs. It saves CI minutes and gives feedback sooner.
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

# Minimum permissions by default (section 18). Each job asks for what it needs.
permissions:
  contents: read

env:
  MAIN_NODE_VERSION: '20'

jobs:
  # --------------------------------------------------------------------------
  # 1. Static quality: fast and with no external dependencies. Fails in 30 s.
  # --------------------------------------------------------------------------
  quality:
    name: Lint and formatting
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.MAIN_NODE_VERSION }}
          cache: npm            # caches ~/.npm using package-lock.json as the key

      - name: Install dependencies
        run: npm ci

      - name: ESLint
        run: npm run lint

      - name: Prettier (check, not write)
        run: npx prettier --check .

  # --------------------------------------------------------------------------
  # 2. The contract: structural validity + style guide + breaking changes.
  #    It runs in parallel with the tests because it does not depend on them.
  # --------------------------------------------------------------------------
  contract:
    name: Contract validation
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0        # we need the history to compare against main

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.MAIN_NODE_VERSION }}
          cache: npm

      - run: npm ci

      - name: Is it a valid OpenAPI document? (05-02)
        run: npx swagger-cli validate openapi.yaml

      - name: Does it follow the style guide? (Spectral, 04-01)
        run: npx spectral lint openapi.yaml --fail-severity=error

      - name: Install oasdiff
        run: |
          curl -fsSL https://raw.githubusercontent.com/oasdiff/oasdiff/main/install.sh | sh

      - name: Does it introduce breaking changes? (05-04)
        if: github.event_name == 'pull_request'
        run: bash tools/check-contract.sh origin/main

  # --------------------------------------------------------------------------
  # 3. Tests: unit + integration + contract, on several Node versions.
  # --------------------------------------------------------------------------
  tests:
    name: Tests (Node ${{ matrix.node }})
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false          # let everything that should fail, fail
      matrix:
        node: ['20', '22']      # current LTS and the next one: catches breakage early

    services:
      redis:
        image: redis:7-alpine
        ports: ['6379:6379']
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 3s
          --health-retries 5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: npm

      - run: npm ci

      - name: Migrate and seed the test database
        run: npm run db:reset
        env:
          DATABASE_PATH: ':memory:'

      - name: Tests with coverage
        run: node --test --experimental-test-coverage tests/
        env:
          NODE_ENV: test
          REDIS_URL: redis://localhost:6379
          JWT_SECRET: fictional-secret-for-ci-only
          LOG_LEVEL: silent

      - name: Publish the coverage report
        if: matrix.node == '20'
        uses: actions/upload-artifact@v4
        with:
          name: coverage
          path: coverage/
          retention-days: 7

  # --------------------------------------------------------------------------
  # 4. Dependency security (04-02).
  # --------------------------------------------------------------------------
  security:
    name: Dependency audit
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.MAIN_NODE_VERSION }}
          cache: npm

      - run: npm ci

      # Fails on high or critical vulnerabilities. Moderate ones are reviewed
      # but do not block: if they did, the pipeline would be permanently red
      # because of transitive dependencies and the team would learn to ignore it.
      - name: npm audit
        run: npm audit --audit-level=high

      - name: Check that the lock is in sync
        run: |
          npm ci --dry-run 2>&1 | tee /tmp/output
          ! grep -q "npm warn" /tmp/output || echo "Review the npm warnings"

  # --------------------------------------------------------------------------
  # 5. Image: only if everything above passed. On a PR it is built but not
  #    published; on main it is published tagged with the commit SHA.
  # --------------------------------------------------------------------------
  image:
    name: Build and publish the image
    needs: [quality, contract, tests, security]
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write           # needed to push to the registry
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - name: Authenticate against the registry
        if: github.ref == 'refs/heads/main'
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Tags and metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}/api
          tags: |
            type=sha,format=long          # the tag = the commit: traceability
            type=ref,event=branch
            type=semver,pattern={{version}}

      - name: Build (and publish only on main)
        uses: docker/build-push-action@v6
        with:
          context: .
          target: production
          push: ${{ github.ref == 'refs/heads/main' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan the image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ghcr.io/${{ github.repository }}/api:sha-${{ github.sha }}
          severity: 'HIGH,CRITICAL'
          exit-code: '1'
          ignore-unfixed: true    # with no patch available, there is nothing to do

  # --------------------------------------------------------------------------
  # 6. Deployment to staging and verification (main only).
  # --------------------------------------------------------------------------
  staging:
    name: Deploy to staging and verify
    needs: [image]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment:
      name: staging
      url: https://api-test.aromastore.example
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.MAIN_NODE_VERSION }}
          cache: npm
      - run: npm ci

      - name: Apply migrations (before the deployment, section 11)
        run: npm run migrate
        env:
          DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}

      - name: Deploy the image
        run: ./tools/deploy.sh staging "sha-${{ github.sha }}"
        env:
          DEPLOY_TOKEN: ${{ secrets.STAGING_DEPLOY_TOKEN }}

      - name: Wait for readiness to go green (04-07)
        run: |
          for attempt in $(seq 1 30); do
            if curl -fsS https://api-test.aromastore.example/health/ready; then
              echo "Ready after ${attempt} attempts."; exit 0
            fi
            sleep 5
          done
          echo "The service did not become ready within 150 s."; exit 1

      - name: End-to-end tests (05-04)
        run: node --test tests/e2e/
        env:
          API_URL: https://api-test.aromastore.example/v1
          TEST_EMAIL: ${{ secrets.TEST_EMAIL }}
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}

      - name: Postman collection with Newman (05-01)
        run: |
          npx newman run postman/aroma-store-v1.postman_collection.json \
            -e postman/test.postman_environment.json \
            --env-var "customerPassword=${{ secrets.TEST_PASSWORD }}" \
            --delay-request 100 \
            --reporters cli,junit \
            --reporter-junit-export reports/newman.xml

      - name: Publish the Newman report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: newman-report
          path: reports/newman.xml

Version matrix and dependency caching

The matrix. matrix: node: ['20', '22'] runs the tests twice. The cost is double the CI time; the benefit is discovering months in advance that something breaks on the next LTS, while there is still time to fix it calmly rather than under end-of-support pressure.

fail-fast: false matters: by default, GitHub cancels the rest of the matrix on the first failure, and then you do not know whether the problem is Node 22 or your code. With false you see both results.

When the matrix is worth it: in libraries, almost always; in a deployed application, when a major version migration is foreseeable and you want to get ahead of it. In Aroma Store, yes, because we deploy on Node 20 and 22 will be the next LTS.

The cache. cache: npm in setup-node caches the ~/.npm directory using the hash of package-lock.json as the key. Typical effect: npm ci drops from 60-90 seconds to 10-15.

A nuance that confuses people: it caches the download, not node_modules. That is deliberate. Caching node_modules is a classic source of phantom failures —native modules compiled for another version, leftovers from earlier installs— and it is not reproducible. npm ci deletes node_modules and reinstalls from scratch; the only saving is the network.

For the image, the cache is different: cache-from: type=gha stores the Docker layers in the GitHub Actions cache, so the expensive stage —npm ci with the better-sqlite3 compilation— is reused as long as package-lock.json does not change.

What makes the pipeline fail and why the gates are inflexible

Gate	Fails if	Does it block the merge?	Why
ESLint	Any error	Yes	The warnings are already filtered out; an error is an error
Prettier	An unformatted file	Yes	Automatic formatting: no excuse and no discussion
`swagger-cli validate`	The contract is not valid OpenAPI	Yes	It breaks the documentation and every generator
Spectral	Any `error` rule	Yes	It is the agreed style guide (04-01)
`oasdiff breaking`	There are breaking changes	Yes, unless explicitly labelled	It breaks Aroma Mobile, and it is permanent
Tests	A single one fails	Yes	Obvious
Coverage	It drops below the agreed threshold	Yes, but with judgement	See below
`npm audit`	A high or critical vulnerability	Yes	With `--audit-level=high`: moderate ones do not block
Trivy	A high or critical CVE with a patch	Yes	`ignore-unfixed`: with no patch there is no possible action
E2E in staging	One fails	Yes, it is not promoted to production	It is the last net before the customers

Why inflexibility matters. A gate that can be skipped "just this once" stops being a gate at the third "just this once". The rule that works is: if a gate is a nuisance, the rule is changed through a pull request that modifies it —discussed and visible— it is not skipped on a particular run. That is exactly the Spectral procedure from 04-01: a rule comes in as warn, the violations are cleaned up and then it is promoted to error.

The only legitimate escape hatch is explicit, leaves a trace and demands justification. For oasdiff:

      - name: Does it introduce breaking changes?
        if: >
          github.event_name == 'pull_request' &&
          !contains(github.event.pull_request.labels.*.name, 'breaking-change')
        run: bash tools/check-contract.sh origin/main

Adding the breaking-change label is a deliberate act, visible on the pull request, that forces an explanation and can additionally require the approval of a specific person through CODEOWNERS.

About coverage. A threshold is useful as a net against carelessness —"let us not drop below 80 %"— and perverse as a goal: chasing 100 % produces tests that execute code without checking anything. The sensible configuration is to require that coverage does not drop relative to main, rather than an absolute number.

Building and publishing the image

The image job always builds and only publishes from main. Two good consequences: pull requests verify that the Dockerfile still works —a build failure is caught in review, not on merge— and the registry does not fill up with images from ephemeral branches.

The tags docker/metadata-action produces:

ghcr.io/aromastore/api:sha-3f9a2c1e8b474d2a9e0177c6b5d3a8129e4c1b2d
ghcr.io/aromastore/api:main
ghcr.io/aromastore/api:1.7.0        (if the commit carries a version tag)

The tag that gets deployed is always the SHA one. Never latest, never main. The reason: those are moving tags. If you deploy main and tomorrow you have to investigate what was in production on Tuesday, the answer is "nobody knows". With sha-3f9a2c… the answer is an exact commit, with its diff, its author and its pull request. It is the traceability principle from section 16.

cache-from/cache-to: type=gha reuses the layers between runs. Without it, every build recompiles better-sqlite3 from scratch: two or three minutes per run that turn into twenty seconds.

Migrations: expand, migrate, contract

Here is the most underestimated problem in continuous deployment. During a zero-downtime deployment, two versions of the code coexist over a single database. If the migration is not compatible with both, errors are guaranteed.

A concrete scenario. We want to rename tasting_notes to notes in the coffees table.

The naive way, which breaks:

-- migrations/013-rename-notes.sql  ← DO NOT DO THIS
ALTER TABLE coffees RENAME COLUMN tasting_notes TO notes;

The sequence of events: the migration is applied; over the following two minutes, the instances with the old code carry on serving traffic and running SELECT tasting_notes FROM coffees; that column no longer exists; every catalogue request returns 500 until the deployment finishes. And if you have to revert, the old code does not work either: you have lost your rollback.

The correct way: expand → migrate → contract. Three separate deployments.

-- STEP 1 — EXPAND (deployment 1). It only adds. Compatible with everything.
ALTER TABLE coffees ADD COLUMN notes TEXT;
UPDATE coffees SET notes = tasting_notes WHERE notes IS NULL;

// Deployment 1's code: writes to BOTH, reads from the old one.
export function saveCoffee(coffee) {
  db.prepare(`
    UPDATE coffees SET tasting_notes = ?, notes = ? WHERE id = ?
  `).run(JSON.stringify(coffee.tastingNotes), JSON.stringify(coffee.tastingNotes), coffee.id);
}

export function readCoffee(id) {
  const row = db.prepare('SELECT * FROM coffees WHERE id = ?').get(id);
  return { ...row, tastingNotes: JSON.parse(row.tasting_notes) };  // still the old one
}

// STEP 2 — MIGRATE (deployment 2). Writes to both, READS FROM THE NEW ONE.
// If something goes wrong, you revert to deployment 1 with no data loss, because
// both columns are populated and in sync.
export function readCoffee(id) {
  const row = db.prepare('SELECT * FROM coffees WHERE id = ?').get(id);
  return { ...row, tastingNotes: JSON.parse(row.notes ?? row.tasting_notes) };
}

-- STEP 3 — CONTRACT (deployment 3, days or weeks later).
-- Only when NO instance with the old code can still be alive
-- and rolling back to that version is no longer a realistic option.
ALTER TABLE coffees DROP COLUMN tasting_notes;

Practical migration rules:

Operation	Safe during a deployment?	Note
`ADD COLUMN` with a default or nullable	Yes	The safe way to add
`ADD COLUMN NOT NULL` with no default	No	The old code's writes fail
`DROP COLUMN`	No	Only in the contract phase
`RENAME COLUMN`	No	It is a `DROP` in disguise. Expand/contract
`CREATE INDEX`	It depends	In PostgreSQL, `CONCURRENTLY`; without it, it locks the table
`ALTER TYPE` that narrows	No	Existing data may not fit
Adding a table	Yes	Nobody uses it yet
Adding a `NOT NULL` constraint	No	Expand: backfill, validate and then constrain

When they are applied. Before deploying the new code, in a pipeline step of its own, as in the staging job in section 7. Never on application start-up: with three instances starting at once you would have three processes migrating in parallel over the same database. If your migration system does not take a lock, the result is unpredictable.

And migrations have to be tested. A test that applies all the migrations to an empty database and checks the resulting schema costs little and avoids the worst kind of incident: the one that happens in the step before the deployment, when there is nothing deployed yet to revert.

Deployment strategies

Strategy	How it works	Downtime	Cost	Risk	Rollback	When
Recreate	Stops everything, starts the new	Yes, seconds or minutes	Minimal	High	Redeploy	Development; systems that tolerate downtime
Rolling	Replaces instances in batches	No	Minimal	Medium	Reverse rolling, slow	The default case
Blue-green	Two complete environments; traffic is switched	No	Double infrastructure	Low	Instant	Critical deployments
Canary	1-5 % of traffic to the new version; increased if the metrics hold	No	Medium	Very low	Automatic, driven by metrics	High volume; risky changes

graph TD
    subgraph Rolling
    R1[3 instances v1] --> R2[2 v1 + 1 v2] --> R3[1 v1 + 2 v2] --> R4[3 instances v2]
    end
    subgraph BlueGreen
    B1[Blue v1 takes traffic<br/>Green v2 is deployed and warmed up] --> B2[Switch the load balancer] --> B3[Green v2 takes traffic<br/>Blue v1 on standby to revert]
    end
    subgraph Canary
    C1[100% to v1] --> C2[95% v1 and 5% v2<br/>watch errors and latency] --> C3{SLOs green?}
    C3 -->|Yes| C4[50% and 50%, then 100% v2]
    C3 -->|No| C5[Back to 100% v1<br/>automatically]
    end

Rolling is the default mode in Kubernetes and in almost every orchestrator, and it is a sensible choice. Its essential condition is the one from the previous section: during the replacement both versions coexist, so the database and the contract must be compatible with both.

Blue-green is very reassuring —reverting is switching the load balancer back, a matter of seconds— and expensive: during the deployment you pay for double the infrastructure. And beware: the database is not blue-green. It is shared, so migrations still have to be backwards-compatible.

Canary is the strategy that makes real continuous deployment viable, because it joins deployment with the observability from 04-07: a small percentage of traffic is sent to the new version and its error rate and p99 latency are automatically compared against the stable version's. If they degrade, it reverts by itself. It requires enough volume for the metrics to be meaningful —with ten requests a minute, 5 % says nothing— and a gateway or mesh that knows how to split traffic, which is exactly what we will see in 05-06.

The connection with /v1 and /v2 from 02-07. It is worth not confusing two things that look alike:

Deploying v1.7.0 over v1.6.0 is a deployment: same contract, new code. Rolling, blue-green or canary apply here.
Publishing /v2 alongside /v1 is API version coexistence: two different contracts alive for months, with Deprecation and Sunset on /v1. It is not a deployment strategy, it is a product decision.

They combine: /v2 is deployed with rolling like any other version, and both paths coexist in the same service (or in separate services behind the gateway, which is cleaner for being able to retire /v1 by switching something off).

The role of liveness and readiness

The two endpoints from 04-07 stop being theory the moment there is an orchestrator in front.

// src/app.js — a reminder of the distinction, positions 10 and 11
// LIVENESS: is the process alive? No dependencies at all.
// If it answers badly, the orchestrator KILLS and RESTARTS the container.
app.get('/health', (req, res) => res.json({ status: 'alive' }));

// READINESS: can I serve traffic NOW? It checks dependencies.
// If it answers badly, the orchestrator TAKES the instance out of the load balancer,
// but does NOT restart it: Redis may come back in ten seconds.
app.get('/health/ready', async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    migrationsUpToDate: await checkMigrations(),
  };
  const ready = Object.values(checks).every(Boolean);
  res.status(ready ? 200 : 503).json({ status: ready ? 'ready' : 'not_ready', checks });
});

The equivalent Kubernetes configuration, which shows how they are used:

        livenessProbe:
          httpGet: { path: /health, port: 3000 }
          initialDelaySeconds: 10
          periodSeconds: 30
          failureThreshold: 3          # three failures in a row → restart

        readinessProbe:
          httpGet: { path: /health/ready, port: 3000 }
          initialDelaySeconds: 5
          periodSeconds: 5             # more frequent: reacts quickly
          failureThreshold: 2

        # startupProbe: gives start-up some slack without relaxing liveness.
        # Until it passes, liveness and readiness are not evaluated.
        startupProbe:
          httpGet: { path: /health, port: 3000 }
          periodSeconds: 5
          failureThreshold: 30         # up to 150 s to start

        lifecycle:
          preStop:
            # A wait before the SIGTERM: it gives the load balancer time to notice
            # that this instance is going away, avoiding requests routed to a
            # process that is already shutting down. It is the number one cause of 502s
            # during "zero-downtime" deployments.
            exec: { command: ["sleep", "5"] }
        terminationGracePeriodSeconds: 30

The three classic mistakes, which produce incidents that are very hard to diagnose:

Using readiness as liveness. Redis goes down, readiness answers 503, the orchestrator restarts every container in a loop, and now you have two problems.
A liveness check that queries the database. A slow query makes liveness fail and restarts a perfectly healthy application, making the load on the database worse.
Forgetting the preStop. Without it, the load balancer can keep sending requests for a second or two to a process that has already received SIGTERM, and sporadic 502s appear on every deployment that nobody manages to reproduce.

The complete chain of a clean shutdown, joining 03-07 with this lesson: preStop (5 s of slack) → SIGTERM → tini forwards it to Node → shutdownGracefully() → server.close() stops accepting new connections and finishes the in-flight ones → the database and Redis are closed → process.exit(0). If something gets stuck, terminationGracePeriodSeconds finishes it with a SIGKILL after 30 seconds.

Rollback and feature flags

Reverting the code is easy. It is redeploying the previous tag:

./tools/deploy.sh production "sha-<previous-commit>"

That it is trivial depends on three things we already have: immutable images tagged by commit, no configuration inside the image, and backwards-compatible migrations. The third is usually the one that is missing.

Reverting the data is the hard problem, and it is worth being clear about before you need it:

If the new version wrote data in a format the old one does not understand, reverting the code fixes nothing.
If the migration dropped a column, reverting the code leaves it pointing at something that no longer exists.
Restoring a backup means losing everything that has happened since the backup: orders, payments, reviews. It is almost never acceptable.

Hence the golden rule: there is no rollback for a destructive migration; you prevent it. Expand → migrate → contract is not bureaucratic ceremony, it is precisely what makes reverting possible.

Feature flags as an alternative. Instead of reverting the deployment, you switch the feature off:

// src/config/flags.js
// Flags read from the environment or from a configuration service.
// Changing them does NOT require a deployment: that is their whole point.
export const flags = {
  recommendationsInCatalogue: environment.readBoolean('FLAG_RECOMMENDATIONS', false),
  newShippingCalculation: environment.readBoolean('FLAG_NEW_SHIPPING_CALCULATION', false),
};

// src/services/orders.js
const shippingCost = flags.newShippingCalculation
  ? calculateShippingByZone(order)     // new logic, switchable off in a second
  : calculateFlatShipping(order);      // old logic, untouched

Advantages: they separate deploying from enabling, they allow enabling for one group only (employees first, then 5 % of customers), and switching off is instant with no deployment. Drawbacks that have to be managed: each flag doubles code paths and therefore test cases, and flags expire: a flag that has been on for a year is technical debt. The discipline that works is writing the removal date in the same commit that creates it.

Where to deploy

Option	Effort	Control	Cost	Scaling	Fit with Aroma Store
PaaS (Render, Railway, Fly.io)	Very low	Low	Medium	Automatic, limited	Excellent to start with: `git push` and done
Managed containers (Cloud Run, ECS Fargate)	Low	Medium	Low-medium	Automatic, down to zero	The sensible option: Docker without operating servers
Managed Kubernetes (GKE, EKS, AKS)	High	High	Medium-high	Total	Only with several services and somebody to operate it
Serverless (Lambda, Cloud Functions)	Low	Low	Very low with no traffic	Automatic	Problematic: cold starts and connections (05-03)
Your own VPS	Medium	Total	Low	Manual	Valid if the team knows systems administration

Recommendation for Aroma Store: Cloud Run or ECS Fargate. The reasoning: we already have the Docker image, which is all they ask for; scaling is automatic; there are no servers to patch; and /health/ready plugs straight in as the health check. Kubernetes would give more control than we need today and demands part-time operational attention that a small team does not have.

A warning about Kubernetes, because it is the industry's most common oversized decision: it is an excellent tool for its problem, which is orchestrating many services with teams able to operate them. Adopting it for one API is swapping the problem "deploy an application" for the problem "operate Kubernetes", which is far bigger. The right question is not "is it good?", but "do we have its problem?".

Artefact versioning and traceability

Faced with an incident, three questions must be answerable in under a minute: which version is deployed, what does it contain and when did it arrive.

The practices that guarantee it:

Image tag = commit. sha-3f9a2c1e…. No possible ambiguity.
The application exposes its version, at the root or at /health:

// src/routes/health.js
app.get('/health', (req, res) => {
  res.json({
    status: 'alive',
    version: environment.apiVersion,      // 1.7.0, from the contract's info.version
    commit: environment.commitSha,        // injected as an environment variable
    deployedAt: environment.deployedAt,
  });
});

The version as a label on the metrics from 04-07. aroma_requests_total{version="1.7.0"} lets you see the exact moment of the deployment on a chart and correlate it with a change in the error rate. It is the signal that resolves incidents fastest: "it started right at the deployment".
The version on every log line, which you already get for free from pino's base logger.
A deployment log: which version, who, when, to which environment. GitHub Deployments does it by itself with environment: in the workflow.

A note about semantic versions: info.version in openapi.yaml describes the contract; the SHA describes the code. They do not coincide: twenty commits can share contract 1.7.0. Both are necessary and they answer different questions.

After the deployment: smoke tests and watching

The deployment does not end when the pipeline goes green. It ends when somebody has checked that the system is fine.

Smoke tests. A minimal, fast, non-destructive subset, run against production right after deploying:

#!/usr/bin/env bash
# tools/smoke.sh — minimal checks after the deployment.
# It does NOT create data: in production, a destructive test is unacceptable.
set -euo pipefail
URL="${1:?usage: smoke.sh https://api.aromastore.example}"

echo "1/5 liveness"
curl -fsS "${URL}/health" | grep -q '"status":"alive"'

echo "2/5 readiness (dependencies)"
curl -fsS "${URL}/health/ready" | grep -q '"status":"ready"'

echo "3/5 the deployed version is the expected one"
DEPLOYED=$(curl -fsS "${URL}/health" | sed -n 's/.*"commit":"\([^"]*\)".*/\1/p')
[ "${DEPLOYED}" = "${EXPECTED_COMMIT}" ] || {
  echo "Deployed version ${DEPLOYED}, expected ${EXPECTED_COMMIT}"; exit 1; }

echo "4/5 the catalogue answers and requires authentication"
curl -fsS -o /dev/null -w '%{http_code}' "${URL}/v1/coffees" | grep -q 401

echo "5/5 the contract is published"
curl -fsS "${URL}/docs/openapi.json" | grep -q '"openapi"'

echo "Smoke tests passed."

Five checks in two seconds that catch the most frequent deployment failures: the application does not start, a dependency is not reachable from production, the wrong image was deployed, or authentication has been misconfigured.

Watching during the deployment. This is where 04-07 pays off. The first fifteen minutes, watching:

Signal	Alarm threshold	Action
`5xx` error rate	Any rise above the baseline	Revert and then investigate
p99 latency	+20 % over the baseline	Investigate; revert if it gets worse
Request rate	A sharp drop	Somebody is not getting through: DNS, load balancer, CORS
SLO error budget	Accelerated consumption	Revert
Memory and event loop	Sustained growth	A leak possibly introduced in this version
Business metrics (`aroma_orders_created_total`)	A drop	The most important signal: the real impact

The last row deserves emphasis. A deployment can be technically perfect —zero errors, wonderful latency— and have broken the business, because a change in the validation means no order completes. The 5xxs do not catch it; orders per minute do. Always watch at least one business metric during a deployment.

And the most important cultural rule: restore the service first, understand what happened afterwards. The temptation to investigate with production broken is enormous and it is always a mistake. You revert, take a breath and investigate with the system stable.

Secrets in CI and least privilege

The CI pipeline is a first-order attack target: it has deployment credentials and the ability to run arbitrary code. The essential controls:

1. Minimum permissions by default. It is already in the workflow in section 7:

permissions:
  contents: read      # the default for every job

and each job raises only its own (packages: write in image alone). Without this declaration, the GITHUB_TOKEN has broad write permissions over the repository, and any compromised third-party action inherits them.

2. Never print secrets. GitHub masks known values in the logs, but it is easy to defeat that by accident: an env | sort or a base64-encoded secret shows up in the clear. Simple rule: never dump the environment into a log.

3. Pin third-party actions. uses: somebody/action@v3 follows a moving tag: if that tag is moved to malicious code, it runs in your pipeline with access to your secrets. It has happened. The safe thing is to pin the SHA:

      - uses: docker/build-push-action@4f58ea79222b3b9dc2c8bbdd6debcef730109a75  # v6.9.0

4. Protected environments. GitHub's environment: production lets you require human approval, restrict which branches can deploy and store secrets accessible only from that environment. That way, the production token is out of reach of a pull request workflow.

5. No secrets in workflows for pull requests from forks. A pull request from a fork runs code you do not control. GitHub does not expose secrets to pull_request from forks by default; do not change that with pull_request_target without understanding exactly what it implies. It is one of the most exploited credential-exfiltration routes.

6. Ephemeral credentials instead of long-lived keys. OIDC between GitHub Actions and the cloud provider issues a short-lived token per run, instead of storing a permanent access key. It is the same idea as 04-03 —short-lived tokens, limited scope— applied to infrastructure.

7. Rotation and auditing. Secrets expire and are rotated; access is logged. And if a secret has ever appeared in a log or a commit, it is considered compromised: you rotate it, you do not delete the commit and carry on as if nothing had happened.

Common Mistakes and Tips

COPY . . with no .dockerignore. It puts your .env with the real keys inside the image. It is one of the most frequent credential leaks there is.
CMD node src/server.js in shell form. PID 1 becomes /bin/sh, SIGTERM does not reach Node and every deployment cuts requests in half. Exec form and tini.
Using readiness as liveness. Redis goes down and the orchestrator restarts every container in a loop. Liveness with no dependencies; readiness with them.
DROP COLUMN during a zero-downtime deployment. The old version is still alive running SELECTs on that column. Expand → migrate → contract, always.
Migrating on application start-up. With three instances, three migrations in parallel over the same database. A step of its own in the pipeline, with a lock.
Deploying the latest tag. It moves: you do not know what is in production and you cannot revert precisely. The tag is the commit SHA.
Rebuilding the image for each environment. Then you are not deploying what you tested. One image, many configurations.
CI gates that can be skipped. By the third "just this once" they no longer exist. They are changed with a pull request, not skipped on a run.
Blocking with npm audit at moderate level. The pipeline will be permanently red because of transitive dependencies and the team will learn to ignore it. --audit-level=high.
A workflow with permissions: write-all. Any compromised third-party action inherits the power to write to your repository.
Tip: check docker stop locally. If it takes ten seconds, the SIGTERM is not getting through and your graceful shutdown will not run in production either.
Tip: export the deployed version as a metric label. Seeing the exact step in the error chart at the moment of the deployment resolves incidents in seconds.
Tip: watch a business metric during every deployment. A technically perfect deployment may have broken order creation, and the 5xxs will not tell you.
Tip: write an ADR (04-01) with the deployment strategy you chose. In a year somebody will ask why you do not do canary, and the answer should be written down.

Exercises

Exercise 1: completing the pipeline with a production deployment job

Extend .github/workflows/ci.yml with a production job that runs after staging, requires human approval, applies migrations, deploys the same image that was verified in staging, runs the smoke tests and reverts automatically if they fail. Explain why exactly the same image tag must be deployed rather than rebuilt.

Exercise 2: designing a safe migration

Aroma Store needs to split the origin field (today "Ethiopia") into two: country and region ("Ethiopia" / "Yirgacheffe"), because the search has to filter by region. There are 2,400 coffees in production and the system deploys with rolling over three instances.

Design the complete expand → migrate → contract sequence: what SQL at each step, what the code does at each deployment, how much time you would leave between steps and why, how you would handle the API contract in each phase (remember 02-07), and at exactly which point the possibility of reverting is lost.

Exercise 3: choosing a deployment strategy for three changes

For each change, choose between recreate, rolling, blue-green and canary, justify the choice, state what you would watch during the deployment and describe the rollback plan:

A) A fix for a bug in the VAT calculation on invoices. It affects every order and there are customers paying right now.

B) A change to the catalogue's default sort algorithm, which now prioritises the best-rated coffees. It does not change the contract, but it changes what every user sees.

C) Migrating persistence from SQLite to PostgreSQL, with the same API contract and the same code except for src/repositories/.

Solutions

Solution 1

  # --------------------------------------------------------------------------
  # 7. Production: human approval, deployment, smoke tests and rollback.
  # --------------------------------------------------------------------------
  production:
    name: Deploy to production
    needs: [staging]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment:
      name: production            # with "required reviewers" configured in GitHub:
      url: https://api.aromastore.example   # the job waits for human approval
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v4

      - name: Note the version that was deployed (for the rollback)
        id: previous
        run: |
          CURRENT=$(curl -fsS https://api.aromastore.example/health \
            | sed -n 's/.*"commit":"\([^"]*\)".*/\1/p')
          echo "sha=${CURRENT}" >> "$GITHUB_OUTPUT"
          echo "Current version in production: ${CURRENT}"

      - name: Apply migrations (backwards-compatible, section 11)
        run: npm ci && npm run migrate
        env:
          DATABASE_URL: ${{ secrets.PRODUCTION_DATABASE_URL }}

      - name: Deploy EXACTLY the image verified in staging
        run: ./tools/deploy.sh production "sha-${{ github.sha }}"
        env:
          DEPLOY_TOKEN: ${{ secrets.PRODUCTION_DEPLOY_TOKEN }}

      - name: Wait for readiness
        run: |
          for i in $(seq 1 40); do
            curl -fsS https://api.aromastore.example/health/ready && exit 0
            sleep 5
          done
          exit 1

      - name: Smoke tests
        id: smoke
        run: ./tools/smoke.sh https://api.aromastore.example
        env:
          EXPECTED_COMMIT: ${{ github.sha }}

      - name: Watch the metrics for 3 minutes
        run: ./tools/watch-deployment.sh --minutes 3 --error-threshold 0.5

      - name: Automatic ROLLBACK if anything failed
        if: failure()
        run: |
          echo "::error::Deployment failed. Reverting to ${{ steps.previous.outputs.sha }}"
          ./tools/deploy.sh production "${{ steps.previous.outputs.sha }}"
          ./tools/smoke.sh https://api.aromastore.example
        env:
          DEPLOY_TOKEN: ${{ secrets.PRODUCTION_DEPLOY_TOKEN }}
          EXPECTED_COMMIT: ${{ steps.previous.outputs.sha }}

      - name: Notify the team
        if: always()
        run: ./tools/notify.sh "${{ job.status }}" "sha-${{ github.sha }}"

Why the same image and not a rebuild. Five reasons, from most to least obvious:

It is the only thing that has been verified. The end-to-end tests and Newman ran against that specific image in staging. Rebuilding produces an artefact nobody has tested, however identical the code looks.
Builds are not bit-for-bit reproducible. npm ci respects the lock for your direct and transitive dependencies, but the node:20-bookworm-slim base image is a moving tag that gets updated; the system packages installed with apt-get change version; and native modules are compiled against whatever is there at the time. Two builds of the same commit a day apart can differ.
Traceability. One tag, one artefact, one commit. If production and staging hold different images with the same name, debugging becomes an exercise in faith.
Time. Rebuilding adds minutes to the deployment's critical path, exactly when there may be an urgent fix waiting.
Attack surface. Every build is an opportunity for something improper to enter the supply chain. Fewer builds, fewer opportunities.

Details of the job worth noting: environment: production with required reviewers is what makes this continuous delivery and not continuous deployment; the previous step queries the deployed version before touching anything, because afterwards it can no longer be known; and the rollback step carries if: failure(), which fires if any previous step in the job fails, including the smoke tests and the metric watch.

An honest limitation of this solution: the rollback reverts the code, not the data. If the migration was destructive, this saves nothing, and that is why section 11 is not optional.

Solution 2

Phase 0 — Preparation (before touching anything).

Add the new fields to the contract as optional, without removing origin. Publish openapi.yaml with country and region documented and origin still in force. oasdiff approves: adding fields to a response is not breaking.

Phase 1 — EXPAND (deployment 1).

-- migrations/014-expand-origin.sql
ALTER TABLE coffees ADD COLUMN country TEXT;
ALTER TABLE coffees ADD COLUMN region TEXT;

-- Initial backfill: everything currently in origin is copied into country.
-- The region is left null: it will be completed manually or by a separate process.
UPDATE coffees SET country = origin WHERE country IS NULL;

-- The index for the region search is created here, not later.
CREATE INDEX IF NOT EXISTS idx_coffees_country_region ON coffees(country, region);

// Deployment 1's code: writes to ALL THREE columns, reads from `origin`.
export function saveCoffee(coffee) {
  db.prepare(`
    UPDATE coffees SET origin = ?, country = ?, region = ? WHERE id = ?
  `).run(
    coffee.region ? `${coffee.country}, ${coffee.region}` : coffee.country,  // composes the legacy value
    coffee.country,
    coffee.region ?? null,
    coffee.id,
  );
}

// The API still returns `origin` and now also returns `country` and `region` when they exist.
export function coffeeToRepresentation(row) {
  return {
    ...fields,
    origin: row.origin,           // still the source for consumers
    country: row.country,
    region: row.region ?? undefined,
  };
}

Contract state: origin in force, country and region available. No consumer notices a thing. It can be reverted with no trouble: the new columns are surplus but harmless.

Phase 1b — Backfilling the region (a separate process, days).

2,400 coffees whose region has to be inferred or entered by hand. A batch script that updates in runs of 200 with pauses, so as not to lock the database or spike the latency:

// tools/backfill-region.js — runnable several times, idempotent
const batch = db.prepare('SELECT id, origin FROM coffees WHERE region IS NULL LIMIT 200').all();
for (const row of batch) {
  const [country, region] = splitOrigin(row.origin);   // heuristics + a manual table
  db.prepare('UPDATE coffees SET country = ?, region = ? WHERE id = ?').run(country, region, row.id);
}

This step is what sets the real pace, and that is why the schema migration and the data backfill must be kept separate: mixing them produces migrations that take minutes and block the deployment.

Phase 2 — MIGRATE (deployment 2, a week or two later).

// The code reads from the NEW columns and keeps writing to all three.
export function coffeeToRepresentation(row) {
  return {
    ...fields,
    // `origin` is still returned, but it is now COMPOSED from country and region.
    // Old consumers see exactly the same as before.
    origin: row.region ? `${row.country}, ${row.region}` : row.country,
    country: row.country,
    region: row.region ?? undefined,
  };
}

The ?region=Yirgacheffe filter is switched on in this phase. It can still be reverted: origin is still populated and in sync, so deployment 1's code works perfectly.

Wait a week or two before continuing. The reason: give odd cases time to surface —coffees whose origin names the heuristic split badly— while the rollback is still available.

Phase 3 — Deprecate origin in the contract (it does not touch the database).

        origin:
          type: string
          deprecated: true
          description: |
            **Deprecated since 1.8.0. It will be retired on 30 June 2027.**
            Use `country` and `region`. This field is composed as
            `"{country}, {region}"` for compatibility.

With Deprecation and Sunset on the responses, and notice to the SPA, Aroma Mobile and CataBox. Because Aroma Mobile has old versions still alive, this window has to be generous: six months at the very least.

Phase 4 — CONTRACT (months later, with /v2 or after the Sunset).

-- migrations/018-contract-origin.sql
-- Only when NO version of the code that reads `origin` can still be alive
-- and the announced Sunset date has passed.
DROP INDEX IF EXISTS idx_coffees_origin;
ALTER TABLE coffees DROP COLUMN origin;

Where the possibility of reverting is lost: exactly in phase 4, when the DROP COLUMN runs. From that instant, deploying any version earlier than phase 2 produces SQL errors on every catalogue read, and recovering the column means restoring a backup and losing everything written since. That is why the contraction happens months later, when reverting to that version is no longer a realistic scenario, and preferably in a deployment of its own with no other change, so that if something fails you know exactly what it was.

With three instances in rolling, each individual phase is safe because at no point do an incompatible schema and code coexist: that is the property the pattern guarantees, and the only reason it exists.

Solution 3

A) VAT fix on invoices → blue-green (or rolling with a prepared rollback).

Aspect	Decision
Strategy	Blue-green, or rolling if there is no duplicate infrastructure
Why not canary	A canary means a percentage of customers receive invoices with the old calculation during the window. On a tax matter, inconsistency between invoices issued on the same day is worse than the original bug: it complicates the accounting and the later correction. Here you want a clean cut with an exact moment of change.
Why blue-green	It gives a precise switching moment —recordable in the accounting log— and a rollback in seconds.
What to watch	The error rate of `POST /orders/{id}/payment` and of invoice generation; the amounts on the first invoices issued, compared by hand against the expected calculation; `aroma_orders_created_total` to confirm purchases are still completing.
Rollback	Switch the load balancer back to blue. And a plan for the invoices issued during the window: identify them by timestamp and reissue them if appropriate. That is the genuinely hard part, and it has to be written down before deploying.
Extra	Deploy during the quietest period. A change with tax implications is not deployed at 18:00 on a Friday.

B) New default catalogue sorting → canary.

Aspect	Decision
Strategy	Canary, starting at 5 %
Why	It is a product change, not a technical one: there is no "correct" answer to verify, but a hypothesis to measure. The canary lets you compare real user behaviour between the two versions, which is exactly the question. It is also reversible with no consequences: nobody loses data if the sorting changes.
What to watch	Business metrics above technical ones: visit-to-order conversion rate, average order value, pagination depth (if people paginate less, the sorting is getting it right). On the technical side, the p99 latency of `GET /v1/coffees`, because sorting by average rating may require a `JOIN` or an aggregation that degrades the query; it is worth checking there is an index (04-06).
Rollback	Feature flag, not a deployment. `FLAG_SORT_BY_RATING` switched off returns the previous order in a second, with nothing deployed. It is the textbook case for a flag: a visible behaviour change, reversible and subject to measurement.
Extra	If the comparison is going to run for weeks, this stops being a canary and becomes an A/B test; you want a stable split per customer so the same user does not see the catalogue reshuffled on every visit.

C) Migrating from SQLite to PostgreSQL → blue-green, with a prior dual-write phase.

Aspect	Decision
Strategy	Blue-green, preceded by a phased data migration. It is not a deployment: it is a project.
Why not rolling	With rolling, instances writing to SQLite and to PostgreSQL would coexist. Writes would be split between two databases that diverge, and reconciling them afterwards is practically impossible.
Phases	1) Deploy the new code writing to both and reading from SQLite. 2) Bulk initial copy and verification that both databases match, record by record. 3) A brief read-only or maintenance window to copy the final delta. 4) Switch reads to PostgreSQL (blue-green). 5) Watch for days. 6) Stop writing to SQLite.
What to watch	The p99 latency of every endpoint (query performance changes completely with another engine and another planner); integrity constraint errors that SQLite tolerated and PostgreSQL does not; connection pool saturation, which did not exist in SQLite and here is a real limit; and data consistency by comparing counts and sums between both databases.
Rollback	Switch back to SQLite, provided you have kept writing to it. That is the reason for the dual-write phase, and its cost is justified: without it, the migration is a one-way trip.
Extra	Thanks to the repository pattern from 03-05, the code change is confined to `src/repositories/` and `src/config/database.js`; the rest of the project never notices. It is the best demonstration of that separation's value, and it is worth checking that the integration tests pass against both engines before starting.

Conclusion

The table from 05-04 is now machinery. The Aroma Store API is packaged into a multi-stage Dockerfile where the build tools stay in the first stage, the final image carries only production dependencies and runs as the node user, with tini and exec form so that SIGTERM really reaches Node and the graceful shutdown from 03-07 runs, and with a HEALTHCHECK on liveness —not readiness— so healthy containers are not restarted when Redis fails. The .dockerignore keeps your .env and your node_modules out, and docker-compose.yml brings up the API, Redis, the Prism mock and, behind a profile, PostgreSQL, with condition: service_healthy so nothing starts before its time.

.github/workflows/ci.yml runs the gates in the right order: static quality first because it fails in thirty seconds, then the contract with swagger-cli validate, Spectral and oasdiff breaking in parallel with the tests on Node 20 and 22 with Redis as a service and coverage, npm audit --audit-level=high, and only then the build and publication of an image tagged with the commit SHA and scanned with Trivy; then migrations, deployment to staging, active waiting on /health/ready, end-to-end tests and the Newman collection from 05-01. The gates are inflexible on purpose, with a single explicit and visible escape hatch: the breaking-change label on the pull request.

And above the tooling, three ideas that separate a team that deploys calmly from one that deploys in fear. Backwards-compatible migrations: during a zero-downtime deployment two versions coexist over a single database, so a RENAME COLUMN is a DROP in disguise and the expand → migrate → contract pattern is not bureaucracy, it is the only thing that preserves the possibility of reverting. The deployment strategies —recreate, rolling, blue-green and canary— with /health and /health/ready from 04-07 deciding when traffic comes in and a preStop that avoids the phantom 502s of every deployment. And the rollback: reverting code is trivial when images are immutable and configuration lives outside; reverting data does not exist, it is prevented; and feature flags separate deploying from enabling, with an expiry date written on the same day they are created. The project's new artefacts are Dockerfile, .dockerignore, docker-compose.yml, docker-compose.test.yml, .github/workflows/ci.yml, tools/deploy.sh, tools/smoke.sh and src/config/flags.js.

One last piece of the module remains, and it is the one that reorganises a good part of what we have done by hand. Many of module 4's mechanisms —rate limiting with Redis, JWT and OAuth token validation, TLS termination, CORS, caching, compression, version routing— exist already solved one layer above the application. In 05-06, API gateways and developer portals, we close the module with that perspective: what a gateway is and which functions it takes on, compared one by one against our implementations; the criterion for deciding what is delegated and what is never delegated —resource-level authorisation, business logic and semantic validation always stay in the application—; a real declarative Kong configuration for Aroma Store with the limits from 04-04 and the allowlist from 04-05, and which middleware from src/app.js we could then retire and which we could not; the risks of the gateway as a single point of failure; and developer portals, where the openapi.yaml from 05-02 becomes the front door for third parties like CataBox, with "time to first successful call" as the metric of your API's quality.

Continuous Integration and Deployment

Contents

Continuous integration, continuous delivery and continuous deployment

Why the main branch must always be deployable

Packaging with Docker: the multi-stage `Dockerfile`

`.dockerignore`, layers and image size

The complete environment with `docker-compose.yml`

Per-environment configuration and secrets outside the image

The CI pipeline: `.github/workflows/ci.yml`

Version matrix and dependency caching

What makes the pipeline fail and why the gates are inflexible

Building and publishing the image

Migrations: expand, migrate, contract

Deployment strategies

The role of liveness and readiness

Rollback and feature flags

Where to deploy

Artefact versioning and traceability

After the deployment: smoke tests and watching

Secrets in CI and least privilege

Common Mistakes and Tips

Exercises

Exercise 1: completing the pipeline with a production deployment job

Exercise 2: designing a safe migration

Exercise 3: choosing a deployment strategy for three changes

Solutions

Solution 1

Solution 2

Solution 3

Conclusion

REST API Course: Principles of Designing and Developing RESTful APIs

Module 1: Introduction to RESTful APIs

Module 2: Designing RESTful APIs

Module 3: Building RESTful APIs

Module 4: Best Practices and Security

Module 5: Tools and Frameworks

Module 6: Case Studies and Projects

Continuous Integration and Deployment

Contents

Continuous integration, continuous delivery and continuous deployment

Why the main branch must always be deployable

Packaging with Docker: the multi-stage Dockerfile

.dockerignore, layers and image size

The complete environment with docker-compose.yml

Per-environment configuration and secrets outside the image

The CI pipeline: .github/workflows/ci.yml

Version matrix and dependency caching

What makes the pipeline fail and why the gates are inflexible

Building and publishing the image

Migrations: expand, migrate, contract

Deployment strategies

The role of liveness and readiness

Rollback and feature flags

Where to deploy

Artefact versioning and traceability

After the deployment: smoke tests and watching

Secrets in CI and least privilege

Common Mistakes and Tips

Exercises

Exercise 1: completing the pipeline with a production deployment job

Exercise 2: designing a safe migration

Exercise 3: choosing a deployment strategy for three changes

Solutions

Solution 1

Solution 2

Solution 3

Conclusion

REST API Course: Principles of Designing and Developing RESTful APIs

Module 1: Introduction to RESTful APIs

Module 2: Designing RESTful APIs

Module 3: Building RESTful APIs

Module 4: Best Practices and Security

Module 5: Tools and Frameworks

Module 6: Case Studies and Projects

Packaging with Docker: the multi-stage `Dockerfile`

`.dockerignore`, layers and image size

The complete environment with `docker-compose.yml`

The CI pipeline: `.github/workflows/ci.yml`