BiblioTech is tested, measured and verified on every change. And it exists for nobody.

It runs on Diego Alonso's laptop when he starts it, and on a GitHub Actions runner for the eight minutes the pipeline lasts. Marta Ruiz cannot open a browser and look at the catalogue, because there is no server anywhere running the application. Nuria Vidal cannot reserve "Refactoring", because the process that would serve that request is not switched on anywhere.

This lesson covers the journey from "it works on my machine" to "it is in production". It is a journey with more traps than it seems, and almost all of them boil down to a phrase you will hear many times in your career: "well, it worked locally". It worked because locally there was a different Java version, a configuration file that is not in the repository, a database schema that Hibernate had created on its own, 32 GB of memory and no other user competing for it.

The goal of everything that follows is to eliminate those differences: package the application with everything it needs, configure it from the outside, and deploy it in a repeatable, observable and reversible way.

By the end you will know how to package and containerise a Java application properly, configure the JVM so that it respects a container's limits, version the database schema with Flyway, choose between the different deployment platforms with good judgement, expose health probes and shut the application down without cutting requests in half, apply deployment strategies with rollback, and build a complete continuous delivery pipeline.

Contents

  1. What deploying means
  2. Packaging: executable jar versus war
  3. The layered jar and why it speeds images up
  4. Reproducible builds and traceability
  5. Containers: image and container
  6. BiblioTech's Dockerfile, line by line
  7. .dockerignore
  8. Alternatives without a Dockerfile: Buildpacks and Jib
  9. Choosing a base image and its size
  10. The JVM inside a container
  11. docker-compose for the local environment
  12. Configuration and secrets at deployment time
  13. Database migrations with Flyway
  14. Two-phase deployment for schema changes
  15. Where it gets deployed: platform comparison
  16. Kubernetes: a minimal Deployment
  17. Startup and health: Actuator probes
  18. Graceful shutdown
  19. Startup time: CDS and Native Image
  20. Deployment strategies
  21. Rollback and the limits of the database
  22. The continuous delivery pipeline
  23. Horizontal scaling and what it demands of the application
  24. Common Mistakes and Tips
  25. Exercises
  26. Conclusion

  1. What deploying means

Deploying means making a specific version of the software available to its users, in an environment you do not fully control. The differences from your laptop are systematic:

Aspect Your machine Production
Java version Whichever you have installed Whichever the operator decides
Configuration application-dev.yml Environment variables
Database Ephemeral container, all yours Shared, with real data
Schema Hibernate creates it Versioned migrations
Memory 32 GB 512 MB with a hard limit
Failures You restart Somebody gets a phone call
Restarting No consequences Cut requests, affected users
Data Test data Unrecoverable if lost

From that come the three principles that govern this lesson:

  1. One artefact, every environment. The same jar and the same image go to development, staging and production. The only thing that changes is the configuration. If you build a different image for production, what you tested is not what you deploy.
  2. Configuration from the outside. Nothing environment-specific inside the artefact.
  3. Everything must be undoable. A deployment with no way back is a gamble.

  1. Packaging: executable jar versus war

Historically, a Java web application was packaged in a .war and deployed inside an application server installed separately. Spring Boot flipped the model with the executable jar, which carries the server inside.

Aspect Executable jar War
Server Embedded Installed separately
Running it java -jar app.jar Copy into webapps/
Deployable units One Two: server and application
Server version The one the POM declares The operator's
Containers Fits perfectly Awkward
Several apps per server No Yes
When to use it Practically always An imposed corporate application server
<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <configuration>
    <mainClass>com.nexussoftware.bibliotech.web.BiblioTechApplication</mainClass>
  </configuration>
</plugin>
./mvnw -pl bibliotech-web clean package
java -jar bibliotech-web/target/bibliotech-web-1.0.0.jar

One detail worth knowing: Spring Boot's executable jar is not an ordinary jar. Your classes live in BOOT-INF/classes/ and the dependencies, as complete jars, in BOOT-INF/lib/. A custom class loader (JarLauncher) takes care of reading them. That is why java -cp app.jar MyClass does not work the way you would expect.

bibliotech-web-1.0.0.jar
├── META-INF/MANIFEST.MF          ← Main-Class: org.springframework.boot.loader.launch.JarLauncher
├── org/springframework/boot/loader/   ← the loader
└── BOOT-INF/
    ├── classes/                  ← your code and your resources
    ├── lib/                      ← ~50 dependency jars
    └── classpath.idx

  1. The layered jar and why it speeds images up

Here is an optimisation that looks like a detail and radically changes deployment times.

A Docker image is made of stacked layers. When an image is pushed or pulled, only the layers that changed travel. If the whole jar (60 MB) is in a single layer, changing one line of code forces a 60 MB transfer.

But the make-up of those 60 MB is very uneven:

Content Typical size Change frequency
Dependencies (Spring, Hibernate, Jackson…) ~55 MB Every few weeks
Spring Boot loader ~200 KB With the Spring Boot version
Internal dependencies (our own modules) ~500 KB Daily
Your code ~1 MB Every commit

Spring Boot offers layertools, which splits the jar into those four parts:

$ java -Djarmode=tools -jar bibliotech-web.jar list-layers
dependencies
spring-boot-loader
snapshot-dependencies
application
# Extract each layer into its own directory
java -Djarmode=tools -jar bibliotech-web.jar extract --layers --launcher --destination extracted/

And in the Dockerfile, each layer is copied by a separate COPY, in order of stability. The measured result:

Scenario Without layers With layers
One line of code changed 60 MB transferred ~1 MB
Adding a dependency 60 MB ~57 MB
Typical push time 40 s 3 s
Pull time at deployment 30 s 2 s

With twenty deployments a day, that is hours a month. And the psychological effect matters as much as the technical one: a three-second deployment gets done without a second thought; a two-minute one gets stockpiled "so we do them all together on Thursday", which is exactly the practice to avoid.

It is enabled in the POM (it is on by default in Spring Boot 3):

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <configuration>
    <layers>
      <enabled>true</enabled>
    </layers>
  </configuration>
</plugin>

  1. Reproducible builds and traceability

A reproducible build means that the same source code produces exactly the same bytes. It sounds academic and it has a very concrete practical consequence: being able to verify that the binary in production corresponds to the code it claims to correspond to.

What breaks reproducibility is the timestamps inside the jar:

<properties>
  <!-- Fixes the date of the jar entries: without this, every build differs -->
  <project.build.outputTimestamp>2026-08-05T00:00:00Z</project.build.outputTimestamp>
</properties>
./mvnw clean package
sha256sum target/bibliotech-web-1.0.0.jar
# a3f7... (the same hash on any machine, today and a year from now)

Traceability. Every artefact must be able to answer: which commit did it come from? when was it built? who built it?

<plugin>
  <groupId>io.github.git-commit-id</groupId>
  <artifactId>git-commit-id-maven-plugin</artifactId>
  <version>9.0.1</version>
  <executions>
    <execution><goals><goal>revision</goal></goals></execution>
  </executions>
  <configuration>
    <generateGitPropertiesFile>true</generateGitPropertiesFile>
    <includeOnlyProperties>
      <property>^git.branch$</property>
      <property>^git.commit.id.abbrev$</property>
      <property>^git.commit.time$</property>
      <property>^git.build.version$</property>
    </includeOnlyProperties>
  </configuration>
</plugin>

With that, Actuator exposes the information:

$ curl https://bibliotech.nexussoftware.com/actuator/info
{
  "git": {
    "branch": "main",
    "commit": { "id": "a3f7e91", "time": "2026-08-05T09:14:22Z" }
  },
  "build": { "version": "1.4.2", "artifact": "bibliotech-web", "time": "2026-08-05T09:20:11Z" }
}

The question "which version is in production right now?" has an exact answer, in one second. Without that, the answer is "I think last week's", and from there on no diagnosis is trustworthy.

Semantic versioning of the artefacts:

Format Example Use
MAJOR.MINOR.PATCH 1.4.2 Released version
MAJOR.MINOR.PATCH-SNAPSHOT 1.5.0-SNAPSHOT In development
MAJOR.MINOR.PATCH-rc.N 1.5.0-rc.1 Release candidate
With the commit 1.4.2-a3f7e91 Exact traceability

  1. Containers: image and container

Two concepts that are constantly confused:

  • An image is an immutable read-only template: a layered file system + metadata (which command to run, which ports, which user). It is like a class.
  • A container is a running instance of an image, with a writable layer on top. It is like an object.
flowchart TD
    B["Base image<br/>eclipse-temurin:21-jre-alpine"]
    L1["Layer: dependencies (~55 MB)"]
    L2["Layer: loader (~200 KB)"]
    L3["Layer: BiblioTech code (~1 MB)"]
    I["IMAGE bibliotech:1.4.2"]
    C1["Container 1<br/>running"]
    C2["Container 2<br/>running"]

    B --> L1 --> L2 --> L3 --> I
    I --> C1
    I --> C2

A container is not a virtual machine: it shares the host's kernel and uses Linux mechanisms (namespaces for isolation, cgroups for resource limits). That is why it starts in milliseconds and weighs megabytes instead of gigabytes.

Aspect Virtual machine Container
Isolation Total (its own kernel) Process-level (shared kernel)
Startup Minutes Milliseconds
Size GB MB
Overhead Noticeable Minimal

What a container really solves, and what justifies all the rest: the artefact includes the base operating system, the JVM, the application and its dependencies. The phrase "it works on my machine" stops meaning anything, because the machine travels with the application.

  1. BiblioTech's Dockerfile, line by line

# ==============================================================================
# STAGE 1: BUILD
# This stage has Maven, the full JDK and the source code. None of that
# reaches the final image: it is used only to produce the jar.
# ==============================================================================
FROM eclipse-temurin:21-jdk-alpine AS builder

WORKDIR /build

# Copy ONLY the dependency files first.
# Docker caches every instruction: as long as the POMs do not change, the
# dependency download (the slowest step, 2-3 minutes) is skipped entirely.
COPY .mvn/ .mvn/
COPY mvnw pom.xml ./
COPY bibliotech-domain/pom.xml          bibliotech-domain/
COPY bibliotech-application/pom.xml     bibliotech-application/
COPY bibliotech-infrastructure/pom.xml  bibliotech-infrastructure/
COPY bibliotech-web/pom.xml             bibliotech-web/
COPY bibliotech-console/pom.xml         bibliotech-console/

# go-offline downloads every dependency without compiling anything.
# --mount=type=cache keeps ~/.m2 between builds (BuildKit).
RUN --mount=type=cache,target=/root/.m2 \
    ./mvnw -B dependency:go-offline -DskipTests

# NOW we copy the source code. If only the code changes,
# Docker reuses the previous layer and downloads nothing again.
COPY bibliotech-domain/src          bibliotech-domain/src
COPY bibliotech-application/src     bibliotech-application/src
COPY bibliotech-infrastructure/src  bibliotech-infrastructure/src
COPY bibliotech-web/src             bibliotech-web/src

# -DskipTests: the tests already ran in CI (12-05). Repeating them here
# doubles the time and would need Docker inside Docker for Testcontainers.
RUN --mount=type=cache,target=/root/.m2 \
    ./mvnw -B -pl bibliotech-web -am clean package -DskipTests

# Extract the jar into layers
RUN java -Djarmode=tools -jar bibliotech-web/target/bibliotech-web-*.jar \
         extract --layers --launcher --destination extracted

# ==============================================================================
# STAGE 2: RUNTIME
# Minimal image: JRE only (not JDK), no Maven, no source code,
# no build tools. Less surface, fewer vulnerabilities.
# ==============================================================================
FROM eclipse-temurin:21-jre-alpine AS runtime

# Standard OCI labels: metadata that ecosystem tools read
LABEL org.opencontainers.image.title="BiblioTech" \
      org.opencontainers.image.description="Nexus Software technical library" \
      org.opencontainers.image.vendor="Nexus Software" \
      org.opencontainers.image.licenses="Proprietary"

# Minimal utilities:
#  - curl for the HEALTHCHECK
#  - tzdata so time zones work (Alpine does not ship them)
#  - dumb-init as PID 1: forwards signals properly to the Java process
RUN apk add --no-cache curl tzdata dumb-init && \
    rm -rf /var/cache/apk/*

ENV TZ=Europe/Madrid

# ---- NON-root user ----
# If an attacker gets code execution, they must not have root inside the
# container. It is the cheapest and most effective mitigation there is.
RUN addgroup -S -g 1001 bibliotech && \
    adduser -S -u 1001 -G bibliotech -h /app bibliotech

WORKDIR /app

# ---- The layers, in order of stability (least changing first) ----
# Every COPY is a Docker layer. When only the code changes, only
# the last layer (~1 MB) is rebuilt and transferred.
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/dependencies/ ./
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/spring-boot-loader/ ./
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/snapshot-dependencies/ ./
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/application/ ./

USER bibliotech

EXPOSE 8080

# ---- JVM options ----
#  MaxRAMPercentage=75      uses 75% of the container limit for the heap
#  InitialRAMPercentage=50  starts at half: fewer resizes
#  UseG1GC                  balanced GC; for <2 vCPU consider SerialGC
#  ExitOnOutOfMemoryError   on OOM, die: let the orchestrator restart
#                           (a JVM in OOM serves half-broken requests, which is worse)
#  HeapDumpOnOutOfMemoryError  dump for diagnosis (10-07)
#  file.encoding=UTF-8      explicit: we do not depend on the environment
ENV JAVA_OPTS="\
    -XX:MaxRAMPercentage=75.0 \
    -XX:InitialRAMPercentage=50.0 \
    -XX:+UseG1GC \
    -XX:+ExitOnOutOfMemoryError \
    -XX:+HeapDumpOnOutOfMemoryError \
    -XX:HeapDumpPath=/tmp/heapdump.hprof \
    -Djava.security.egd=file:/dev/./urandom \
    -Dfile.encoding=UTF-8"

# Container-level health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=45s --retries=3 \
  CMD curl -fsS http://localhost:8080/actuator/health/readiness || exit 1

# dumb-init as PID 1 forwards SIGTERM to the Java process: without this,
# graceful shutdown (section 18) does not work.
ENTRYPOINT ["dumb-init", "--"]

# "shell" form so that $JAVA_OPTS is expanded. exec makes Java the
# direct child process so that it receives the signals.
CMD exec java $JAVA_OPTS org.springframework.boot.loader.launch.JarLauncher

Building and running:

docker build -t bibliotech:1.4.2 -t bibliotech:latest .

docker run -d --name bibliotech \
  -p 8080:8080 \
  --memory=768m --cpus=1.5 \
  -e SPRING_PROFILES_ACTIVE=prod \
  -e BIBLIOTECH_DB_URL=jdbc:postgresql://db:5432/bibliotech \
  -e BIBLIOTECH_DB_USER=bibliotech \
  -e BIBLIOTECH_DB_PASSWORD="$DB_PASSWORD" \
  bibliotech:1.4.2

docker logs -f bibliotech
docker exec bibliotech curl -s localhost:8080/actuator/health

The five points of the Dockerfile that matter most, in case you only remember five:

  1. Multi-stage: the final image contains neither Maven nor the JDK nor the source code. It goes from ~700 MB to ~180 MB, and removes a complete compiler from the attack surface.
  2. POMs before the code: the dependency cache survives between builds.
  3. Non-root user: a basic and mandatory mitigation.
  4. Layers ordered by stability: 1 MB deployments instead of 60 MB.
  5. dumb-init + exec: without them, SIGTERM never reaches the JVM and graceful shutdown does not happen.

  1. .dockerignore

Without it, the build context includes the whole .git directory, the target/ folders and possibly files with secrets, which end up inside the image or at the very least get sent to the Docker daemon.

# Everything that is not needed to build
.git/
.github/
.idea/
.vscode/
*.iml

target/
**/target/

*.md
docs/
LICENSE

# CRITICAL: none of this must enter the build context
.env
*.env
**/application-local.yml
*.pem
*.p12
*.jks
secrets/

Dockerfile
.dockerignore
compose.yaml

Checking the context size:

docker build --no-cache --progress=plain . 2>&1 | head -5
# => transferring context: 1.24MB    (good)
# Without .dockerignore it would typically be 250 MB.

  1. Alternatives without a Dockerfile: Buildpacks and Jib

Cloud Native Buildpacks — built into Spring Boot, without writing a single Dockerfile:

./mvnw -pl bibliotech-web spring-boot:build-image \
  -Dspring-boot.build-image.imageName=bibliotech:1.4.2
<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <configuration>
    <image>
      <name>registry.nexussoftware.com/bibliotech:${project.version}</name>
      <env>
        <BP_JVM_VERSION>21</BP_JVM_VERSION>
        <BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
        <BPE_APPEND_JAVA_TOOL_OPTIONS>-XX:MaxRAMPercentage=75</BPE_APPEND_JAVA_TOOL_OPTIONS>
      </env>
    </image>
  </configuration>
</plugin>

Buildpacks detects that it is a Java application, picks the JVM, applies layers, configures the memory automatically from the container limit and adds an SBOM (a component inventory, useful for security, 12-07).

Jib (Google) builds the image without needing a Docker daemon, which makes it ideal for CI:

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.4.3</version>
  <configuration>
    <from><image>eclipse-temurin:21-jre-alpine</image></from>
    <to><image>registry.nexussoftware.com/bibliotech:${project.version}</image></to>
    <container>
      <user>1001:1001</user>
      <ports><port>8080</port></ports>
      <jvmFlags>
        <jvmFlag>-XX:MaxRAMPercentage=75.0</jvmFlag>
      </jvmFlags>
    </container>
  </configuration>
</plugin>
./mvnw -pl bibliotech-web jib:build         # pushes straight to the registry
./mvnw -pl bibliotech-web jib:dockerBuild   # or builds into the local Docker

Comparison:

Criterion Dockerfile Buildpacks Jib
Full control Yes No Partial
Needs Docker to build Yes Yes No
Optimised layers Manual Automatic Automatic
Updating the base Manual Automatic (rebase) Change one line
Speed Medium Slow the 1st time Very fast
Learning curve Medium Low Low
When You need fine control You want to forget about it CI without Docker

Recommendation for BiblioTech: an explicit Dockerfile. On a course, and in a team that wants to understand its own deployment, control and transparency are worth more than convenience. In a large team with many services, Buildpacks or Jib save repeated work.

  1. Choosing a base image and its size

Base image Size (with JRE 21) Characteristics
eclipse-temurin:21-jdk ~450 MB Full JDK. Do not use in production
eclipse-temurin:21-jre ~270 MB JRE on Ubuntu. Compatible and predictable
eclipse-temurin:21-jre-alpine ~180 MB Alpine + musl libc. Lightweight
gcr.io/distroless/java21 ~190 MB No shell, no package manager. Very secure
Custom image with jlink ~90 MB JRE trimmed to the required modules

Real-world considerations:

  • Alpine uses musl instead of glibc. 99% of Java code behaves identically, but libraries with native code can fail. If odd library-loading errors appear, try the non-Alpine variant before losing an afternoon.
  • Distroless is the most secure: with no shell, an attacker who gets code execution cannot run commands. The price is that you cannot either: there is no docker exec ... sh for diagnosis. It requires good observability (12-07).
  • Size matters less than it seems. The base layer is downloaded once and shared by every image on that base. What travels on each deployment is the application layer (~1 MB).

Trimming with jlink, for anyone who needs the bare minimum:

FROM eclipse-temurin:21-jdk-alpine AS jre-minimal
RUN jlink \
    --add-modules java.base,java.logging,java.sql,java.naming,java.management,\
java.instrument,java.security.jgss,java.desktop,jdk.unsupported,jdk.crypto.ec \
    --strip-debug --no-man-pages --no-header-files --compress=2 \
    --output /jre-minimal

  1. The JVM inside a container

This is by far the most common deployment mistake with Java, and it deserves a section of its own.

The historical problem. Before Java 10, the JVM read the memory of the host machine, ignoring the container's limit. On a 64 GB server with a container limited to 512 MB, the JVM computed a maximum heap of 16 GB (a quarter of 64), tried to use it, and the kernel killed the process with OOMKilled (code 137) without a single message from the JVM.

Since Java 10 — and refined in 11, 15 and 17 — the JVM is cgroup-aware:

$ docker run --memory=512m eclipse-temurin:21-jre java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
   size_t MaxHeapSize = 134217728    # 128 MB = 25% of 512 MB

The options to tune and why:

Option Recommended value Reason
-XX:MaxRAMPercentage 75.0 The default 25% wastes memory
-XX:InitialRAMPercentage 50.0 Fewer heap resizes at startup
-XX:+UseG1GC With ≥ 2 vCPU A balance between pauses and throughput
-XX:+UseSerialGC With < 2 vCPU Less overhead in small containers
-XX:MaxMetaspaceSize 256m Metaspace is not inside the heap
-XX:+ExitOnOutOfMemoryError Always Die fast and let the orchestrator restart
-XX:ActiveProcessorCount If the CPU limit is fractional The JVM rounds fractional CPUs badly

A calculation that prevents many incidents. With a container limit of 512 MB:

Component Memory
Heap (75%) 384 MB
Metaspace ~60 MB
Thread stacks (200 × 1 MB reserved) ~30 MB actual
Code cache (JIT) ~40 MB
GC and internal structures ~30 MB
NIO direct buffers ~20 MB
Total ~564 MB > 512 MB → OOMKilled

The JVM's memory is not only the heap. This calculation is the reason so many Java containers die with no apparent explanation. To diagnose it, NativeMemoryTracking (10-07):

docker run -e JAVA_OPTS="-XX:NativeMemoryTracking=summary" bibliotech:1.4.2
docker exec bibliotech jcmd 1 VM.native_memory summary

Rules of thumb:

  • With MaxRAMPercentage=75, leave at least 256 MB of container limit above what the heap needs.
  • A typical Spring Boot service needs at least 512 MB; 768 MB is more comfortable.
  • If you see OOMKilled (code 137), it is not an application bug: it is memory outside the heap.

  1. docker-compose for the local environment

In 12-01 we only started PostgreSQL. Now, the complete application:

# compose.yaml
name: bibliotech

services:

  db:
    image: postgres:16-alpine
    container_name: bibliotech-db
    environment:
      POSTGRES_DB: bibliotech
      POSTGRES_USER: bibliotech
      POSTGRES_PASSWORD: ${DB_PASSWORD:-bibliotech}
    ports:
      - "5432:5432"
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U bibliotech -d bibliotech"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 10s

  app:
    build:
      context: .
      dockerfile: Dockerfile
    image: bibliotech:${VERSION:-dev}
    container_name: bibliotech-app
    depends_on:
      db:
        condition: service_healthy      # waits for the healthcheck, not just the start
    environment:
      SPRING_PROFILES_ACTIVE: docker
      BIBLIOTECH_DB_URL: jdbc:postgresql://db:5432/bibliotech
      BIBLIOTECH_DB_USER: bibliotech
      BIBLIOTECH_DB_PASSWORD: ${DB_PASSWORD:-bibliotech}
      JAVA_OPTS: "-XX:MaxRAMPercentage=75 -XX:+UseG1GC"
    ports:
      - "8080:8080"
    deploy:
      resources:
        limits:
          memory: 768M
          cpus: '1.5'
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/actuator/health/readiness"]
      interval: 15s
      timeout: 3s
      retries: 5
      start_period: 60s
    restart: unless-stopped

volumes:
  db-data:
docker compose up -d --build       # build and start
docker compose logs -f app         # follow the logs
docker compose ps                  # status and health
docker compose exec db psql -U bibliotech    # get into the database
docker compose down                # stop (the data survives)
docker compose down -v             # stop and DELETE the data

depends_on with condition: service_healthy is the detail that prevents the most frequent failure in development: the application starts before PostgreSQL accepts connections and dies on the first attempt.

  1. Configuration and secrets at deployment time

The rule that picks up 12-01 and governs everything:

One image for every environment. The same bibliotech:1.4.2 image goes to development, staging and production. Only the environment variables change.

If you build a different image for production, what you tested in staging is not what you deploy.

Mechanisms, in order of robustness:

Mechanism How Advantages Risks
Environment variables -e KEY=value De facto standard, simple Visible in docker inspect and in the process environment
Mounted files -v /secrets:/app/config:ro Do not appear in the environment Permissions have to be managed
Orchestrator secrets Kubernetes Secret, Docker Secret Integrated into the platform Base64 is not encryption
Secret manager Vault, AWS Secrets Manager Rotation, auditing, encryption More operational complexity

External configuration by file, useful on your own servers:

java -jar bibliotech-web.jar \
     --spring.config.additional-location=file:/etc/bibliotech/

Spring Boot looks for application.yml and application-{profile}.yml in that path, with higher precedence than the packaged ones (12-01).

In Kubernetes, the separation between configuration and secrets:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bibliotech-config
data:
  SPRING_PROFILES_ACTIVE: "prod"
  BIBLIOTECH_LOAN_DEFAULTDAYS: "15"
  LOGGING_LEVEL_COM_NEXUSSOFTWARE_BIBLIOTECH: "INFO"
---
apiVersion: v1
kind: Secret
metadata:
  name: bibliotech-secrets
type: Opaque
stringData:
  BIBLIOTECH_DB_PASSWORD: "…"        # managed with Sealed Secrets or an external manager
  METADATA_API_KEY: "…"

Warning. A Kubernetes Secret is encoded in base64, which is not encryption: anybody with read access to the namespace can decode it. Real secrets need Sealed Secrets, the External Secrets Operator or an external manager, plus encryption at rest in etcd. Picked up again in 12-07.

Checking that the configuration is what you expect:

curl -s localhost:8080/actuator/env/bibliotech.fine.euros-per-day | jq
# shows the effective value AND which source it came from

  1. Database migrations with Flyway

Here a debt from 11-03 gets paid: ddl-auto is no good in production.

Value What it does Production
create-drop Drops and recreates at startup Catastrophic
create Drops and recreates Catastrophic
update Adds what is missing No. See below
validate Checks that the schema matches Yes
none Does nothing Yes

Why update is no good, precisely:

  1. It removes nothing. Columns and tables deleted from the model stay forever.
  2. It does not rename. Changing due_dt to due_date creates a new column and leaves the data in the old one.
  3. It does not version. There is no way to know what schema an environment has, or to reproduce it.
  4. It is not reversible. There is no way back.
  5. It depends on startup order. With several instances starting at once, they can issue DDL simultaneously and block each other.
  6. It does not migrate data. Splitting full_name into first_name and surname is impossible.

Flyway solves all of that with versioned SQL scripts applied in order, exactly once, recorded in a control table.

<dependency>
  <groupId>org.flywaydb</groupId>
  <artifactId>flyway-core</artifactId>
</dependency>
<dependency>
  <groupId>org.flywaydb</groupId>
  <artifactId>flyway-database-postgresql</artifactId>
</dependency>
spring:
  jpa:
    hibernate:
      ddl-auto: validate        # Hibernate VERIFIES, Flyway DECIDES
  flyway:
    enabled: true
    locations: classpath:db/migration
    baseline-on-migrate: true   # for a database that already existed
    validate-on-migrate: true   # detects already-applied scripts that have changed
    out-of-order: false         # forbids applying a version older than the last one

Naming convention: V<version>__<description>.sql

bibliotech-infrastructure/src/main/resources/db/migration/
├── V1__initial_schema.sql
├── V2__catalog_indexes.sql
├── V3__add_material_value_column.sql
├── V4__reservations_table.sql
├── V5__return_condition_and_surcharges.sql
└── R__usage_statistics_view.sql          ← R = repeatable: re-run when it changes
-- V1__initial_schema.sql
CREATE TABLE materials (
    id                    BIGSERIAL PRIMARY KEY,
    type                  VARCHAR(20)  NOT NULL,   -- SINGLE_TABLE discriminator (ADR-004)
    isbn                  VARCHAR(17)  NOT NULL UNIQUE,
    title                 VARCHAR(200) NOT NULL,
    author                VARCHAR(150),
    publication_year      INTEGER,
    total_copies          INTEGER      NOT NULL DEFAULT 1 CHECK (total_copies >= 0),
    available_copies      INTEGER      NOT NULL DEFAULT 1 CHECK (available_copies >= 0),
    version               BIGINT       NOT NULL DEFAULT 0,   -- @Version (11-03)
    created_at            TIMESTAMPTZ  NOT NULL DEFAULT now(),
    CONSTRAINT chk_available_not_above_total
        CHECK (available_copies <= total_copies)
);

CREATE TABLE employees (
    id            BIGSERIAL PRIMARY KEY,
    name          VARCHAR(150) NOT NULL,
    email         VARCHAR(200) NOT NULL UNIQUE,
    department    VARCHAR(100),
    start_date    DATE         NOT NULL,
    version       BIGINT       NOT NULL DEFAULT 0
);

CREATE TABLE loans (
    id                 BIGSERIAL PRIMARY KEY,
    material_id        BIGINT      NOT NULL REFERENCES materials(id),
    employee_id        BIGINT      NOT NULL REFERENCES employees(id),
    loan_date          DATE        NOT NULL,
    due_date           DATE        NOT NULL,
    return_date        DATE,
    status             VARCHAR(20) NOT NULL,
    version            BIGINT      NOT NULL DEFAULT 0,
    CONSTRAINT chk_due_after_loan CHECK (due_date >= loan_date),
    CONSTRAINT chk_return_after_loan
        CHECK (return_date IS NULL OR return_date >= loan_date)
);

-- Indexes for the queries that are actually made
CREATE INDEX idx_loans_employee_status ON loans(employee_id, status);
CREATE INDEX idx_loans_due_date        ON loans(due_date)
                                       WHERE return_date IS NULL;        -- partial
CREATE INDEX idx_materials_title       ON materials USING gin(to_tsvector('english', title));

-- Reference data the application needs in order to start
INSERT INTO employees (name, email, department, start_date) VALUES
    ('Marta Ruiz',  '[email protected]',  'Architecture', '2024-03-01'),
    ('Diego Alonso','[email protected]','Backend',      '2025-01-15'),
    ('Nuria Vidal', '[email protected]', 'Platform',     '2023-09-10');

Golden rules for migrations:

Rule Reason
An applied script is NEVER modified Flyway stores a checksum; if it changes, startup fails
To fix something, a new script It is the only way for every environment to converge
Idempotent migrations where possible CREATE TABLE IF NOT EXISTS
Destructive changes, in two phases See the next section
Test the migration with real data An ALTER TABLE over 10 million rows can take hours and block
A blocking ALTER, in a maintenance window In PostgreSQL, ADD COLUMN with a default is fast; ALTER TYPE rewrites the table

Useful commands:

./mvnw flyway:info       # which migrations exist and which are applied
./mvnw flyway:validate   # checks the checksums
./mvnw flyway:migrate    # applies the pending ones
./mvnw flyway:repair     # fixes the control table (carefully!)

And a test that prevents surprises, wired into Testcontainers (12-05):

@Test
void allMigrationsApplyOnACleanPostgres() {
    Flyway flyway = Flyway.configure()
            .dataSource(POSTGRES.getJdbcUrl(), POSTGRES.getUsername(), POSTGRES.getPassword())
            .locations("classpath:db/migration")
            .load();

    MigrateResult result = flyway.migrate();

    assertThat(result.success).isTrue();
    assertThat(result.migrationsExecuted).isGreaterThan(0);
    // And that the resulting schema matches what Hibernate expects:
    assertThatNoException().isThrownBy(() -> validateSchemaAgainstEntities());
}

  1. Two-phase deployment for schema changes

The problem: during a rolling update the old and the new version of the application coexist against the same database. A destructive change breaks the old version before it has finished retiring.

Example: renaming loans.due_dt to loans.due_date.

What you CANNOT do:

-- V6__rename_column.sql
ALTER TABLE loans RENAME COLUMN due_dt TO due_date;

The instant it is applied, every instance of the old version — still serving requests — fails with "column does not exist".

The solution, in three deployments:

flowchart TD
    F1["PHASE 1 · Expand<br/>Add due_date<br/>Copy data + sync trigger<br/>App v1 uses the old one; both columns coexist"]
    F2["PHASE 2 · Migrate<br/>App v2 writes and reads the new one<br/>The trigger keeps the old one up to date<br/>Rollback to v1 still possible"]
    F3["PHASE 3 · Contract<br/>Drop the trigger and the old column<br/>Only once v1 no longer exists"]
    F1 --> F2 --> F3
-- PHASE 1: V6__add_due_date.sql   (compatible with app v1)
ALTER TABLE loans ADD COLUMN due_date DATE;

UPDATE loans SET due_date = due_dt WHERE due_date IS NULL;

-- Dual-write trigger: it does not matter which app version writes
CREATE OR REPLACE FUNCTION sync_due_date() RETURNS TRIGGER AS $$
BEGIN
    IF NEW.due_date IS DISTINCT FROM OLD.due_date THEN
        NEW.due_dt := NEW.due_date;
    ELSIF NEW.due_dt IS DISTINCT FROM OLD.due_dt THEN
        NEW.due_date := NEW.due_dt;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_sync_due_date
    BEFORE INSERT OR UPDATE ON loans
    FOR EACH ROW EXECUTE FUNCTION sync_due_date();
-- PHASE 3: V8__drop_due_dt.sql   (only after confirming v1 is no longer running)
DROP TRIGGER IF EXISTS trg_sync_due_date ON loans;
DROP FUNCTION IF EXISTS sync_due_date();
ALTER TABLE loans DROP COLUMN due_dt;
ALTER TABLE loans ALTER COLUMN due_date SET NOT NULL;

This pattern is called expand and contract, and the rule that sums it up is easy to remember:

Every migration must be compatible with the previous version of the application. Destructive changes are applied at least one deployment after the one that stopped needing what is being removed.

  1. Where it gets deployed: platform comparison

Platform How it works Control Complexity Cost When
Own server + systemd jar as a Linux service Total Low Low 1-2 services, small team
PaaS (Heroku, Render, Railway, Fly.io) You push code or an image Low Very low Medium-high Prototypes, teams with no ops
Managed containers (ECS, Cloud Run, App Service) You deploy images Medium Medium Medium Most applications
Kubernetes A full orchestrator Total High Variable Many services, real scale

Your own server with systemd, which is still perfectly valid and often the best option:

# /etc/systemd/system/bibliotech.service
[Unit]
Description=BiblioTech - Nexus Software technical library
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=bibliotech
Group=bibliotech
WorkingDirectory=/opt/bibliotech

EnvironmentFile=/etc/bibliotech/environment   # secrets, with 600 permissions
ExecStart=/usr/bin/java $JAVA_OPTS -jar /opt/bibliotech/bibliotech-web.jar

SuccessExitStatus=143                        # 128 + SIGTERM: graceful shutdown, not a failure
Restart=on-failure
RestartSec=10

# Graceful shutdown: SIGTERM, and 60 s before SIGKILL
KillSignal=SIGTERM
TimeoutStopSec=60

# Hardening: least privilege (12-07)
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/bibliotech /var/lib/bibliotech

StandardOutput=journal
StandardError=journal
SyslogIdentifier=bibliotech

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now bibliotech
sudo systemctl status bibliotech
sudo journalctl -u bibliotech -f

Selection criteria, bluntly:

If… Choose
You have 1-3 services and a small team Your own server or managed containers
You do not want to manage infrastructure PaaS or Cloud Run
You need to scale to zero when there is no traffic Cloud Run, Fly.io
You have 20+ services and a platform team Kubernetes
You have 3 services and no platform team Not Kubernetes

On that last point it is worth being explicit: Kubernetes is an excellent tool with a real and permanent operational cost. Adopting it for three services because "it is what everybody uses" is a decision you pay for every week in the team's time.

  1. Kubernetes: a minimal Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bibliotech
  labels:
    app: bibliotech
spec:
  replicas: 3                       # three instances: high availability
  revisionHistoryLimit: 5           # history for rollback

  strategy:
    type: RollingUpdate             # rolling update (section 20)
    rollingUpdate:
      maxSurge: 1                   # at most 1 extra pod during the transition
      maxUnavailable: 0             # NEVER fewer than 3 available: no service interruption

  selector:
    matchLabels:
      app: bibliotech

  template:
    metadata:
      labels:
        app: bibliotech
        version: "1.4.2"
    spec:
      # Room for the graceful shutdown to finish (section 18)
      terminationGracePeriodSeconds: 60

      securityContext:              # least privilege at pod level
        runAsNonRoot: true
        runAsUser: 1001
        fsGroup: 1001

      containers:
        - name: bibliotech
          image: registry.nexussoftware.com/bibliotech:1.4.2   # EXACT tag, never 'latest'
          imagePullPolicy: IfNotPresent

          ports:
            - name: http
              containerPort: 8080

          envFrom:
            - configMapRef: { name: bibliotech-config }
            - secretRef:    { name: bibliotech-secrets }

          resources:
            requests:                # what the scheduler reserves
              memory: "512Mi"
              cpu: "250m"
            limits:                  # the ceiling; exceeding it on memory = OOMKilled
              memory: "768Mi"
              cpu: "1500m"

          # --- Probes (section 17) ---
          startupProbe:              # protects startup: up to 100 s
            httpGet: { path: /actuator/health/liveness, port: http }
            failureThreshold: 20
            periodSeconds: 5

          livenessProbe:             # still alive? If not, RESTART
            httpGet: { path: /actuator/health/liveness, port: http }
            periodSeconds: 10
            failureThreshold: 3

          readinessProbe:            # can it serve? If not, TAKE IT OUT OF THE BALANCER
            httpGet: { path: /actuator/health/readiness, port: http }
            periodSeconds: 5
            failureThreshold: 2

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true       # nothing writes to the root
            capabilities: { drop: ["ALL"] }

          volumeMounts:
            - name: tmp
              mountPath: /tmp                  # required: Tomcat and the dumps write here

      volumes:
        - name: tmp
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: bibliotech
spec:
  selector:
    app: bibliotech
  ports:
    - port: 80
      targetPort: http
  type: ClusterIP
---
apiVersion: policy/v1
kind: PodDisruptionBudget           # protects during cluster maintenance
metadata:
  name: bibliotech
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: bibliotech
kubectl apply -f k8s/
kubectl rollout status deployment/bibliotech
kubectl get pods -l app=bibliotech
kubectl logs -f -l app=bibliotech --tail=100
kubectl rollout undo deployment/bibliotech         # immediate rollback

Two details that deserve special attention:

  • maxUnavailable: 0 guarantees that during the update there are never fewer instances than declared. That is what turns a deployment into something invisible to users.
  • image: bibliotech:1.4.2, never :latest. With latest you do not know what is running, rollback does not work and two pods can end up on different versions.

  1. Startup and health: Actuator probes

Spring Boot Actuator distinguishes two questions that look the same and are not:

Probe Question If it fails
Liveness Is the process alive and not stuck? Restart the container
Readiness Can it serve requests right now? Take it out of the balancer, without restarting

The difference is critical. If the database goes down, the application is not ready (readiness fails) but it is alive (liveness passes). Restarting it would fix nothing, and restarting every instance at once because the database hiccupped is an excellent way to turn a minor incident into a total outage.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus     # ONLY what is needed (12-07)
      base-path: /actuator
  endpoint:
    health:
      probes:
        enabled: true                # enables /health/liveness and /health/readiness
      show-details: when-authorized  # the detail, only to whoever is allowed
      group:
        readiness:
          include: db, diskSpace     # if the DB does not answer, we are not ready
        liveness:
          include: livenessState     # only the process state
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true
$ curl localhost:8080/actuator/health/liveness
{"status":"UP"}

$ curl localhost:8080/actuator/health/readiness
{"status":"UP","components":{"db":{"status":"UP"},"diskSpace":{"status":"UP"}}}

A health indicator of your own, for what only you know is critical:

@Component
public class MetadataGatewayHealth implements HealthIndicator {

    private final MetadataGateway gateway;

    @Override
    public Health health() {
        try {
            boolean available = gateway.checkAvailability();
            return available
                    ? Health.up().withDetail("gateway", "available").build()
                    // DEGRADED, not DOWN: the application works without external metadata.
                    // Marking DOWN here would pull a perfectly usable app out of the balancer.
                    : Health.status("DEGRADED").withDetail("gateway", "not responding").build();
        } catch (Exception e) {
            return Health.status("DEGRADED").withException(e).build();
        }
    }
}

That nuance — degraded rather than down for non-essential dependencies — is what separates a resilient system from one that falls over entirely because a secondary service had a problem.

  1. Graceful shutdown

When the orchestrator wants to stop an instance, it sends SIGTERM. With no preparation, the JVM dies immediately and the in-flight requests are cut: users with errors, half-finished transactions, unacknowledged messages.

server:
  shutdown: graceful            # stops accepting new requests and waits for the current ones

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

The full sequence:

sequenceDiagram
    participant O as Orchestrator
    participant K as Kubelet / Docker
    participant A as BiblioTech
    participant B as Load balancer

    O->>K: stop the pod
    K->>B: remove it from the endpoints
    K->>A: SIGTERM
    A->>A: readiness = DOWN
    A->>A: stop accepting new requests
    A->>A: finish the 12 in-flight requests
    A->>A: close the connection pool
    A->>A: run the shutdown hooks
    A-->>K: process finished (code 143)
    Note over K,A: if it is still alive after terminationGracePeriodSeconds, SIGKILL

And the shutdown hooks, which pick up module 7 and 12-03:

@Component
public class GracefulShutdown {

    private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

    private final ExecutorService importExecutor;

    /**
     * @PreDestroy runs when the Spring context closes,
     * which is what happens on receiving SIGTERM.
     */
    @PreDestroy
    public void onShutdown() {
        log.info("Graceful shutdown: closing resources");

        importExecutor.shutdown();            // accepts no new tasks
        try {
            if (!importExecutor.awaitTermination(20, TimeUnit.SECONDS)) {
                log.warn("Imports unfinished after 20 s; forcing");
                importExecutor.shutdownNow();
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            importExecutor.shutdownNow();
        }
        log.info("Graceful shutdown complete");
    }
}

A critical and little-known detail: there is a race window between the moment the orchestrator sends SIGTERM and the moment the load balancer stops sending traffic. During that gap — up to a few seconds — requests arrive at an instance that is already shutting down. The standard solution is a wait before the shutdown begins:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]     # give the balancer time to update

And in the Dockerfile, dumb-init as PID 1: without it, the Java process (which would be PID 1) does not receive signals by default and none of the above is worth anything.

  1. Startup time: CDS and Native Image

Startup matters in three situations: autoscaling on a traffic spike, restarting after a failure, and serverless functions that scale to zero.

Technique BiblioTech startup Cost
Ordinary jar ~4.5 s
CDS (class data sharing archive) ~3.2 s An extra build step
Spring AOT (-Dspring.aot.enabled) ~2.8 s Limits dynamic configuration
CRaC (restore from a checkpoint) ~0.3 s Specific JVM, high complexity
GraalVM Native Image ~0.08 s 5-10 min build; reflection must be declared

CDS is the cheapest improvement: it memorises the result of loading and verifying the classes.

# Generate the CDS archive during the image build
RUN java -XX:ArchiveClassesAtExit=/app/app.jsa \
         -Dspring.context.exit=onRefresh \
         org.springframework.boot.loader.launch.JarLauncher

ENV JAVA_OPTS="$JAVA_OPTS -XX:SharedArchiveFile=/app/app.jsa"

GraalVM Native Image compiles to a native executable:

<profile>
  <id>native</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.graalvm.buildtools</groupId>
        <artifactId>native-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>
</profile>
./mvnw -Pnative native:compile -pl bibliotech-web
./bibliotech-web/target/bibliotech-web       # starts in 80 ms
Aspect JVM Native Image
Startup 4.5 s 0.08 s
Memory at rest ~350 MB ~90 MB
Peak throughput Higher (JIT optimises with real data) Lower
Build time 30 s 5-10 min
Dynamic reflection Free Must be declared
Diagnostic tooling JFR, jcmd, JMX Limited

When Native Image pays off: serverless functions, CLIs (12-03), microservices that scale to zero, environments with very tight memory. When it does not: long-lived services under sustained load, where the JIT ends up producing faster code than ahead-of-time compilation.

For BiblioTech as a long-lived API: JVM with CDS. For the CLI: Native Image makes a lot of sense.

  1. Deployment strategies

Strategy How it works Downtime Cost Rollback When
Recreate Stop everything, start the new one Yes Low Redeploy Development; apps that cannot tolerate two versions
Rolling Replace instance by instance No Low Roll forward in reverse The default
Blue-green Two complete environments; switch the traffic No Double Instant Risky changes
Canary Send 5% of the traffic to the new one; ramp up No Medium Instant High-risk changes, high traffic
flowchart TD
    subgraph BLUE_GREEN["Blue-green"]
        LB1["Load balancer"] -->|"100%"| AZ["BLUE v1.4.1<br/>(in production)"]
        LB1 -.->|"0%"| VE["GREEN v1.4.2<br/>(deployed, tested)"]
        N1["Switch: traffic moves to GREEN in an instant.<br/>BLUE stays switched on in case you need to go back."]
    end

    subgraph CANARY["Canary"]
        LB2["Load balancer"] -->|"95%"| E1["v1.4.1 (9 instances)"]
        LB2 -->|"5%"| E2["v1.4.2 (1 instance)"]
        N2["Watch errors and latency.<br/>If all is well: 25%, 50%, 100%.<br/>If not: back to 0% at once."]
    end

Feature flags. They are the complement that changes the game, because they separate deployment from activation:

@Service
public class LoanManager {

    private final BiblioTechProperties props;

    public Loan lend(Isbn isbn, Long employeeId, Integer days) {
        if (props.features().autoReservation()) {
            // New code, deployed but switched off until somebody decides
            return lendWithAutoReservation(isbn, employeeId, days);
        }
        return lendClassic(isbn, employeeId, days);
    }
}
bibliotech:
  features:
    auto-reservation: false          # switched on with an environment variable, no deployment
    priority-loan: true
    pdf-report: false

With flags, switching off a problematic feature is a configuration change taking seconds, not a rollback deployment taking minutes. And they let you deploy incomplete code with no risk, which in turn lets you integrate daily instead of keeping long-lived branches (12-01).

The price: every flag is one more branch to test, and forgotten flags pile up. They get removed as soon as the decision is final.

  1. Rollback and the limits of the database

Rolling back the code is easy:

kubectl rollout undo deployment/bibliotech
docker compose up -d --force-recreate   # with the previous tag
sudo systemctl stop bibliotech && cp bibliotech-1.4.1.jar bibliotech-web.jar && sudo systemctl start bibliotech

Rolling back the database almost never is, and this is why:

Change Reversible? Why
ADD COLUMN (nullable) Yes The old version ignores it
CREATE TABLE Yes Nobody uses it
CREATE INDEX Yes It only affects performance
ADD COLUMN NOT NULL with no default No The old version does not fill it on insert
DROP COLUMN No The data is gone
RENAME COLUMN No The old version looks for the old name
ALTER TYPE with data loss No Truncated data does not come back
Data migration It depends Only if the previous state was kept

Hence the safe deployment rule:

The database always goes first and is always backward compatible. First the compatible migration is deployed; then the code that uses it. Never the other way round, and never both in the same step if the change is destructive.

With the expand-contract pattern (section 14), rollback works in phases 1 and 2. In phase 3 it no longer does: that is why phase 3 is applied days later, once the new version is confirmed.

Backups — and the part almost nobody does:

# Daily backup
pg_dump -h db -U bibliotech -Fc bibliotech > bibliotech-$(date +%F).dump

# Backup before ANY destructive migration
pg_dump -h db -U bibliotech -Fc bibliotech > pre-migration-v8-$(date +%F-%H%M).dump

Note. A backup that has never been restored is not a backup: it is a file with hopes attached. Schedule a periodic test restore in a separate environment and measure how long it takes. That time is your real RTO (recovery time objective), and it is usually far higher than people assume. Just as important is the RPO (recovery point objective): with daily backups, you can lose up to 24 hours of data. If that is unacceptable, you need WAL archiving or replication.

  1. The continuous delivery pipeline

The CI from 12-05 verified. Continuous delivery also builds the image, publishes it and deploys it.

flowchart LR
    A["Push to main"] --> B["CI: tests<br/>coverage, analysis"]
    B --> C["Build image<br/>multi-architecture"]
    C --> D["Push to the<br/>registry"]
    D --> E["Deploy to<br/>staging"]
    E --> F["Smoke tests"]
    F --> G{"Manual<br/>approval"}
    G -->|"approved"| H["Deploy to<br/>production"]
    H --> I["Verify health"]
    I -->|"failure"| J["Automatic<br/>rollback"]

    style G fill:#fff3e0,stroke:#e65100
    style J fill:#ffebee,stroke:#c62828
# .github/workflows/cd.yml
name: Continuous delivery

on:
  push:
    branches: [main]
    tags: ['v*']

env:
  REGISTRY: ghcr.io
  IMAGE: ${{ github.repository }}

permissions:
  contents: read
  packages: write
  id-token: write          # to sign the image with cosign

jobs:

  # ---------------------------------------------------------------
  # 1. Reuses the full verification from 12-05
  # ---------------------------------------------------------------
  verify:
    uses: ./.github/workflows/ci.yml

  # ---------------------------------------------------------------
  # 2. Build and publish the image
  # ---------------------------------------------------------------
  image:
    name: Build and publish image
    runs-on: ubuntu-latest
    needs: verify
    outputs:
      digest: ${{ steps.build.outputs.digest }}
      tags: ${{ steps.meta.outputs.tags }}

    steps:
      - uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3        # to build for arm64

      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to the registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Compute tags and metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE }}
          tags: |
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha,prefix=,format=short          # traceability to the commit
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha                    # layer cache between runs
          cache-to: type=gha,mode=max
          provenance: true
          sbom: true                              # component inventory (12-07)

      - name: Scan the image for vulnerabilities
        uses: aquasecurity/[email protected]
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
          severity: 'CRITICAL,HIGH'
          exit-code: '1'                          # blocks if there are serious vulnerabilities

      - name: Sign the image
        uses: sigstore/cosign-installer@v3
      - run: cosign sign --yes ${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ steps.build.outputs.digest }}

  # ---------------------------------------------------------------
  # 3. Staging: automatic
  # ---------------------------------------------------------------
  staging:
    name: Deploy to staging
    runs-on: ubuntu-latest
    needs: image
    environment:
      name: staging
      url: https://staging.bibliotech.nexussoftware.com

    steps:
      - name: Deploy
        run: |
          kubectl set image deployment/bibliotech \
            bibliotech=${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ needs.image.outputs.digest }} \
            --namespace=staging
          kubectl rollout status deployment/bibliotech -n staging --timeout=5m

      - name: Smoke tests
        run: |
          BASE=https://staging.bibliotech.nexussoftware.com
          curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"'
          curl -fsS "$BASE/api/materials?size=1"      | jq -e '.content | length >= 0'
          echo "Smoke tests passed"

  # ---------------------------------------------------------------
  # 4. Production: requires manual approval
  # ---------------------------------------------------------------
  production:
    name: Deploy to production
    runs-on: ubuntu-latest
    needs: [image, staging]
    if: startsWith(github.ref, 'refs/tags/v')     # only from a version tag
    environment:
      name: production                            # with required reviewers configured
      url: https://bibliotech.nexussoftware.com

    steps:
      - name: Record the current revision, in case we have to go back
        id: current
        run: |
          CURRENT=$(kubectl get deployment/bibliotech -n production \
                    -o jsonpath='{.spec.template.spec.containers[0].image}')
          echo "previous_image=$CURRENT" >> $GITHUB_OUTPUT

      - name: Deploy (rolling update)
        run: |
          kubectl set image deployment/bibliotech \
            bibliotech=${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ needs.image.outputs.digest }} \
            --namespace=production
          kubectl rollout status deployment/bibliotech -n production --timeout=10m

      - name: Verify health after the deployment
        id: verify
        run: |
          sleep 30
          BASE=https://bibliotech.nexussoftware.com
          for i in $(seq 1 10); do
            if curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"' > /dev/null; then
              echo "Instance healthy (attempt $i)"; sleep 5
            else
              echo "::error::Health verification failed"; exit 1
            fi
          done
          # Check that the error rate has not shot up
          ERRORS=$(curl -fsS "$BASE/actuator/metrics/http.server.requests?tag=outcome:SERVER_ERROR" \
                   | jq '.measurements[0].value // 0')
          if (( $(echo "$ERRORS > 10" | bc -l) )); then
            echo "::error::Too many 5xx errors after the deployment"; exit 1
          fi

      - name: Automatic rollback if something failed
        if: failure()
        run: |
          echo "::warning::Deployment failed; going back to the previous version"
          kubectl rollout undo deployment/bibliotech -n production
          kubectl rollout status deployment/bibliotech -n production --timeout=5m

      - name: Report the result
        if: always()
        run: |
          STATUS="${{ job.status }}"
          curl -X POST "${{ secrets.TEAM_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d "{\"text\":\"BiblioTech deployment ${{ github.ref_name }}: $STATUS\"}"

Key points of this pipeline:

Decision Reason
Deploy by digest, not by tag A tag can be moved; a digest identifies exact bytes
Blocking vulnerability scan Do not publish an image with critical CVEs (12-07)
Signing with cosign Verifiable: this image was built by our pipeline
Staging automatic, production manual Speed where there is no risk, control where there is
environment with reviewers The approval belongs to the system, not to a chat message
Health verification + automatic rollback A failed deployment is reverted in 2 minutes, with no humans
Production only from a v* tag Every production deployment has an identifiable version

  1. Horizontal scaling and what it demands of the application

Scaling vertically means giving one instance more resources; scaling horizontally means having more instances. The second is what gives high availability and scales without an upper limit, but it demands properties of the application.

Requirements, all of them verifiable:

Requirement Why BiblioTech's status
No in-memory state Request 2 may land on another instance ✅ Nothing in memory since 12-01
Shared session or no session Same reason ✅ Stateless API; JWT in 12-07
No local files Each instance has its own disk ⚠️ Review the exports
Coordinated scheduled tasks Three instances = three runs of the same @Scheduled ⚠️ Pending
Distributed or coherent local cache Local caches diverge ⚠️ Review CatalogWithCache
Migrations with locking Three instances starting at once ✅ Flyway uses a lock

Here the session work from module 7 comes back. Any state living in memory today — the session Map, the local cache, the CLI's undo stack — stops working with more than one instance. The answer is not "do not scale", but moving that state to a shared place: the database, Redis, or removing it by design.

The scheduled-task problem, which is the most frequent and the most surprising:

// WITH 3 INSTANCES: this sends THREE notices to every employee, every day
@Scheduled(cron = "0 0 8 * * *")
public void sendDailyNotices() {
    noticeService.notifyUpcomingDueDates(3);
}

Solutions, from least to most robust:

// Option A: ShedLock — a distributed lock in the database
@Scheduled(cron = "0 0 8 * * *")
@SchedulerLock(name = "dailyNotices", lockAtMostFor = "10m", lockAtLeastFor = "1m")
public void sendDailyNotices() {
    noticeService.notifyUpcomingDueDates(3);
}
# Option B: a Kubernetes CronJob that invokes the CLI from 12-03.
# Advantage: the scheduler does not live inside the application.
apiVersion: batch/v1
kind: CronJob
metadata:
  name: bibliotech-notices
spec:
  schedule: "0 8 * * *"
  concurrencyPolicy: Forbid           # do not overlap runs
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cli
              image: registry.nexussoftware.com/bibliotech-cli:1.4.2
              args: ["notices", "send", "--days-ahead=3"]
              envFrom:
                - secretRef: { name: bibliotech-secrets }

And autoscaling, which in Kubernetes is declarative:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bibliotech
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bibliotech
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300     # do not scale down on the first dip: avoids flapping

With one important warning: scaling the application does not scale the database. Ten instances with 20 connections each are 200 connections to PostgreSQL, which by default accepts 100. Tune the pool or put a pgbouncer in front. The bottleneck moves; it does not disappear.

Common Mistakes and Tips

1. Using the latest tag in production. You do not know what is running, rollback does not work and two instances can end up on different versions. Semantic tags or digests.

2. Running the container as root. It is the default and it is a free vulnerability. A non-root USER, always.

3. Baking secrets into the image. They stay in the layer history; docker history reveals them. Environment variables or secret managers.

4. Not limiting the container's memory. A Java process can eat all the host's memory and take the neighbours down.

5. Limiting memory without tuning MaxRAMPercentage. The default 25% wastes resources, and not accounting for memory outside the heap causes OOMKilled with no message from the JVM.

6. ddl-auto: update in production. It does not delete, does not rename, does not version, is not reversible and fails with several instances starting at once. Flyway.

7. Modifying an already-applied migration. Flyway detects the checksum change and the application does not start. To fix something, a new script.

8. Not testing the rollback. It is exactly what you need on the day everything goes wrong. Test it in staging, timed.

9. Badly configured probes. Confusing liveness with readiness makes the application restart in a loop when the database has a transient problem, turning a minor incident into a total outage.

10. Forgetting dumb-init or exec. Without them, SIGTERM never reaches the JVM and graceful shutdown does not happen: cut requests on every deployment.

11. Uncoordinated scheduled tasks. With three instances, three emails to every employee. ShedLock or an external CronJob.

12. Backups that have never been restored. They are not backups. Test the restore and time it.

A final tip: the best measure of a deployment's quality is how long it takes to roll back. If it is thirty seconds, you will deploy often and calmly. If it is two hours, you will deploy rarely, in large batches and in fear — which is precisely what makes deployments go wrong.

Exercises

Exercise 1: a Dockerfile for the CLI

Write the multi-stage Dockerfile for the bibliotech-console module (12-03), bearing in mind that:

  • The CLI runs and exits: it is not a long-lived service.
  • It must start as fast as possible (it is invoked from cron).
  • It needs no exposed port and no health check.
  • It must accept arguments: docker run bibliotech-cli catalog list --format=json.
  • It must be usable as the image of a Kubernetes CronJob.
  • Tune the JVM options for startup, not for sustained throughput.

Include as well the Kubernetes CronJob that sends the daily notices.

Exercise 2: a two-phase migration

BiblioTech has to split the employees.name field (which today holds "Marta Ruiz") into first_name and surname, with no downtime and with rollback possible at every moment.

Write:

  • The Flyway scripts for the three phases.
  • Which application version accompanies each phase and what its code does.
  • The deployment plan with the points where rollback is possible and where it stops being possible.
  • A test that verifies the migration is correct with real data, including the hard cases (a single name, compound surnames, names with particles).

Exercise 3: a blue-green pipeline

Write the GitHub Actions workflow that deploys BiblioTech with a blue-green strategy on Kubernetes:

  • Determine which colour is currently active.
  • Deploy the new version to the idle colour.
  • Run smoke tests against the idle colour, with no real traffic.
  • Switch the traffic by changing the Service selector.
  • Watch for five minutes and roll back automatically if the error rate rises.
  • Leave the previous colour switched on for an hour before retiring it.

Include the necessary Kubernetes manifests.


Solutions

Solution 1

# ==============================================================================
# STAGE 1: BUILD
# ==============================================================================
FROM eclipse-temurin:21-jdk-alpine AS builder

WORKDIR /build

COPY .mvn/ .mvn/
COPY mvnw pom.xml ./
COPY bibliotech-domain/pom.xml          bibliotech-domain/
COPY bibliotech-application/pom.xml     bibliotech-application/
COPY bibliotech-infrastructure/pom.xml  bibliotech-infrastructure/
COPY bibliotech-console/pom.xml         bibliotech-console/

RUN --mount=type=cache,target=/root/.m2 \
    ./mvnw -B -pl bibliotech-console -am dependency:go-offline -DskipTests

COPY bibliotech-domain/src          bibliotech-domain/src
COPY bibliotech-application/src     bibliotech-application/src
COPY bibliotech-infrastructure/src  bibliotech-infrastructure/src
COPY bibliotech-console/src         bibliotech-console/src

RUN --mount=type=cache,target=/root/.m2 \
    ./mvnw -B -pl bibliotech-console -am clean package -DskipTests

RUN java -Djarmode=tools -jar bibliotech-console/target/bibliotech-cli.jar \
         extract --layers --launcher --destination extracted

# ==============================================================================
# STAGE 2: GENERATE THE CDS ARCHIVE
# The application is run once with --help so that it loads the classes,
# and the result is memorised. This trims ~1.5 s off every startup, which
# multiplied by the cron runs does matter.
# ==============================================================================
FROM eclipse-temurin:21-jre-alpine AS cds

WORKDIR /app
COPY --from=builder /build/extracted/ ./

RUN java -XX:ArchiveClassesAtExit=/app/cli.jsa \
         org.springframework.boot.loader.launch.JarLauncher --help > /dev/null 2>&1 || true

# ==============================================================================
# STAGE 3: RUNTIME
# ==============================================================================
FROM eclipse-temurin:21-jre-alpine AS runtime

LABEL org.opencontainers.image.title="BiblioTech CLI" \
      org.opencontainers.image.description="BiblioTech command-line tool" \
      org.opencontainers.image.vendor="Nexus Software"

# tzdata for the dates; dumb-init so that Ctrl+C and SIGTERM reach the process
# (the clean cancellation from 12-03 depends on this).
# curl is NOT installed: there is no healthcheck to run.
RUN apk add --no-cache tzdata dumb-init && rm -rf /var/cache/apk/*
ENV TZ=Europe/Madrid

RUN addgroup -S -g 1001 bibliotech && \
    adduser -S -u 1001 -G bibliotech -h /app bibliotech

WORKDIR /app

COPY --from=cds --chown=bibliotech:bibliotech /app/ ./

USER bibliotech

# Options AIMED AT STARTUP, not at sustained throughput.
# The process lives seconds: compiling fully with C2 never pays for itself.
#   TieredStopAtLevel=1    C1 only: fast, lightweight compilation
#   UseSerialGC            the cheapest GC to initialise (one task, little memory)
#   SharedArchiveFile      uses the CDS from stage 2
#   MaxRAMPercentage=75    cgroup-aware, as always
ENV JAVA_OPTS="\
    -XX:TieredStopAtLevel=1 \
    -XX:+UseSerialGC \
    -XX:SharedArchiveFile=/app/cli.jsa \
    -XX:MaxRAMPercentage=75.0 \
    -Xshare:auto \
    -Dspring.main.banner-mode=off \
    -Dfile.encoding=UTF-8"

# NO EXPOSE: the CLI listens on no port.
# NO HEALTHCHECK: the process exits; there is nothing to watch.

ENTRYPOINT ["dumb-init", "--", "sh", "-c", \
            "exec java $JAVA_OPTS org.springframework.boot.loader.launch.JarLauncher \"$@\"", "--"]

# CMD holds the DEFAULT arguments, replaceable at run time
CMD ["--help"]
docker build -f Dockerfile.cli -t bibliotech-cli:1.4.2 .

docker run --rm bibliotech-cli:1.4.2 catalog list --format=json
docker run --rm -e BIBLIOTECH_DB_URL=… bibliotech-cli:1.4.2 notices send --days-ahead=3

# Check the effect of CDS
docker run --rm bibliotech-cli:1.4.2 --version   # ~1.1 s instead of ~2.6 s

The ENTRYPOINT deserves an explanation, because it is the part people get stuck on: the sh -c plus "$@" form makes it possible to expand $JAVA_OPTS and receive the user's arguments at the same time. The trailing -- is the script's $0, without which the user's first argument would be lost.

The Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: bibliotech-daily-notices
  labels:
    app: bibliotech
    component: scheduled-tasks
spec:
  schedule: "0 8 * * 1-5"           # Monday to Friday at 8:00
  timeZone: "Europe/Madrid"         # Kubernetes 1.27+: essential with daylight saving

  concurrencyPolicy: Forbid         # if the previous one is still running, do NOT start another
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  startingDeadlineSeconds: 600      # if the cluster was down, there is a 10 min margin

  jobTemplate:
    spec:
      backoffLimit: 2               # 2 retries on failure
      activeDeadlineSeconds: 900    # kill it if it exceeds 15 minutes
      ttlSecondsAfterFinished: 86400

      template:
        metadata:
          labels:
            app: bibliotech
            task: notices
        spec:
          restartPolicy: OnFailure

          securityContext:
            runAsNonRoot: true
            runAsUser: 1001

          containers:
            - name: cli
              image: registry.nexussoftware.com/bibliotech-cli:1.4.2
              imagePullPolicy: IfNotPresent

              args:
                - "notices"
                - "send"
                - "--days-ahead=3"
                - "--quiet"            # no decoration: the output goes to the log

              envFrom:
                - configMapRef: { name: bibliotech-config }
                - secretRef:    { name: bibliotech-secrets }

              resources:
                requests: { memory: "256Mi", cpu: "100m" }
                limits:   { memory: "512Mi", cpu: "1000m" }

              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities: { drop: ["ALL"] }

              volumeMounts:
                - name: tmp
                  mountPath: /tmp

          volumes:
            - name: tmp
              emptyDir: {}

And this is where the design from 12-03 pays off: the exit codes govern Kubernetes' behaviour. Code 0 (OK) and 3 (nothing to send) mark the Job as successful; code 7 (email unavailable) marks it as failed and backoffLimit: 2 retries automatically. Without those differentiated codes, it would either always retry or never retry.

kubectl get cronjob bibliotech-daily-notices
kubectl create job --from=cronjob/bibliotech-daily-notices manual-run   # run it now
kubectl logs job/manual-run

Solution 2

Phase 1 — Expand. A migration compatible with application v1.4.x, which still uses name.

-- V9__split_employee_name_phase1.sql

ALTER TABLE employees ADD COLUMN first_name VARCHAR(80);
ALTER TABLE employees ADD COLUMN surname    VARCHAR(120);

-- Populate with a conservative heuristic:
-- the FIRST word is the given name; the rest is the surname.
-- It is imperfect for compound names ("José María"), which is why
-- the original column is kept and a review report is produced.
UPDATE employees
SET first_name = split_part(trim(name), ' ', 1),
    surname    = NULLIF(trim(substring(trim(name) from position(' ' in trim(name)) + 1)), '')
WHERE first_name IS NULL;

-- Special case: a single term (no spaces) → it is all given name
UPDATE employees
SET first_name = trim(name), surname = NULL
WHERE position(' ' in trim(name)) = 0;

-- Bidirectional sync trigger: it does not matter which app version writes
CREATE OR REPLACE FUNCTION sync_employee_name() RETURNS TRIGGER AS $$
BEGIN
    IF TG_OP = 'INSERT' THEN
        IF NEW.first_name IS NULL AND NEW.name IS NOT NULL THEN
            -- App v1 wrote (old column): derive the new ones
            NEW.first_name := split_part(trim(NEW.name), ' ', 1);
            NEW.surname    := NULLIF(trim(substring(trim(NEW.name)
                                   from position(' ' in trim(NEW.name)) + 1)), '');
        ELSIF NEW.name IS NULL AND NEW.first_name IS NOT NULL THEN
            -- App v2 wrote (new columns): derive the old one
            NEW.name := trim(NEW.first_name || ' ' || COALESCE(NEW.surname, ''));
        END IF;
    ELSIF TG_OP = 'UPDATE' THEN
        IF NEW.name IS DISTINCT FROM OLD.name THEN
            NEW.first_name := split_part(trim(NEW.name), ' ', 1);
            NEW.surname    := NULLIF(trim(substring(trim(NEW.name)
                                   from position(' ' in trim(NEW.name)) + 1)), '');
        ELSIF NEW.first_name IS DISTINCT FROM OLD.first_name
           OR NEW.surname    IS DISTINCT FROM OLD.surname THEN
            NEW.name := trim(NEW.first_name || ' ' || COALESCE(NEW.surname, ''));
        END IF;
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_sync_employee_name
    BEFORE INSERT OR UPDATE ON employees
    FOR EACH ROW EXECUTE FUNCTION sync_employee_name();

-- Manual review view: the cases the heuristic probably got wrong
CREATE OR REPLACE VIEW v_employees_name_review AS
SELECT id, name, first_name, surname,
       CASE
         WHEN first_name IN ('José','Jose','María','Maria','Juan','Ana','Luis','Francisco')
              AND surname LIKE '% %'                     THEN 'possible compound given name'
         WHEN surname ~ '^(de|del|la|las|los|van|von|di|da) ' THEN 'surname with particle'
         WHEN surname IS NULL                            THEN 'no surname'
         ELSE 'review'
       END AS reason
FROM employees
WHERE first_name IN ('José','Jose','María','Maria','Juan','Ana','Luis','Francisco')
   OR surname ~ '^(de|del|la|las|los|van|von|di|da) '
   OR surname IS NULL;

Phase 2 — Migrate. Application v1.5.0 uses the new columns.

@Entity
public class Employee {

    @Column(name = "first_name", length = 80)
    private String firstName;

    @Column(name = "surname", length = 120)
    private String surname;

    /**
     * The old column still exists and the trigger maintains it.
     * insertable/updatable false: JPA NEVER writes it.
     * It is kept mapped only so that it can be read if ever needed.
     */
    @Column(name = "name", insertable = false, updatable = false)
    private String legacyFullName;

    public String fullName() {
        return surname == null ? firstName : firstName + " " + surname;
    }
}

Phase 3 — Contract. Only once v1.4.x no longer exists in any environment.

-- V11__split_employee_name_phase3.sql

-- Pre-check: if anything ended up inconsistent, ABORT
DO $$
DECLARE inconsistent INTEGER;
BEGIN
    SELECT count(*) INTO inconsistent
    FROM employees
    WHERE first_name IS NULL
       OR trim(name) IS DISTINCT FROM trim(first_name || ' ' || COALESCE(surname, ''));

    IF inconsistent > 0 THEN
        RAISE EXCEPTION 'There are % employees with an inconsistent name. Check v_employees_name_review before contracting.', inconsistent;
    END IF;
END $$;

DROP TRIGGER IF EXISTS trg_sync_employee_name ON employees;
DROP FUNCTION IF EXISTS sync_employee_name();
DROP VIEW IF EXISTS v_employees_name_review;

ALTER TABLE employees ALTER COLUMN first_name SET NOT NULL;
ALTER TABLE employees DROP COLUMN name;

CREATE INDEX idx_employees_surname ON employees(surname, first_name);

Deployment plan with the points of no return:

Step Action Rollback Duration
1 Full backup 10 min
2 Apply V9 (phase 1) Yes: drop the columns and the trigger 2 min
3 Check the review view and fix by hand Yes 1-2 h
4 Deploy app v1.5.0 Yes: go back to v1.4.x 5 min
5 Watch for 48 hours Yes 2 days
6 Apply V11 (phase 3) NO. Point of no return 1 min

Testing the migration with hard data:

@Tag("integration")
class EmployeeNameMigrationIT extends PostgresTestBase {

    @Test
    void splitsTheKnownNamesCorrectly() {
        // Initial state: up to V8, with the old column
        flywayUpTo("8");
        jdbc.update("""
                insert into employees (name, email, start_date) values
                    ('Marta Ruiz',              '[email protected]',  '2024-03-01'),
                    ('Diego Alonso',            '[email protected]',  '2025-01-15'),
                    ('Nuria Vidal',             '[email protected]',  '2023-09-10'),
                    ('José María Pérez Gómez',  '[email protected]',   '2022-05-20'),
                    ('Ana de la Torre',         '[email protected]',    '2021-11-02'),
                    ('Prince',                  '[email protected]', '2020-01-01')
                """);

        flywayUpTo("9");      // apply phase 1

        assertThat(namesOf("[email protected]"))
                .containsExactly("Marta", "Ruiz");
        assertThat(namesOf("[email protected]"))
                .containsExactly("Nuria", "Vidal");

        // Hard cases: the heuristic splits them wrongly, and that is EXPECTED
        assertThat(namesOf("[email protected]"))
                .containsExactly("José", "María Pérez Gómez");     // needs manual review
        assertThat(namesOf("[email protected]"))
                .containsExactly("Ana", "de la Torre");

        // A single term
        assertThat(namesOf("[email protected]"))
                .containsExactly("Prince", null);

        // And they all appear in the review view, which is what matters:
        // the migration does not aim to always get it right, it aims NOT TO LOSE DATA
        // and to flag what needs reviewing.
        assertThat(jdbc.queryForList("select email from v_employees_name_review", String.class))
                .contains("[email protected]", "[email protected]",
                          "[email protected]");
    }

    @Test
    void theTriggerSyncsInBothDirections() {
        flywayUpTo("9");

        // App v1 writes the old column
        jdbc.update("insert into employees (name, email, start_date) values (?,?,?)",
                    "Carlos Sanz", "[email protected]", Date.valueOf("2026-01-01"));
        assertThat(namesOf("[email protected]")).containsExactly("Carlos", "Sanz");

        // App v2 writes the new columns
        jdbc.update("""
                insert into employees (first_name, surname, email, start_date)
                values (?,?,?,?)""",
                "Elena", "Ferrer Rico", "[email protected]", Date.valueOf("2026-01-02"));
        assertThat(jdbc.queryForObject(
                "select name from employees where email = ?", String.class,
                "[email protected]"))
                .isEqualTo("Elena Ferrer Rico");
    }

    @Test
    void phase3AbortsIfInconsistenciesRemain() {
        flywayUpTo("9");
        // Force an inconsistency by bypassing the trigger
        jdbc.update("alter table employees disable trigger trg_sync_employee_name");
        jdbc.update("insert into employees (name, email, start_date) values (?,?,?)",
                    "Not Split", "[email protected]", Date.valueOf("2026-01-03"));
        jdbc.update("alter table employees enable trigger trg_sync_employee_name");

        assertThatThrownBy(() -> flywayUpTo("11"))
                .hasMessageContaining("inconsistent name");

        // And most important of all: the old column IS STILL THERE. Nothing has been lost.
        assertThat(columnExists("employees", "name")).isTrue();
    }
}

Solution 3

Kubernetes manifests:

# k8s/blue-green/service.yaml
# The Service points at ONE colour. Switching = changing the selector.
apiVersion: v1
kind: Service
metadata:
  name: bibliotech
  labels: { app: bibliotech }
spec:
  selector:
    app: bibliotech
    colour: blue               # ← this is what changes on the switch
  ports:
    - port: 80
      targetPort: 8080
---
# Auxiliary Service to test the idle colour WITHOUT real traffic
apiVersion: v1
kind: Service
metadata:
  name: bibliotech-preview
spec:
  selector:
    app: bibliotech
    colour: green              # adjusted before the smoke tests
  ports:
    - port: 80
      targetPort: 8080
---
# k8s/blue-green/deployment-template.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bibliotech-COLOUR
  labels: { app: bibliotech, colour: COLOUR }
spec:
  replicas: 3
  selector:
    matchLabels: { app: bibliotech, colour: COLOUR }
  template:
    metadata:
      labels: { app: bibliotech, colour: COLOUR }
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: bibliotech
          image: IMAGE
          ports: [{ name: http, containerPort: 8080 }]
          envFrom:
            - configMapRef: { name: bibliotech-config }
            - secretRef:    { name: bibliotech-secrets }
          resources:
            requests: { memory: "512Mi", cpu: "250m" }
            limits:   { memory: "768Mi", cpu: "1500m" }
          startupProbe:
            httpGet: { path: /actuator/health/liveness, port: http }
            failureThreshold: 20
            periodSeconds: 5
          readinessProbe:
            httpGet: { path: /actuator/health/readiness, port: http }
            periodSeconds: 5

The workflow:

# .github/workflows/cd-blue-green.yml
name: Blue-green deployment

on:
  push:
    tags: ['v*']

env:
  NAMESPACE: production
  REGISTRY: ghcr.io
  IMAGE: ${{ github.repository }}

jobs:

  # ---------------------------------------------------------------
  # 1. Determine the colours
  # ---------------------------------------------------------------
  colours:
    runs-on: ubuntu-latest
    outputs:
      active: ${{ steps.detect.outputs.active }}
      idle:   ${{ steps.detect.outputs.idle }}
    steps:
      - name: Set up kubectl
        uses: azure/k8s-set-context@v4
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Detect the active colour
        id: detect
        run: |
          ACTIVE=$(kubectl get service bibliotech -n $NAMESPACE \
                   -o jsonpath='{.spec.selector.colour}')
          if [ "$ACTIVE" = "blue" ]; then IDLE="green"; else IDLE="blue"; fi
          echo "active=$ACTIVE" >> $GITHUB_OUTPUT
          echo "idle=$IDLE"     >> $GITHUB_OUTPUT
          echo "::notice::Active: $ACTIVE — will deploy to: $IDLE"

  # ---------------------------------------------------------------
  # 2. Deploy to the idle colour (no traffic)
  # ---------------------------------------------------------------
  deploy-idle:
    runs-on: ubuntu-latest
    needs: colours
    steps:
      - uses: actions/checkout@v4
      - uses: azure/k8s-set-context@v4
        with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }

      - name: Render and apply the idle colour's Deployment
        run: |
          COLOUR=${{ needs.colours.outputs.idle }}
          IMG=${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ github.ref_name }}

          sed -e "s|COLOUR|$COLOUR|g" -e "s|IMAGE|$IMG|g" \
              k8s/blue-green/deployment-template.yaml | kubectl apply -n $NAMESPACE -f -

          kubectl rollout status deployment/bibliotech-$COLOUR -n $NAMESPACE --timeout=10m

      - name: Point the preview Service at the idle colour
        run: |
          kubectl patch service bibliotech-preview -n $NAMESPACE \
            -p '{"spec":{"selector":{"app":"bibliotech","colour":"${{ needs.colours.outputs.idle }}"}}}'

  # ---------------------------------------------------------------
  # 3. Smoke tests against the idle colour
  # ---------------------------------------------------------------
  smoke:
    runs-on: ubuntu-latest
    needs: [colours, deploy-idle]
    steps:
      - uses: azure/k8s-set-context@v4
        with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }

      - name: Run the tests against the idle colour
        run: |
          kubectl port-forward service/bibliotech-preview 18080:80 -n $NAMESPACE &
          PF=$!
          sleep 8
          set -e

          BASE=http://localhost:18080

          echo "→ Health"
          curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"'

          echo "→ Deployed version"
          VERSION=$(curl -fsS "$BASE/actuator/info" | jq -r '.build.version')
          test "v$VERSION" = "${{ github.ref_name }}" \
            || { echo "::error::Unexpected version: $VERSION"; exit 1; }

          echo "→ Catalogue"
          curl -fsS "$BASE/api/materials?size=1" | jq -e '.content | length >= 0'

          echo "→ Expected errors (404 and 400)"
          test "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/api/materials/978-9999999999")" = "404"
          test "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/api/materials/not-an-isbn")" = "400"

          kill $PF
          echo "Smoke tests passed"

  # ---------------------------------------------------------------
  # 4. Switch the traffic (with manual approval)
  # ---------------------------------------------------------------
  switch:
    runs-on: ubuntu-latest
    needs: [colours, smoke]
    environment:
      name: production          # with required reviewers
      url: https://bibliotech.nexussoftware.com
    steps:
      - uses: azure/k8s-set-context@v4
        with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }

      - name: Switch the Service to the new colour
        run: |
          kubectl patch service bibliotech -n $NAMESPACE \
            -p '{"spec":{"selector":{"app":"bibliotech","colour":"${{ needs.colours.outputs.idle }}"}}}'
          echo "::notice::Traffic switched to ${{ needs.colours.outputs.idle }}"

  # ---------------------------------------------------------------
  # 5. Watch for 5 minutes; roll back if the error rate rises
  # ---------------------------------------------------------------
  monitor:
    runs-on: ubuntu-latest
    needs: [colours, switch]
    steps:
      - uses: azure/k8s-set-context@v4
        with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }

      - name: Watch the error rate for 5 minutes
        id: monitoring
        run: |
          BASE=https://bibliotech.nexussoftware.com
          ERROR_THRESHOLD=0.02         # 2% of 5xx

          for i in $(seq 1 10); do
            sleep 30

            TOTAL=$(curl -fsS "$BASE/actuator/metrics/http.server.requests" \
                    | jq '.measurements[] | select(.statistic=="COUNT") | .value')
            ERRORS=$(curl -fsS "$BASE/actuator/metrics/http.server.requests?tag=outcome:SERVER_ERROR" \
                     | jq '.measurements[] | select(.statistic=="COUNT") | .value // 0')

            RATE=$(echo "scale=4; $ERRORS / ($TOTAL + 1)" | bc)
            echo "Check $i/10 — total=$TOTAL errors=$ERRORS rate=$RATE"

            if (( $(echo "$RATE > $ERROR_THRESHOLD" | bc -l) )); then
              echo "::error::Error rate $RATE above the threshold $ERROR_THRESHOLD"
              exit 1
            fi

            if ! curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"' > /dev/null; then
              echo "::error::The readiness probe is failing"
              exit 1
            fi
          done
          echo "Monitoring passed"

      - name: Automatic ROLLBACK
        if: failure()
        run: |
          echo "::warning::Going back to colour ${{ needs.colours.outputs.active }}"
          # The rollback is INSTANT: the previous colour is still on and healthy
          kubectl patch service bibliotech -n $NAMESPACE \
            -p '{"spec":{"selector":{"app":"bibliotech","colour":"${{ needs.colours.outputs.active }}"}}}'

          curl -X POST "${{ secrets.TEAM_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d '{"text":"🔴 BiblioTech ${{ github.ref_name }}: automatic rollback to ${{ needs.colours.outputs.active }}"}'
          exit 1

  # ---------------------------------------------------------------
  # 6. Retire the old colour, one hour later
  # ---------------------------------------------------------------
  retire-old:
    runs-on: ubuntu-latest
    needs: [colours, monitor]
    steps:
      - uses: azure/k8s-set-context@v4
        with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }

      - name: Wait an hour before retiring
        run: sleep 3600      # safety window: instant rollback available for 1 hour

      - name: Scale the old colour down to zero replicas
        run: |
          # The Deployment is not DELETED: it is scaled to 0.
          # That way the manifest survives and bringing it back is one command.
          kubectl scale deployment/bibliotech-${{ needs.colours.outputs.active }} \
            --replicas=0 -n $NAMESPACE

          curl -X POST "${{ secrets.TEAM_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d '{"text":"✅ BiblioTech ${{ github.ref_name }} stable. Colour ${{ needs.colours.outputs.active }} retired."}'

Advantages of blue-green over a rolling update, which is what the exercise assesses:

Aspect Rolling Blue-green
Rollback Roll forward in reverse: minutes Instant: one patch
Versions coexisting Yes, unavoidably No: all the traffic goes to one
Testing before exposing No Yes, with the preview Service
Resources needed 1× + 1 pod during the transition
Schema compatibility Mandatory Still advisable, because of rollback

And the point that closes the circle with section 21: instant rollback only works if the database is compatible with both versions. If the new version applied a destructive migration, patching the Service back to the old colour fixes nothing: the old application will find a schema it does not understand. That is why the expand-contract pattern is not optional but the condition that makes blue-green mean anything.

Conclusion

BiblioTech is in production.

You understand what really separates your laptop from a real environment — Java version, configuration, schema, memory, the consequences of a failure — and the three principles that govern a healthy deployment: one artefact for every environment, configuration from the outside and everything must be undoable.

You package into an executable jar, knowing why it displaced the war and how it is built inside. And you use the layered jar, which is not a detail: it turns a 60 MB deployment into a 1 MB one, from forty seconds to three, and with that it changes the team's behaviour — because a three-second deployment gets done without a second thought and a two-minute one gets stockpiled "for Thursday". With reproducible builds and traceability to the commit, so that "which version is in production?" has an exact answer in one second.

You containerise with a multi-stage Dockerfile that you understand line by line: the build stage with Maven that never reaches the final image; the POMs copied before the code so the dependency cache works; the non-root user; the layers ordered by stability; dumb-init as PID 1 without which SIGTERM never reaches the JVM; and the JVM options with MaxRAMPercentage and ExitOnOutOfMemoryError. With a .dockerignore that stops secrets and the .git directory entering the context, and knowing the alternatives — Buildpacks and Jib — with their real advantages.

You know what almost nobody knows about the JVM in a container: that since Java 10 it is cgroup-aware, that the default 25% wastes memory, and above all that the JVM's memory is not only the heap — metaspace, stacks, code cache, direct buffers — which is why a Java container dies with OOMKilled without the application logging a thing.

You govern the schema with Flyway, with the six concrete reasons why ddl-auto: update is no good in production, the version convention, the golden rules — an applied script is never modified — and the three-phase expand-contract pattern, which is what lets a migration coexist with two versions of the application while keeping rollback possible.

You choose where to deploy with judgement, knowing that systemd on your own server is still a perfectly valid answer and that adopting Kubernetes for three services with no platform team is a decision you pay for every week. And if it is Kubernetes, you have a Deployment with maxUnavailable: 0, exact tags instead of latest, resource limits and a PodDisruptionBudget.

You expose probes distinguishing liveness from readiness — the difference between restarting every instance because the database hiccupped and simply taking them out of the balancer — with custom indicators that report degraded rather than down for non-essential dependencies. You shut down gracefully with server.shutdown: graceful, @PreDestroy and the preStop wait that covers the race window with the balancer. And you know the fast-startup options — CDS, AOT, CRaC, Native Image — with the criteria for when each one pays off.

You handle the four deployment strategies with their costs, the feature flags that separate deployment from activation, and the hard limit of rollback: the code comes back, the data does not. Hence the rule that sums it all up: the database goes first and is always backward compatible.

You have the complete continuous delivery pipeline: multi-architecture builds, a blocking vulnerability scan, image signing, deployment by digest, automatic staging, production with approval, post-deployment health verification and automatic rollback. And you know what horizontal scaling demands of the application — no in-memory state, scheduled tasks coordinated with ShedLock or a CronJob, and the reminder that scaling the application does not scale the database.

BiblioTech works, it is tested, it is deployed and it can be updated without interrupting the service. And it has two holes the size of an entire project.

The first: anybody can do anything. There is no authentication, no authorisation, passwords do not exist, the API is wide open and nobody has checked whether it is vulnerable to the things that put applications in the news.

The second: when something fails, you will find out from a phone call. There are no metrics, no traces, no alerts, and the only log is plain text in an ephemeral container.

The final lesson closes both, and closes the course: security — common vulnerabilities and their concrete prevention, Spring Security with BCrypt and JWT, and the warning about what a course cannot replace —, observability — the three pillars, Micrometer and Actuator, Prometheus and Grafana, distributed tracing and useful alerts —, and evolution — API versioning, technical debt, upgrades and how to grow the system. And at the end, a recap of BiblioTech's complete journey, what you can do now, and where to go next.

Java Programming Course

Module 1: Introduction to Java

Module 2: Control Flow

Module 3: Object-Oriented Programming

Module 4: Advanced Object-Oriented Programming

Module 5: Data Structures and Collections

Module 6: Exception Handling

Module 7: File Input/Output

Module 8: Multithreading and Concurrency

Module 9: Networking

Module 10: Advanced Topics

Module 11: Java Frameworks and Libraries

Module 12: Building Real-World Applications

© Copyright 2026. All rights reserved