BiblioTech is tested, measured and verified on every change. And it exists for nobody.
It runs on Diego Alonso's laptop when he starts it, and on a GitHub Actions runner for the eight minutes the pipeline lasts. Marta Ruiz cannot open a browser and look at the catalogue, because there is no server anywhere running the application. Nuria Vidal cannot reserve "Refactoring", because the process that would serve that request is not switched on anywhere.
This lesson covers the journey from "it works on my machine" to "it is in production". It is a journey with more traps than it seems, and almost all of them boil down to a phrase you will hear many times in your career: "well, it worked locally". It worked because locally there was a different Java version, a configuration file that is not in the repository, a database schema that Hibernate had created on its own, 32 GB of memory and no other user competing for it.
The goal of everything that follows is to eliminate those differences: package the application with everything it needs, configure it from the outside, and deploy it in a repeatable, observable and reversible way.
By the end you will know how to package and containerise a Java application properly, configure the JVM so that it respects a container's limits, version the database schema with Flyway, choose between the different deployment platforms with good judgement, expose health probes and shut the application down without cutting requests in half, apply deployment strategies with rollback, and build a complete continuous delivery pipeline.
Contents
- What deploying means
- Packaging: executable jar versus war
- The layered jar and why it speeds images up
- Reproducible builds and traceability
- Containers: image and container
- BiblioTech's
Dockerfile, line by line .dockerignore- Alternatives without a
Dockerfile: Buildpacks and Jib - Choosing a base image and its size
- The JVM inside a container
docker-composefor the local environment- Configuration and secrets at deployment time
- Database migrations with Flyway
- Two-phase deployment for schema changes
- Where it gets deployed: platform comparison
- Kubernetes: a minimal
Deployment - Startup and health: Actuator probes
- Graceful shutdown
- Startup time: CDS and Native Image
- Deployment strategies
- Rollback and the limits of the database
- The continuous delivery pipeline
- Horizontal scaling and what it demands of the application
- Common Mistakes and Tips
- Exercises
- Conclusion
- What deploying means
Deploying means making a specific version of the software available to its users, in an environment you do not fully control. The differences from your laptop are systematic:
| Aspect | Your machine | Production |
|---|---|---|
| Java version | Whichever you have installed | Whichever the operator decides |
| Configuration | application-dev.yml |
Environment variables |
| Database | Ephemeral container, all yours | Shared, with real data |
| Schema | Hibernate creates it | Versioned migrations |
| Memory | 32 GB | 512 MB with a hard limit |
| Failures | You restart | Somebody gets a phone call |
| Restarting | No consequences | Cut requests, affected users |
| Data | Test data | Unrecoverable if lost |
From that come the three principles that govern this lesson:
- One artefact, every environment. The same jar and the same image go to development, staging and production. The only thing that changes is the configuration. If you build a different image for production, what you tested is not what you deploy.
- Configuration from the outside. Nothing environment-specific inside the artefact.
- Everything must be undoable. A deployment with no way back is a gamble.
- Packaging: executable jar versus war
Historically, a Java web application was packaged in a .war and deployed inside an application server installed separately. Spring Boot flipped the model with the executable jar, which carries the server inside.
| Aspect | Executable jar | War |
|---|---|---|
| Server | Embedded | Installed separately |
| Running it | java -jar app.jar |
Copy into webapps/ |
| Deployable units | One | Two: server and application |
| Server version | The one the POM declares | The operator's |
| Containers | Fits perfectly | Awkward |
| Several apps per server | No | Yes |
| When to use it | Practically always | An imposed corporate application server |
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>com.nexussoftware.bibliotech.web.BiblioTechApplication</mainClass>
</configuration>
</plugin>One detail worth knowing: Spring Boot's executable jar is not an ordinary jar. Your classes live in BOOT-INF/classes/ and the dependencies, as complete jars, in BOOT-INF/lib/. A custom class loader (JarLauncher) takes care of reading them. That is why java -cp app.jar MyClass does not work the way you would expect.
bibliotech-web-1.0.0.jar
├── META-INF/MANIFEST.MF ← Main-Class: org.springframework.boot.loader.launch.JarLauncher
├── org/springframework/boot/loader/ ← the loader
└── BOOT-INF/
├── classes/ ← your code and your resources
├── lib/ ← ~50 dependency jars
└── classpath.idx
- The layered jar and why it speeds images up
Here is an optimisation that looks like a detail and radically changes deployment times.
A Docker image is made of stacked layers. When an image is pushed or pulled, only the layers that changed travel. If the whole jar (60 MB) is in a single layer, changing one line of code forces a 60 MB transfer.
But the make-up of those 60 MB is very uneven:
| Content | Typical size | Change frequency |
|---|---|---|
| Dependencies (Spring, Hibernate, Jackson…) | ~55 MB | Every few weeks |
| Spring Boot loader | ~200 KB | With the Spring Boot version |
| Internal dependencies (our own modules) | ~500 KB | Daily |
| Your code | ~1 MB | Every commit |
Spring Boot offers layertools, which splits the jar into those four parts:
$ java -Djarmode=tools -jar bibliotech-web.jar list-layers
dependencies
spring-boot-loader
snapshot-dependencies
application# Extract each layer into its own directory
java -Djarmode=tools -jar bibliotech-web.jar extract --layers --launcher --destination extracted/And in the Dockerfile, each layer is copied by a separate COPY, in order of stability. The measured result:
| Scenario | Without layers | With layers |
|---|---|---|
| One line of code changed | 60 MB transferred | ~1 MB |
| Adding a dependency | 60 MB | ~57 MB |
| Typical push time | 40 s | 3 s |
| Pull time at deployment | 30 s | 2 s |
With twenty deployments a day, that is hours a month. And the psychological effect matters as much as the technical one: a three-second deployment gets done without a second thought; a two-minute one gets stockpiled "so we do them all together on Thursday", which is exactly the practice to avoid.
It is enabled in the POM (it is on by default in Spring Boot 3):
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<layers>
<enabled>true</enabled>
</layers>
</configuration>
</plugin>
- Reproducible builds and traceability
A reproducible build means that the same source code produces exactly the same bytes. It sounds academic and it has a very concrete practical consequence: being able to verify that the binary in production corresponds to the code it claims to correspond to.
What breaks reproducibility is the timestamps inside the jar:
<properties>
<!-- Fixes the date of the jar entries: without this, every build differs -->
<project.build.outputTimestamp>2026-08-05T00:00:00Z</project.build.outputTimestamp>
</properties>./mvnw clean package
sha256sum target/bibliotech-web-1.0.0.jar
# a3f7... (the same hash on any machine, today and a year from now)Traceability. Every artefact must be able to answer: which commit did it come from? when was it built? who built it?
<plugin>
<groupId>io.github.git-commit-id</groupId>
<artifactId>git-commit-id-maven-plugin</artifactId>
<version>9.0.1</version>
<executions>
<execution><goals><goal>revision</goal></goals></execution>
</executions>
<configuration>
<generateGitPropertiesFile>true</generateGitPropertiesFile>
<includeOnlyProperties>
<property>^git.branch$</property>
<property>^git.commit.id.abbrev$</property>
<property>^git.commit.time$</property>
<property>^git.build.version$</property>
</includeOnlyProperties>
</configuration>
</plugin>With that, Actuator exposes the information:
$ curl https://bibliotech.nexussoftware.com/actuator/info
{
"git": {
"branch": "main",
"commit": { "id": "a3f7e91", "time": "2026-08-05T09:14:22Z" }
},
"build": { "version": "1.4.2", "artifact": "bibliotech-web", "time": "2026-08-05T09:20:11Z" }
}The question "which version is in production right now?" has an exact answer, in one second. Without that, the answer is "I think last week's", and from there on no diagnosis is trustworthy.
Semantic versioning of the artefacts:
| Format | Example | Use |
|---|---|---|
MAJOR.MINOR.PATCH |
1.4.2 |
Released version |
MAJOR.MINOR.PATCH-SNAPSHOT |
1.5.0-SNAPSHOT |
In development |
MAJOR.MINOR.PATCH-rc.N |
1.5.0-rc.1 |
Release candidate |
| With the commit | 1.4.2-a3f7e91 |
Exact traceability |
- Containers: image and container
Two concepts that are constantly confused:
- An image is an immutable read-only template: a layered file system + metadata (which command to run, which ports, which user). It is like a class.
- A container is a running instance of an image, with a writable layer on top. It is like an object.
flowchart TD
B["Base image<br/>eclipse-temurin:21-jre-alpine"]
L1["Layer: dependencies (~55 MB)"]
L2["Layer: loader (~200 KB)"]
L3["Layer: BiblioTech code (~1 MB)"]
I["IMAGE bibliotech:1.4.2"]
C1["Container 1<br/>running"]
C2["Container 2<br/>running"]
B --> L1 --> L2 --> L3 --> I
I --> C1
I --> C2
A container is not a virtual machine: it shares the host's kernel and uses Linux mechanisms (namespaces for isolation, cgroups for resource limits). That is why it starts in milliseconds and weighs megabytes instead of gigabytes.
| Aspect | Virtual machine | Container |
|---|---|---|
| Isolation | Total (its own kernel) | Process-level (shared kernel) |
| Startup | Minutes | Milliseconds |
| Size | GB | MB |
| Overhead | Noticeable | Minimal |
What a container really solves, and what justifies all the rest: the artefact includes the base operating system, the JVM, the application and its dependencies. The phrase "it works on my machine" stops meaning anything, because the machine travels with the application.
- BiblioTech's
Dockerfile, line by line
Dockerfile, line by line# ==============================================================================
# STAGE 1: BUILD
# This stage has Maven, the full JDK and the source code. None of that
# reaches the final image: it is used only to produce the jar.
# ==============================================================================
FROM eclipse-temurin:21-jdk-alpine AS builder
WORKDIR /build
# Copy ONLY the dependency files first.
# Docker caches every instruction: as long as the POMs do not change, the
# dependency download (the slowest step, 2-3 minutes) is skipped entirely.
COPY .mvn/ .mvn/
COPY mvnw pom.xml ./
COPY bibliotech-domain/pom.xml bibliotech-domain/
COPY bibliotech-application/pom.xml bibliotech-application/
COPY bibliotech-infrastructure/pom.xml bibliotech-infrastructure/
COPY bibliotech-web/pom.xml bibliotech-web/
COPY bibliotech-console/pom.xml bibliotech-console/
# go-offline downloads every dependency without compiling anything.
# --mount=type=cache keeps ~/.m2 between builds (BuildKit).
RUN --mount=type=cache,target=/root/.m2 \
./mvnw -B dependency:go-offline -DskipTests
# NOW we copy the source code. If only the code changes,
# Docker reuses the previous layer and downloads nothing again.
COPY bibliotech-domain/src bibliotech-domain/src
COPY bibliotech-application/src bibliotech-application/src
COPY bibliotech-infrastructure/src bibliotech-infrastructure/src
COPY bibliotech-web/src bibliotech-web/src
# -DskipTests: the tests already ran in CI (12-05). Repeating them here
# doubles the time and would need Docker inside Docker for Testcontainers.
RUN --mount=type=cache,target=/root/.m2 \
./mvnw -B -pl bibliotech-web -am clean package -DskipTests
# Extract the jar into layers
RUN java -Djarmode=tools -jar bibliotech-web/target/bibliotech-web-*.jar \
extract --layers --launcher --destination extracted
# ==============================================================================
# STAGE 2: RUNTIME
# Minimal image: JRE only (not JDK), no Maven, no source code,
# no build tools. Less surface, fewer vulnerabilities.
# ==============================================================================
FROM eclipse-temurin:21-jre-alpine AS runtime
# Standard OCI labels: metadata that ecosystem tools read
LABEL org.opencontainers.image.title="BiblioTech" \
org.opencontainers.image.description="Nexus Software technical library" \
org.opencontainers.image.vendor="Nexus Software" \
org.opencontainers.image.licenses="Proprietary"
# Minimal utilities:
# - curl for the HEALTHCHECK
# - tzdata so time zones work (Alpine does not ship them)
# - dumb-init as PID 1: forwards signals properly to the Java process
RUN apk add --no-cache curl tzdata dumb-init && \
rm -rf /var/cache/apk/*
ENV TZ=Europe/Madrid
# ---- NON-root user ----
# If an attacker gets code execution, they must not have root inside the
# container. It is the cheapest and most effective mitigation there is.
RUN addgroup -S -g 1001 bibliotech && \
adduser -S -u 1001 -G bibliotech -h /app bibliotech
WORKDIR /app
# ---- The layers, in order of stability (least changing first) ----
# Every COPY is a Docker layer. When only the code changes, only
# the last layer (~1 MB) is rebuilt and transferred.
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/dependencies/ ./
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/spring-boot-loader/ ./
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/snapshot-dependencies/ ./
COPY --from=builder --chown=bibliotech:bibliotech /build/extracted/application/ ./
USER bibliotech
EXPOSE 8080
# ---- JVM options ----
# MaxRAMPercentage=75 uses 75% of the container limit for the heap
# InitialRAMPercentage=50 starts at half: fewer resizes
# UseG1GC balanced GC; for <2 vCPU consider SerialGC
# ExitOnOutOfMemoryError on OOM, die: let the orchestrator restart
# (a JVM in OOM serves half-broken requests, which is worse)
# HeapDumpOnOutOfMemoryError dump for diagnosis (10-07)
# file.encoding=UTF-8 explicit: we do not depend on the environment
ENV JAVA_OPTS="\
-XX:MaxRAMPercentage=75.0 \
-XX:InitialRAMPercentage=50.0 \
-XX:+UseG1GC \
-XX:+ExitOnOutOfMemoryError \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/tmp/heapdump.hprof \
-Djava.security.egd=file:/dev/./urandom \
-Dfile.encoding=UTF-8"
# Container-level health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=45s --retries=3 \
CMD curl -fsS http://localhost:8080/actuator/health/readiness || exit 1
# dumb-init as PID 1 forwards SIGTERM to the Java process: without this,
# graceful shutdown (section 18) does not work.
ENTRYPOINT ["dumb-init", "--"]
# "shell" form so that $JAVA_OPTS is expanded. exec makes Java the
# direct child process so that it receives the signals.
CMD exec java $JAVA_OPTS org.springframework.boot.loader.launch.JarLauncherBuilding and running:
docker build -t bibliotech:1.4.2 -t bibliotech:latest .
docker run -d --name bibliotech \
-p 8080:8080 \
--memory=768m --cpus=1.5 \
-e SPRING_PROFILES_ACTIVE=prod \
-e BIBLIOTECH_DB_URL=jdbc:postgresql://db:5432/bibliotech \
-e BIBLIOTECH_DB_USER=bibliotech \
-e BIBLIOTECH_DB_PASSWORD="$DB_PASSWORD" \
bibliotech:1.4.2
docker logs -f bibliotech
docker exec bibliotech curl -s localhost:8080/actuator/healthThe five points of the Dockerfile that matter most, in case you only remember five:
- Multi-stage: the final image contains neither Maven nor the JDK nor the source code. It goes from ~700 MB to ~180 MB, and removes a complete compiler from the attack surface.
- POMs before the code: the dependency cache survives between builds.
- Non-root user: a basic and mandatory mitigation.
- Layers ordered by stability: 1 MB deployments instead of 60 MB.
dumb-init+exec: without them,SIGTERMnever reaches the JVM and graceful shutdown does not happen.
.dockerignore
.dockerignoreWithout it, the build context includes the whole .git directory, the target/ folders and possibly files with secrets, which end up inside the image or at the very least get sent to the Docker daemon.
# Everything that is not needed to build
.git/
.github/
.idea/
.vscode/
*.iml
target/
**/target/
*.md
docs/
LICENSE
# CRITICAL: none of this must enter the build context
.env
*.env
**/application-local.yml
*.pem
*.p12
*.jks
secrets/
Dockerfile
.dockerignore
compose.yamlChecking the context size:
docker build --no-cache --progress=plain . 2>&1 | head -5
# => transferring context: 1.24MB (good)
# Without .dockerignore it would typically be 250 MB.
- Alternatives without a
Dockerfile: Buildpacks and Jib
Dockerfile: Buildpacks and JibCloud Native Buildpacks — built into Spring Boot, without writing a single Dockerfile:
./mvnw -pl bibliotech-web spring-boot:build-image \
-Dspring-boot.build-image.imageName=bibliotech:1.4.2<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<image>
<name>registry.nexussoftware.com/bibliotech:${project.version}</name>
<env>
<BP_JVM_VERSION>21</BP_JVM_VERSION>
<BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
<BPE_APPEND_JAVA_TOOL_OPTIONS>-XX:MaxRAMPercentage=75</BPE_APPEND_JAVA_TOOL_OPTIONS>
</env>
</image>
</configuration>
</plugin>Buildpacks detects that it is a Java application, picks the JVM, applies layers, configures the memory automatically from the container limit and adds an SBOM (a component inventory, useful for security, 12-07).
Jib (Google) builds the image without needing a Docker daemon, which makes it ideal for CI:
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>3.4.3</version>
<configuration>
<from><image>eclipse-temurin:21-jre-alpine</image></from>
<to><image>registry.nexussoftware.com/bibliotech:${project.version}</image></to>
<container>
<user>1001:1001</user>
<ports><port>8080</port></ports>
<jvmFlags>
<jvmFlag>-XX:MaxRAMPercentage=75.0</jvmFlag>
</jvmFlags>
</container>
</configuration>
</plugin>./mvnw -pl bibliotech-web jib:build # pushes straight to the registry
./mvnw -pl bibliotech-web jib:dockerBuild # or builds into the local DockerComparison:
| Criterion | Dockerfile |
Buildpacks | Jib |
|---|---|---|---|
| Full control | Yes | No | Partial |
| Needs Docker to build | Yes | Yes | No |
| Optimised layers | Manual | Automatic | Automatic |
| Updating the base | Manual | Automatic (rebase) | Change one line |
| Speed | Medium | Slow the 1st time | Very fast |
| Learning curve | Medium | Low | Low |
| When | You need fine control | You want to forget about it | CI without Docker |
Recommendation for BiblioTech: an explicit Dockerfile. On a course, and in a team that wants to understand its own deployment, control and transparency are worth more than convenience. In a large team with many services, Buildpacks or Jib save repeated work.
- Choosing a base image and its size
| Base image | Size (with JRE 21) | Characteristics |
|---|---|---|
eclipse-temurin:21-jdk |
~450 MB | Full JDK. Do not use in production |
eclipse-temurin:21-jre |
~270 MB | JRE on Ubuntu. Compatible and predictable |
eclipse-temurin:21-jre-alpine |
~180 MB | Alpine + musl libc. Lightweight |
gcr.io/distroless/java21 |
~190 MB | No shell, no package manager. Very secure |
Custom image with jlink |
~90 MB | JRE trimmed to the required modules |
Real-world considerations:
- Alpine uses musl instead of glibc. 99% of Java code behaves identically, but libraries with native code can fail. If odd library-loading errors appear, try the non-Alpine variant before losing an afternoon.
- Distroless is the most secure: with no shell, an attacker who gets code execution cannot run commands. The price is that you cannot either: there is no
docker exec ... shfor diagnosis. It requires good observability (12-07). - Size matters less than it seems. The base layer is downloaded once and shared by every image on that base. What travels on each deployment is the application layer (~1 MB).
Trimming with jlink, for anyone who needs the bare minimum:
FROM eclipse-temurin:21-jdk-alpine AS jre-minimal
RUN jlink \
--add-modules java.base,java.logging,java.sql,java.naming,java.management,\
java.instrument,java.security.jgss,java.desktop,jdk.unsupported,jdk.crypto.ec \
--strip-debug --no-man-pages --no-header-files --compress=2 \
--output /jre-minimal
- The JVM inside a container
This is by far the most common deployment mistake with Java, and it deserves a section of its own.
The historical problem. Before Java 10, the JVM read the memory of the host machine, ignoring the container's limit. On a 64 GB server with a container limited to 512 MB, the JVM computed a maximum heap of 16 GB (a quarter of 64), tried to use it, and the kernel killed the process with OOMKilled (code 137) without a single message from the JVM.
Since Java 10 — and refined in 11, 15 and 17 — the JVM is cgroup-aware:
$ docker run --memory=512m eclipse-temurin:21-jre java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
size_t MaxHeapSize = 134217728 # 128 MB = 25% of 512 MBThe options to tune and why:
| Option | Recommended value | Reason |
|---|---|---|
-XX:MaxRAMPercentage |
75.0 |
The default 25% wastes memory |
-XX:InitialRAMPercentage |
50.0 |
Fewer heap resizes at startup |
-XX:+UseG1GC |
With ≥ 2 vCPU | A balance between pauses and throughput |
-XX:+UseSerialGC |
With < 2 vCPU | Less overhead in small containers |
-XX:MaxMetaspaceSize |
256m |
Metaspace is not inside the heap |
-XX:+ExitOnOutOfMemoryError |
Always | Die fast and let the orchestrator restart |
-XX:ActiveProcessorCount |
If the CPU limit is fractional | The JVM rounds fractional CPUs badly |
A calculation that prevents many incidents. With a container limit of 512 MB:
| Component | Memory |
|---|---|
| Heap (75%) | 384 MB |
| Metaspace | ~60 MB |
| Thread stacks (200 × 1 MB reserved) | ~30 MB actual |
| Code cache (JIT) | ~40 MB |
| GC and internal structures | ~30 MB |
| NIO direct buffers | ~20 MB |
| Total | ~564 MB > 512 MB → OOMKilled |
The JVM's memory is not only the heap. This calculation is the reason so many Java containers die with no apparent explanation. To diagnose it, NativeMemoryTracking (10-07):
docker run -e JAVA_OPTS="-XX:NativeMemoryTracking=summary" bibliotech:1.4.2
docker exec bibliotech jcmd 1 VM.native_memory summaryRules of thumb:
- With
MaxRAMPercentage=75, leave at least 256 MB of container limit above what the heap needs. - A typical Spring Boot service needs at least 512 MB; 768 MB is more comfortable.
- If you see
OOMKilled(code 137), it is not an application bug: it is memory outside the heap.
docker-compose for the local environment
docker-compose for the local environmentIn 12-01 we only started PostgreSQL. Now, the complete application:
# compose.yaml
name: bibliotech
services:
db:
image: postgres:16-alpine
container_name: bibliotech-db
environment:
POSTGRES_DB: bibliotech
POSTGRES_USER: bibliotech
POSTGRES_PASSWORD: ${DB_PASSWORD:-bibliotech}
ports:
- "5432:5432"
volumes:
- db-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U bibliotech -d bibliotech"]
interval: 5s
timeout: 3s
retries: 10
start_period: 10s
app:
build:
context: .
dockerfile: Dockerfile
image: bibliotech:${VERSION:-dev}
container_name: bibliotech-app
depends_on:
db:
condition: service_healthy # waits for the healthcheck, not just the start
environment:
SPRING_PROFILES_ACTIVE: docker
BIBLIOTECH_DB_URL: jdbc:postgresql://db:5432/bibliotech
BIBLIOTECH_DB_USER: bibliotech
BIBLIOTECH_DB_PASSWORD: ${DB_PASSWORD:-bibliotech}
JAVA_OPTS: "-XX:MaxRAMPercentage=75 -XX:+UseG1GC"
ports:
- "8080:8080"
deploy:
resources:
limits:
memory: 768M
cpus: '1.5'
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:8080/actuator/health/readiness"]
interval: 15s
timeout: 3s
retries: 5
start_period: 60s
restart: unless-stopped
volumes:
db-data:docker compose up -d --build # build and start
docker compose logs -f app # follow the logs
docker compose ps # status and health
docker compose exec db psql -U bibliotech # get into the database
docker compose down # stop (the data survives)
docker compose down -v # stop and DELETE the datadepends_on with condition: service_healthy is the detail that prevents the most frequent failure in development: the application starts before PostgreSQL accepts connections and dies on the first attempt.
- Configuration and secrets at deployment time
The rule that picks up 12-01 and governs everything:
One image for every environment. The same
bibliotech:1.4.2image goes to development, staging and production. Only the environment variables change.
If you build a different image for production, what you tested in staging is not what you deploy.
Mechanisms, in order of robustness:
| Mechanism | How | Advantages | Risks |
|---|---|---|---|
| Environment variables | -e KEY=value |
De facto standard, simple | Visible in docker inspect and in the process environment |
| Mounted files | -v /secrets:/app/config:ro |
Do not appear in the environment | Permissions have to be managed |
| Orchestrator secrets | Kubernetes Secret, Docker Secret | Integrated into the platform | Base64 is not encryption |
| Secret manager | Vault, AWS Secrets Manager | Rotation, auditing, encryption | More operational complexity |
External configuration by file, useful on your own servers:
Spring Boot looks for application.yml and application-{profile}.yml in that path, with higher precedence than the packaged ones (12-01).
In Kubernetes, the separation between configuration and secrets:
apiVersion: v1
kind: ConfigMap
metadata:
name: bibliotech-config
data:
SPRING_PROFILES_ACTIVE: "prod"
BIBLIOTECH_LOAN_DEFAULTDAYS: "15"
LOGGING_LEVEL_COM_NEXUSSOFTWARE_BIBLIOTECH: "INFO"
---
apiVersion: v1
kind: Secret
metadata:
name: bibliotech-secrets
type: Opaque
stringData:
BIBLIOTECH_DB_PASSWORD: "…" # managed with Sealed Secrets or an external manager
METADATA_API_KEY: "…"Warning. A Kubernetes
Secretis encoded in base64, which is not encryption: anybody with read access to the namespace can decode it. Real secrets need Sealed Secrets, the External Secrets Operator or an external manager, plus encryption at rest in etcd. Picked up again in 12-07.
Checking that the configuration is what you expect:
curl -s localhost:8080/actuator/env/bibliotech.fine.euros-per-day | jq
# shows the effective value AND which source it came from
- Database migrations with Flyway
Here a debt from 11-03 gets paid: ddl-auto is no good in production.
| Value | What it does | Production |
|---|---|---|
create-drop |
Drops and recreates at startup | Catastrophic |
create |
Drops and recreates | Catastrophic |
update |
Adds what is missing | No. See below |
validate |
Checks that the schema matches | Yes |
none |
Does nothing | Yes |
Why update is no good, precisely:
- It removes nothing. Columns and tables deleted from the model stay forever.
- It does not rename. Changing
due_dttodue_datecreates a new column and leaves the data in the old one. - It does not version. There is no way to know what schema an environment has, or to reproduce it.
- It is not reversible. There is no way back.
- It depends on startup order. With several instances starting at once, they can issue DDL simultaneously and block each other.
- It does not migrate data. Splitting
full_nameintofirst_nameandsurnameis impossible.
Flyway solves all of that with versioned SQL scripts applied in order, exactly once, recorded in a control table.
<dependency>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-core</artifactId>
</dependency>
<dependency>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-database-postgresql</artifactId>
</dependency>spring:
jpa:
hibernate:
ddl-auto: validate # Hibernate VERIFIES, Flyway DECIDES
flyway:
enabled: true
locations: classpath:db/migration
baseline-on-migrate: true # for a database that already existed
validate-on-migrate: true # detects already-applied scripts that have changed
out-of-order: false # forbids applying a version older than the last oneNaming convention: V<version>__<description>.sql
bibliotech-infrastructure/src/main/resources/db/migration/ ├── V1__initial_schema.sql ├── V2__catalog_indexes.sql ├── V3__add_material_value_column.sql ├── V4__reservations_table.sql ├── V5__return_condition_and_surcharges.sql └── R__usage_statistics_view.sql ← R = repeatable: re-run when it changes
-- V1__initial_schema.sql
CREATE TABLE materials (
id BIGSERIAL PRIMARY KEY,
type VARCHAR(20) NOT NULL, -- SINGLE_TABLE discriminator (ADR-004)
isbn VARCHAR(17) NOT NULL UNIQUE,
title VARCHAR(200) NOT NULL,
author VARCHAR(150),
publication_year INTEGER,
total_copies INTEGER NOT NULL DEFAULT 1 CHECK (total_copies >= 0),
available_copies INTEGER NOT NULL DEFAULT 1 CHECK (available_copies >= 0),
version BIGINT NOT NULL DEFAULT 0, -- @Version (11-03)
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT chk_available_not_above_total
CHECK (available_copies <= total_copies)
);
CREATE TABLE employees (
id BIGSERIAL PRIMARY KEY,
name VARCHAR(150) NOT NULL,
email VARCHAR(200) NOT NULL UNIQUE,
department VARCHAR(100),
start_date DATE NOT NULL,
version BIGINT NOT NULL DEFAULT 0
);
CREATE TABLE loans (
id BIGSERIAL PRIMARY KEY,
material_id BIGINT NOT NULL REFERENCES materials(id),
employee_id BIGINT NOT NULL REFERENCES employees(id),
loan_date DATE NOT NULL,
due_date DATE NOT NULL,
return_date DATE,
status VARCHAR(20) NOT NULL,
version BIGINT NOT NULL DEFAULT 0,
CONSTRAINT chk_due_after_loan CHECK (due_date >= loan_date),
CONSTRAINT chk_return_after_loan
CHECK (return_date IS NULL OR return_date >= loan_date)
);
-- Indexes for the queries that are actually made
CREATE INDEX idx_loans_employee_status ON loans(employee_id, status);
CREATE INDEX idx_loans_due_date ON loans(due_date)
WHERE return_date IS NULL; -- partial
CREATE INDEX idx_materials_title ON materials USING gin(to_tsvector('english', title));
-- Reference data the application needs in order to start
INSERT INTO employees (name, email, department, start_date) VALUES
('Marta Ruiz', '[email protected]', 'Architecture', '2024-03-01'),
('Diego Alonso','[email protected]','Backend', '2025-01-15'),
('Nuria Vidal', '[email protected]', 'Platform', '2023-09-10');Golden rules for migrations:
| Rule | Reason |
|---|---|
| An applied script is NEVER modified | Flyway stores a checksum; if it changes, startup fails |
| To fix something, a new script | It is the only way for every environment to converge |
| Idempotent migrations where possible | CREATE TABLE IF NOT EXISTS |
| Destructive changes, in two phases | See the next section |
| Test the migration with real data | An ALTER TABLE over 10 million rows can take hours and block |
A blocking ALTER, in a maintenance window |
In PostgreSQL, ADD COLUMN with a default is fast; ALTER TYPE rewrites the table |
Useful commands:
./mvnw flyway:info # which migrations exist and which are applied
./mvnw flyway:validate # checks the checksums
./mvnw flyway:migrate # applies the pending ones
./mvnw flyway:repair # fixes the control table (carefully!)And a test that prevents surprises, wired into Testcontainers (12-05):
@Test
void allMigrationsApplyOnACleanPostgres() {
Flyway flyway = Flyway.configure()
.dataSource(POSTGRES.getJdbcUrl(), POSTGRES.getUsername(), POSTGRES.getPassword())
.locations("classpath:db/migration")
.load();
MigrateResult result = flyway.migrate();
assertThat(result.success).isTrue();
assertThat(result.migrationsExecuted).isGreaterThan(0);
// And that the resulting schema matches what Hibernate expects:
assertThatNoException().isThrownBy(() -> validateSchemaAgainstEntities());
}
- Two-phase deployment for schema changes
The problem: during a rolling update the old and the new version of the application coexist against the same database. A destructive change breaks the old version before it has finished retiring.
Example: renaming loans.due_dt to loans.due_date.
What you CANNOT do:
The instant it is applied, every instance of the old version — still serving requests — fails with "column does not exist".
The solution, in three deployments:
flowchart TD
F1["PHASE 1 · Expand<br/>Add due_date<br/>Copy data + sync trigger<br/>App v1 uses the old one; both columns coexist"]
F2["PHASE 2 · Migrate<br/>App v2 writes and reads the new one<br/>The trigger keeps the old one up to date<br/>Rollback to v1 still possible"]
F3["PHASE 3 · Contract<br/>Drop the trigger and the old column<br/>Only once v1 no longer exists"]
F1 --> F2 --> F3
-- PHASE 1: V6__add_due_date.sql (compatible with app v1)
ALTER TABLE loans ADD COLUMN due_date DATE;
UPDATE loans SET due_date = due_dt WHERE due_date IS NULL;
-- Dual-write trigger: it does not matter which app version writes
CREATE OR REPLACE FUNCTION sync_due_date() RETURNS TRIGGER AS $$
BEGIN
IF NEW.due_date IS DISTINCT FROM OLD.due_date THEN
NEW.due_dt := NEW.due_date;
ELSIF NEW.due_dt IS DISTINCT FROM OLD.due_dt THEN
NEW.due_date := NEW.due_dt;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_sync_due_date
BEFORE INSERT OR UPDATE ON loans
FOR EACH ROW EXECUTE FUNCTION sync_due_date();-- PHASE 3: V8__drop_due_dt.sql (only after confirming v1 is no longer running)
DROP TRIGGER IF EXISTS trg_sync_due_date ON loans;
DROP FUNCTION IF EXISTS sync_due_date();
ALTER TABLE loans DROP COLUMN due_dt;
ALTER TABLE loans ALTER COLUMN due_date SET NOT NULL;This pattern is called expand and contract, and the rule that sums it up is easy to remember:
Every migration must be compatible with the previous version of the application. Destructive changes are applied at least one deployment after the one that stopped needing what is being removed.
- Where it gets deployed: platform comparison
| Platform | How it works | Control | Complexity | Cost | When |
|---|---|---|---|---|---|
| Own server + systemd | jar as a Linux service | Total | Low | Low | 1-2 services, small team |
| PaaS (Heroku, Render, Railway, Fly.io) | You push code or an image | Low | Very low | Medium-high | Prototypes, teams with no ops |
| Managed containers (ECS, Cloud Run, App Service) | You deploy images | Medium | Medium | Medium | Most applications |
| Kubernetes | A full orchestrator | Total | High | Variable | Many services, real scale |
Your own server with systemd, which is still perfectly valid and often the best option:
# /etc/systemd/system/bibliotech.service
[Unit]
Description=BiblioTech - Nexus Software technical library
After=network.target postgresql.service
Wants=postgresql.service
[Service]
Type=simple
User=bibliotech
Group=bibliotech
WorkingDirectory=/opt/bibliotech
EnvironmentFile=/etc/bibliotech/environment # secrets, with 600 permissions
ExecStart=/usr/bin/java $JAVA_OPTS -jar /opt/bibliotech/bibliotech-web.jar
SuccessExitStatus=143 # 128 + SIGTERM: graceful shutdown, not a failure
Restart=on-failure
RestartSec=10
# Graceful shutdown: SIGTERM, and 60 s before SIGKILL
KillSignal=SIGTERM
TimeoutStopSec=60
# Hardening: least privilege (12-07)
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/bibliotech /var/lib/bibliotech
StandardOutput=journal
StandardError=journal
SyslogIdentifier=bibliotech
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now bibliotech
sudo systemctl status bibliotech
sudo journalctl -u bibliotech -fSelection criteria, bluntly:
| If… | Choose |
|---|---|
| You have 1-3 services and a small team | Your own server or managed containers |
| You do not want to manage infrastructure | PaaS or Cloud Run |
| You need to scale to zero when there is no traffic | Cloud Run, Fly.io |
| You have 20+ services and a platform team | Kubernetes |
| You have 3 services and no platform team | Not Kubernetes |
On that last point it is worth being explicit: Kubernetes is an excellent tool with a real and permanent operational cost. Adopting it for three services because "it is what everybody uses" is a decision you pay for every week in the team's time.
- Kubernetes: a minimal
Deployment
DeploymentapiVersion: apps/v1
kind: Deployment
metadata:
name: bibliotech
labels:
app: bibliotech
spec:
replicas: 3 # three instances: high availability
revisionHistoryLimit: 5 # history for rollback
strategy:
type: RollingUpdate # rolling update (section 20)
rollingUpdate:
maxSurge: 1 # at most 1 extra pod during the transition
maxUnavailable: 0 # NEVER fewer than 3 available: no service interruption
selector:
matchLabels:
app: bibliotech
template:
metadata:
labels:
app: bibliotech
version: "1.4.2"
spec:
# Room for the graceful shutdown to finish (section 18)
terminationGracePeriodSeconds: 60
securityContext: # least privilege at pod level
runAsNonRoot: true
runAsUser: 1001
fsGroup: 1001
containers:
- name: bibliotech
image: registry.nexussoftware.com/bibliotech:1.4.2 # EXACT tag, never 'latest'
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
envFrom:
- configMapRef: { name: bibliotech-config }
- secretRef: { name: bibliotech-secrets }
resources:
requests: # what the scheduler reserves
memory: "512Mi"
cpu: "250m"
limits: # the ceiling; exceeding it on memory = OOMKilled
memory: "768Mi"
cpu: "1500m"
# --- Probes (section 17) ---
startupProbe: # protects startup: up to 100 s
httpGet: { path: /actuator/health/liveness, port: http }
failureThreshold: 20
periodSeconds: 5
livenessProbe: # still alive? If not, RESTART
httpGet: { path: /actuator/health/liveness, port: http }
periodSeconds: 10
failureThreshold: 3
readinessProbe: # can it serve? If not, TAKE IT OUT OF THE BALANCER
httpGet: { path: /actuator/health/readiness, port: http }
periodSeconds: 5
failureThreshold: 2
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true # nothing writes to the root
capabilities: { drop: ["ALL"] }
volumeMounts:
- name: tmp
mountPath: /tmp # required: Tomcat and the dumps write here
volumes:
- name: tmp
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: bibliotech
spec:
selector:
app: bibliotech
ports:
- port: 80
targetPort: http
type: ClusterIP
---
apiVersion: policy/v1
kind: PodDisruptionBudget # protects during cluster maintenance
metadata:
name: bibliotech
spec:
minAvailable: 2
selector:
matchLabels:
app: bibliotechkubectl apply -f k8s/
kubectl rollout status deployment/bibliotech
kubectl get pods -l app=bibliotech
kubectl logs -f -l app=bibliotech --tail=100
kubectl rollout undo deployment/bibliotech # immediate rollbackTwo details that deserve special attention:
maxUnavailable: 0guarantees that during the update there are never fewer instances than declared. That is what turns a deployment into something invisible to users.image: bibliotech:1.4.2, never:latest. Withlatestyou do not know what is running, rollback does not work and two pods can end up on different versions.
- Startup and health: Actuator probes
Spring Boot Actuator distinguishes two questions that look the same and are not:
| Probe | Question | If it fails |
|---|---|---|
| Liveness | Is the process alive and not stuck? | Restart the container |
| Readiness | Can it serve requests right now? | Take it out of the balancer, without restarting |
The difference is critical. If the database goes down, the application is not ready (readiness fails) but it is alive (liveness passes). Restarting it would fix nothing, and restarting every instance at once because the database hiccupped is an excellent way to turn a minor incident into a total outage.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus # ONLY what is needed (12-07)
base-path: /actuator
endpoint:
health:
probes:
enabled: true # enables /health/liveness and /health/readiness
show-details: when-authorized # the detail, only to whoever is allowed
group:
readiness:
include: db, diskSpace # if the DB does not answer, we are not ready
liveness:
include: livenessState # only the process state
health:
livenessstate:
enabled: true
readinessstate:
enabled: true$ curl localhost:8080/actuator/health/liveness
{"status":"UP"}
$ curl localhost:8080/actuator/health/readiness
{"status":"UP","components":{"db":{"status":"UP"},"diskSpace":{"status":"UP"}}}A health indicator of your own, for what only you know is critical:
@Component
public class MetadataGatewayHealth implements HealthIndicator {
private final MetadataGateway gateway;
@Override
public Health health() {
try {
boolean available = gateway.checkAvailability();
return available
? Health.up().withDetail("gateway", "available").build()
// DEGRADED, not DOWN: the application works without external metadata.
// Marking DOWN here would pull a perfectly usable app out of the balancer.
: Health.status("DEGRADED").withDetail("gateway", "not responding").build();
} catch (Exception e) {
return Health.status("DEGRADED").withException(e).build();
}
}
}That nuance — degraded rather than down for non-essential dependencies — is what separates a resilient system from one that falls over entirely because a secondary service had a problem.
- Graceful shutdown
When the orchestrator wants to stop an instance, it sends SIGTERM. With no preparation, the JVM dies immediately and the in-flight requests are cut: users with errors, half-finished transactions, unacknowledged messages.
server:
shutdown: graceful # stops accepting new requests and waits for the current ones
spring:
lifecycle:
timeout-per-shutdown-phase: 30sThe full sequence:
sequenceDiagram
participant O as Orchestrator
participant K as Kubelet / Docker
participant A as BiblioTech
participant B as Load balancer
O->>K: stop the pod
K->>B: remove it from the endpoints
K->>A: SIGTERM
A->>A: readiness = DOWN
A->>A: stop accepting new requests
A->>A: finish the 12 in-flight requests
A->>A: close the connection pool
A->>A: run the shutdown hooks
A-->>K: process finished (code 143)
Note over K,A: if it is still alive after terminationGracePeriodSeconds, SIGKILL
And the shutdown hooks, which pick up module 7 and 12-03:
@Component
public class GracefulShutdown {
private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);
private final ExecutorService importExecutor;
/**
* @PreDestroy runs when the Spring context closes,
* which is what happens on receiving SIGTERM.
*/
@PreDestroy
public void onShutdown() {
log.info("Graceful shutdown: closing resources");
importExecutor.shutdown(); // accepts no new tasks
try {
if (!importExecutor.awaitTermination(20, TimeUnit.SECONDS)) {
log.warn("Imports unfinished after 20 s; forcing");
importExecutor.shutdownNow();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
importExecutor.shutdownNow();
}
log.info("Graceful shutdown complete");
}
}A critical and little-known detail: there is a race window between the moment the orchestrator sends SIGTERM and the moment the load balancer stops sending traffic. During that gap — up to a few seconds — requests arrive at an instance that is already shutting down. The standard solution is a wait before the shutdown begins:
And in the Dockerfile, dumb-init as PID 1: without it, the Java process (which would be PID 1) does not receive signals by default and none of the above is worth anything.
- Startup time: CDS and Native Image
Startup matters in three situations: autoscaling on a traffic spike, restarting after a failure, and serverless functions that scale to zero.
| Technique | BiblioTech startup | Cost |
|---|---|---|
| Ordinary jar | ~4.5 s | — |
| CDS (class data sharing archive) | ~3.2 s | An extra build step |
Spring AOT (-Dspring.aot.enabled) |
~2.8 s | Limits dynamic configuration |
| CRaC (restore from a checkpoint) | ~0.3 s | Specific JVM, high complexity |
| GraalVM Native Image | ~0.08 s | 5-10 min build; reflection must be declared |
CDS is the cheapest improvement: it memorises the result of loading and verifying the classes.
# Generate the CDS archive during the image build
RUN java -XX:ArchiveClassesAtExit=/app/app.jsa \
-Dspring.context.exit=onRefresh \
org.springframework.boot.loader.launch.JarLauncher
ENV JAVA_OPTS="$JAVA_OPTS -XX:SharedArchiveFile=/app/app.jsa"GraalVM Native Image compiles to a native executable:
<profile>
<id>native</id>
<build>
<plugins>
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</profile>./mvnw -Pnative native:compile -pl bibliotech-web
./bibliotech-web/target/bibliotech-web # starts in 80 ms| Aspect | JVM | Native Image |
|---|---|---|
| Startup | 4.5 s | 0.08 s |
| Memory at rest | ~350 MB | ~90 MB |
| Peak throughput | Higher (JIT optimises with real data) | Lower |
| Build time | 30 s | 5-10 min |
| Dynamic reflection | Free | Must be declared |
| Diagnostic tooling | JFR, jcmd, JMX | Limited |
When Native Image pays off: serverless functions, CLIs (12-03), microservices that scale to zero, environments with very tight memory. When it does not: long-lived services under sustained load, where the JIT ends up producing faster code than ahead-of-time compilation.
For BiblioTech as a long-lived API: JVM with CDS. For the CLI: Native Image makes a lot of sense.
- Deployment strategies
| Strategy | How it works | Downtime | Cost | Rollback | When |
|---|---|---|---|---|---|
| Recreate | Stop everything, start the new one | Yes | Low | Redeploy | Development; apps that cannot tolerate two versions |
| Rolling | Replace instance by instance | No | Low | Roll forward in reverse | The default |
| Blue-green | Two complete environments; switch the traffic | No | Double | Instant | Risky changes |
| Canary | Send 5% of the traffic to the new one; ramp up | No | Medium | Instant | High-risk changes, high traffic |
flowchart TD
subgraph BLUE_GREEN["Blue-green"]
LB1["Load balancer"] -->|"100%"| AZ["BLUE v1.4.1<br/>(in production)"]
LB1 -.->|"0%"| VE["GREEN v1.4.2<br/>(deployed, tested)"]
N1["Switch: traffic moves to GREEN in an instant.<br/>BLUE stays switched on in case you need to go back."]
end
subgraph CANARY["Canary"]
LB2["Load balancer"] -->|"95%"| E1["v1.4.1 (9 instances)"]
LB2 -->|"5%"| E2["v1.4.2 (1 instance)"]
N2["Watch errors and latency.<br/>If all is well: 25%, 50%, 100%.<br/>If not: back to 0% at once."]
end
Feature flags. They are the complement that changes the game, because they separate deployment from activation:
@Service
public class LoanManager {
private final BiblioTechProperties props;
public Loan lend(Isbn isbn, Long employeeId, Integer days) {
if (props.features().autoReservation()) {
// New code, deployed but switched off until somebody decides
return lendWithAutoReservation(isbn, employeeId, days);
}
return lendClassic(isbn, employeeId, days);
}
}bibliotech:
features:
auto-reservation: false # switched on with an environment variable, no deployment
priority-loan: true
pdf-report: falseWith flags, switching off a problematic feature is a configuration change taking seconds, not a rollback deployment taking minutes. And they let you deploy incomplete code with no risk, which in turn lets you integrate daily instead of keeping long-lived branches (12-01).
The price: every flag is one more branch to test, and forgotten flags pile up. They get removed as soon as the decision is final.
- Rollback and the limits of the database
Rolling back the code is easy:
kubectl rollout undo deployment/bibliotech
docker compose up -d --force-recreate # with the previous tag
sudo systemctl stop bibliotech && cp bibliotech-1.4.1.jar bibliotech-web.jar && sudo systemctl start bibliotechRolling back the database almost never is, and this is why:
| Change | Reversible? | Why |
|---|---|---|
ADD COLUMN (nullable) |
Yes | The old version ignores it |
CREATE TABLE |
Yes | Nobody uses it |
CREATE INDEX |
Yes | It only affects performance |
ADD COLUMN NOT NULL with no default |
No | The old version does not fill it on insert |
DROP COLUMN |
No | The data is gone |
RENAME COLUMN |
No | The old version looks for the old name |
ALTER TYPE with data loss |
No | Truncated data does not come back |
| Data migration | It depends | Only if the previous state was kept |
Hence the safe deployment rule:
The database always goes first and is always backward compatible. First the compatible migration is deployed; then the code that uses it. Never the other way round, and never both in the same step if the change is destructive.
With the expand-contract pattern (section 14), rollback works in phases 1 and 2. In phase 3 it no longer does: that is why phase 3 is applied days later, once the new version is confirmed.
Backups — and the part almost nobody does:
# Daily backup
pg_dump -h db -U bibliotech -Fc bibliotech > bibliotech-$(date +%F).dump
# Backup before ANY destructive migration
pg_dump -h db -U bibliotech -Fc bibliotech > pre-migration-v8-$(date +%F-%H%M).dumpNote. A backup that has never been restored is not a backup: it is a file with hopes attached. Schedule a periodic test restore in a separate environment and measure how long it takes. That time is your real RTO (recovery time objective), and it is usually far higher than people assume. Just as important is the RPO (recovery point objective): with daily backups, you can lose up to 24 hours of data. If that is unacceptable, you need WAL archiving or replication.
- The continuous delivery pipeline
The CI from 12-05 verified. Continuous delivery also builds the image, publishes it and deploys it.
flowchart LR
A["Push to main"] --> B["CI: tests<br/>coverage, analysis"]
B --> C["Build image<br/>multi-architecture"]
C --> D["Push to the<br/>registry"]
D --> E["Deploy to<br/>staging"]
E --> F["Smoke tests"]
F --> G{"Manual<br/>approval"}
G -->|"approved"| H["Deploy to<br/>production"]
H --> I["Verify health"]
I -->|"failure"| J["Automatic<br/>rollback"]
style G fill:#fff3e0,stroke:#e65100
style J fill:#ffebee,stroke:#c62828
# .github/workflows/cd.yml
name: Continuous delivery
on:
push:
branches: [main]
tags: ['v*']
env:
REGISTRY: ghcr.io
IMAGE: ${{ github.repository }}
permissions:
contents: read
packages: write
id-token: write # to sign the image with cosign
jobs:
# ---------------------------------------------------------------
# 1. Reuses the full verification from 12-05
# ---------------------------------------------------------------
verify:
uses: ./.github/workflows/ci.yml
# ---------------------------------------------------------------
# 2. Build and publish the image
# ---------------------------------------------------------------
image:
name: Build and publish image
runs-on: ubuntu-latest
needs: verify
outputs:
digest: ${{ steps.build.outputs.digest }}
tags: ${{ steps.meta.outputs.tags }}
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3 # to build for arm64
- name: Set up Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to the registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Compute tags and metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha,prefix=,format=short # traceability to the commit
type=raw,value=latest,enable={{is_default_branch}}
- name: Build and push
id: build
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha # layer cache between runs
cache-to: type=gha,mode=max
provenance: true
sbom: true # component inventory (12-07)
- name: Scan the image for vulnerabilities
uses: aquasecurity/[email protected]
with:
image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
severity: 'CRITICAL,HIGH'
exit-code: '1' # blocks if there are serious vulnerabilities
- name: Sign the image
uses: sigstore/cosign-installer@v3
- run: cosign sign --yes ${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
# ---------------------------------------------------------------
# 3. Staging: automatic
# ---------------------------------------------------------------
staging:
name: Deploy to staging
runs-on: ubuntu-latest
needs: image
environment:
name: staging
url: https://staging.bibliotech.nexussoftware.com
steps:
- name: Deploy
run: |
kubectl set image deployment/bibliotech \
bibliotech=${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ needs.image.outputs.digest }} \
--namespace=staging
kubectl rollout status deployment/bibliotech -n staging --timeout=5m
- name: Smoke tests
run: |
BASE=https://staging.bibliotech.nexussoftware.com
curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"'
curl -fsS "$BASE/api/materials?size=1" | jq -e '.content | length >= 0'
echo "Smoke tests passed"
# ---------------------------------------------------------------
# 4. Production: requires manual approval
# ---------------------------------------------------------------
production:
name: Deploy to production
runs-on: ubuntu-latest
needs: [image, staging]
if: startsWith(github.ref, 'refs/tags/v') # only from a version tag
environment:
name: production # with required reviewers configured
url: https://bibliotech.nexussoftware.com
steps:
- name: Record the current revision, in case we have to go back
id: current
run: |
CURRENT=$(kubectl get deployment/bibliotech -n production \
-o jsonpath='{.spec.template.spec.containers[0].image}')
echo "previous_image=$CURRENT" >> $GITHUB_OUTPUT
- name: Deploy (rolling update)
run: |
kubectl set image deployment/bibliotech \
bibliotech=${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ needs.image.outputs.digest }} \
--namespace=production
kubectl rollout status deployment/bibliotech -n production --timeout=10m
- name: Verify health after the deployment
id: verify
run: |
sleep 30
BASE=https://bibliotech.nexussoftware.com
for i in $(seq 1 10); do
if curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"' > /dev/null; then
echo "Instance healthy (attempt $i)"; sleep 5
else
echo "::error::Health verification failed"; exit 1
fi
done
# Check that the error rate has not shot up
ERRORS=$(curl -fsS "$BASE/actuator/metrics/http.server.requests?tag=outcome:SERVER_ERROR" \
| jq '.measurements[0].value // 0')
if (( $(echo "$ERRORS > 10" | bc -l) )); then
echo "::error::Too many 5xx errors after the deployment"; exit 1
fi
- name: Automatic rollback if something failed
if: failure()
run: |
echo "::warning::Deployment failed; going back to the previous version"
kubectl rollout undo deployment/bibliotech -n production
kubectl rollout status deployment/bibliotech -n production --timeout=5m
- name: Report the result
if: always()
run: |
STATUS="${{ job.status }}"
curl -X POST "${{ secrets.TEAM_WEBHOOK }}" \
-H 'Content-Type: application/json' \
-d "{\"text\":\"BiblioTech deployment ${{ github.ref_name }}: $STATUS\"}"Key points of this pipeline:
| Decision | Reason |
|---|---|
| Deploy by digest, not by tag | A tag can be moved; a digest identifies exact bytes |
| Blocking vulnerability scan | Do not publish an image with critical CVEs (12-07) |
| Signing with cosign | Verifiable: this image was built by our pipeline |
| Staging automatic, production manual | Speed where there is no risk, control where there is |
environment with reviewers |
The approval belongs to the system, not to a chat message |
| Health verification + automatic rollback | A failed deployment is reverted in 2 minutes, with no humans |
Production only from a v* tag |
Every production deployment has an identifiable version |
- Horizontal scaling and what it demands of the application
Scaling vertically means giving one instance more resources; scaling horizontally means having more instances. The second is what gives high availability and scales without an upper limit, but it demands properties of the application.
Requirements, all of them verifiable:
| Requirement | Why | BiblioTech's status |
|---|---|---|
| No in-memory state | Request 2 may land on another instance | ✅ Nothing in memory since 12-01 |
| Shared session or no session | Same reason | ✅ Stateless API; JWT in 12-07 |
| No local files | Each instance has its own disk | ⚠️ Review the exports |
| Coordinated scheduled tasks | Three instances = three runs of the same @Scheduled |
⚠️ Pending |
| Distributed or coherent local cache | Local caches diverge | ⚠️ Review CatalogWithCache |
| Migrations with locking | Three instances starting at once | ✅ Flyway uses a lock |
Here the session work from module 7 comes back. Any state living in memory today — the session Map, the local cache, the CLI's undo stack — stops working with more than one instance. The answer is not "do not scale", but moving that state to a shared place: the database, Redis, or removing it by design.
The scheduled-task problem, which is the most frequent and the most surprising:
// WITH 3 INSTANCES: this sends THREE notices to every employee, every day
@Scheduled(cron = "0 0 8 * * *")
public void sendDailyNotices() {
noticeService.notifyUpcomingDueDates(3);
}Solutions, from least to most robust:
// Option A: ShedLock — a distributed lock in the database
@Scheduled(cron = "0 0 8 * * *")
@SchedulerLock(name = "dailyNotices", lockAtMostFor = "10m", lockAtLeastFor = "1m")
public void sendDailyNotices() {
noticeService.notifyUpcomingDueDates(3);
}# Option B: a Kubernetes CronJob that invokes the CLI from 12-03.
# Advantage: the scheduler does not live inside the application.
apiVersion: batch/v1
kind: CronJob
metadata:
name: bibliotech-notices
spec:
schedule: "0 8 * * *"
concurrencyPolicy: Forbid # do not overlap runs
jobTemplate:
spec:
backoffLimit: 3
template:
spec:
restartPolicy: OnFailure
containers:
- name: cli
image: registry.nexussoftware.com/bibliotech-cli:1.4.2
args: ["notices", "send", "--days-ahead=3"]
envFrom:
- secretRef: { name: bibliotech-secrets }And autoscaling, which in Kubernetes is declarative:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: bibliotech
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: bibliotech
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 70 }
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # do not scale down on the first dip: avoids flappingWith one important warning: scaling the application does not scale the database. Ten instances with 20 connections each are 200 connections to PostgreSQL, which by default accepts 100. Tune the pool or put a pgbouncer in front. The bottleneck moves; it does not disappear.
Common Mistakes and Tips
1. Using the latest tag in production. You do not know what is running, rollback does not work and two instances can end up on different versions. Semantic tags or digests.
2. Running the container as root. It is the default and it is a free vulnerability. A non-root USER, always.
3. Baking secrets into the image. They stay in the layer history; docker history reveals them. Environment variables or secret managers.
4. Not limiting the container's memory. A Java process can eat all the host's memory and take the neighbours down.
5. Limiting memory without tuning MaxRAMPercentage. The default 25% wastes resources, and not accounting for memory outside the heap causes OOMKilled with no message from the JVM.
6. ddl-auto: update in production. It does not delete, does not rename, does not version, is not reversible and fails with several instances starting at once. Flyway.
7. Modifying an already-applied migration. Flyway detects the checksum change and the application does not start. To fix something, a new script.
8. Not testing the rollback. It is exactly what you need on the day everything goes wrong. Test it in staging, timed.
9. Badly configured probes. Confusing liveness with readiness makes the application restart in a loop when the database has a transient problem, turning a minor incident into a total outage.
10. Forgetting dumb-init or exec. Without them, SIGTERM never reaches the JVM and graceful shutdown does not happen: cut requests on every deployment.
11. Uncoordinated scheduled tasks. With three instances, three emails to every employee. ShedLock or an external CronJob.
12. Backups that have never been restored. They are not backups. Test the restore and time it.
A final tip: the best measure of a deployment's quality is how long it takes to roll back. If it is thirty seconds, you will deploy often and calmly. If it is two hours, you will deploy rarely, in large batches and in fear — which is precisely what makes deployments go wrong.
Exercises
Exercise 1: a Dockerfile for the CLI
Write the multi-stage Dockerfile for the bibliotech-console module (12-03), bearing in mind that:
- The CLI runs and exits: it is not a long-lived service.
- It must start as fast as possible (it is invoked from
cron). - It needs no exposed port and no health check.
- It must accept arguments:
docker run bibliotech-cli catalog list --format=json. - It must be usable as the image of a Kubernetes
CronJob. - Tune the JVM options for startup, not for sustained throughput.
Include as well the Kubernetes CronJob that sends the daily notices.
Exercise 2: a two-phase migration
BiblioTech has to split the employees.name field (which today holds "Marta Ruiz") into first_name and surname, with no downtime and with rollback possible at every moment.
Write:
- The Flyway scripts for the three phases.
- Which application version accompanies each phase and what its code does.
- The deployment plan with the points where rollback is possible and where it stops being possible.
- A test that verifies the migration is correct with real data, including the hard cases (a single name, compound surnames, names with particles).
Exercise 3: a blue-green pipeline
Write the GitHub Actions workflow that deploys BiblioTech with a blue-green strategy on Kubernetes:
- Determine which colour is currently active.
- Deploy the new version to the idle colour.
- Run smoke tests against the idle colour, with no real traffic.
- Switch the traffic by changing the
Serviceselector. - Watch for five minutes and roll back automatically if the error rate rises.
- Leave the previous colour switched on for an hour before retiring it.
Include the necessary Kubernetes manifests.
Solutions
Solution 1
# ==============================================================================
# STAGE 1: BUILD
# ==============================================================================
FROM eclipse-temurin:21-jdk-alpine AS builder
WORKDIR /build
COPY .mvn/ .mvn/
COPY mvnw pom.xml ./
COPY bibliotech-domain/pom.xml bibliotech-domain/
COPY bibliotech-application/pom.xml bibliotech-application/
COPY bibliotech-infrastructure/pom.xml bibliotech-infrastructure/
COPY bibliotech-console/pom.xml bibliotech-console/
RUN --mount=type=cache,target=/root/.m2 \
./mvnw -B -pl bibliotech-console -am dependency:go-offline -DskipTests
COPY bibliotech-domain/src bibliotech-domain/src
COPY bibliotech-application/src bibliotech-application/src
COPY bibliotech-infrastructure/src bibliotech-infrastructure/src
COPY bibliotech-console/src bibliotech-console/src
RUN --mount=type=cache,target=/root/.m2 \
./mvnw -B -pl bibliotech-console -am clean package -DskipTests
RUN java -Djarmode=tools -jar bibliotech-console/target/bibliotech-cli.jar \
extract --layers --launcher --destination extracted
# ==============================================================================
# STAGE 2: GENERATE THE CDS ARCHIVE
# The application is run once with --help so that it loads the classes,
# and the result is memorised. This trims ~1.5 s off every startup, which
# multiplied by the cron runs does matter.
# ==============================================================================
FROM eclipse-temurin:21-jre-alpine AS cds
WORKDIR /app
COPY --from=builder /build/extracted/ ./
RUN java -XX:ArchiveClassesAtExit=/app/cli.jsa \
org.springframework.boot.loader.launch.JarLauncher --help > /dev/null 2>&1 || true
# ==============================================================================
# STAGE 3: RUNTIME
# ==============================================================================
FROM eclipse-temurin:21-jre-alpine AS runtime
LABEL org.opencontainers.image.title="BiblioTech CLI" \
org.opencontainers.image.description="BiblioTech command-line tool" \
org.opencontainers.image.vendor="Nexus Software"
# tzdata for the dates; dumb-init so that Ctrl+C and SIGTERM reach the process
# (the clean cancellation from 12-03 depends on this).
# curl is NOT installed: there is no healthcheck to run.
RUN apk add --no-cache tzdata dumb-init && rm -rf /var/cache/apk/*
ENV TZ=Europe/Madrid
RUN addgroup -S -g 1001 bibliotech && \
adduser -S -u 1001 -G bibliotech -h /app bibliotech
WORKDIR /app
COPY --from=cds --chown=bibliotech:bibliotech /app/ ./
USER bibliotech
# Options AIMED AT STARTUP, not at sustained throughput.
# The process lives seconds: compiling fully with C2 never pays for itself.
# TieredStopAtLevel=1 C1 only: fast, lightweight compilation
# UseSerialGC the cheapest GC to initialise (one task, little memory)
# SharedArchiveFile uses the CDS from stage 2
# MaxRAMPercentage=75 cgroup-aware, as always
ENV JAVA_OPTS="\
-XX:TieredStopAtLevel=1 \
-XX:+UseSerialGC \
-XX:SharedArchiveFile=/app/cli.jsa \
-XX:MaxRAMPercentage=75.0 \
-Xshare:auto \
-Dspring.main.banner-mode=off \
-Dfile.encoding=UTF-8"
# NO EXPOSE: the CLI listens on no port.
# NO HEALTHCHECK: the process exits; there is nothing to watch.
ENTRYPOINT ["dumb-init", "--", "sh", "-c", \
"exec java $JAVA_OPTS org.springframework.boot.loader.launch.JarLauncher \"$@\"", "--"]
# CMD holds the DEFAULT arguments, replaceable at run time
CMD ["--help"]docker build -f Dockerfile.cli -t bibliotech-cli:1.4.2 .
docker run --rm bibliotech-cli:1.4.2 catalog list --format=json
docker run --rm -e BIBLIOTECH_DB_URL=… bibliotech-cli:1.4.2 notices send --days-ahead=3
# Check the effect of CDS
docker run --rm bibliotech-cli:1.4.2 --version # ~1.1 s instead of ~2.6 sThe ENTRYPOINT deserves an explanation, because it is the part people get stuck on: the sh -c plus "$@" form makes it possible to expand $JAVA_OPTS and receive the user's arguments at the same time. The trailing -- is the script's $0, without which the user's first argument would be lost.
The Kubernetes CronJob:
apiVersion: batch/v1
kind: CronJob
metadata:
name: bibliotech-daily-notices
labels:
app: bibliotech
component: scheduled-tasks
spec:
schedule: "0 8 * * 1-5" # Monday to Friday at 8:00
timeZone: "Europe/Madrid" # Kubernetes 1.27+: essential with daylight saving
concurrencyPolicy: Forbid # if the previous one is still running, do NOT start another
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
startingDeadlineSeconds: 600 # if the cluster was down, there is a 10 min margin
jobTemplate:
spec:
backoffLimit: 2 # 2 retries on failure
activeDeadlineSeconds: 900 # kill it if it exceeds 15 minutes
ttlSecondsAfterFinished: 86400
template:
metadata:
labels:
app: bibliotech
task: notices
spec:
restartPolicy: OnFailure
securityContext:
runAsNonRoot: true
runAsUser: 1001
containers:
- name: cli
image: registry.nexussoftware.com/bibliotech-cli:1.4.2
imagePullPolicy: IfNotPresent
args:
- "notices"
- "send"
- "--days-ahead=3"
- "--quiet" # no decoration: the output goes to the log
envFrom:
- configMapRef: { name: bibliotech-config }
- secretRef: { name: bibliotech-secrets }
resources:
requests: { memory: "256Mi", cpu: "100m" }
limits: { memory: "512Mi", cpu: "1000m" }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}And this is where the design from 12-03 pays off: the exit codes govern Kubernetes' behaviour. Code 0 (OK) and 3 (nothing to send) mark the Job as successful; code 7 (email unavailable) marks it as failed and backoffLimit: 2 retries automatically. Without those differentiated codes, it would either always retry or never retry.
kubectl get cronjob bibliotech-daily-notices
kubectl create job --from=cronjob/bibliotech-daily-notices manual-run # run it now
kubectl logs job/manual-runSolution 2
Phase 1 — Expand. A migration compatible with application v1.4.x, which still uses name.
-- V9__split_employee_name_phase1.sql
ALTER TABLE employees ADD COLUMN first_name VARCHAR(80);
ALTER TABLE employees ADD COLUMN surname VARCHAR(120);
-- Populate with a conservative heuristic:
-- the FIRST word is the given name; the rest is the surname.
-- It is imperfect for compound names ("José María"), which is why
-- the original column is kept and a review report is produced.
UPDATE employees
SET first_name = split_part(trim(name), ' ', 1),
surname = NULLIF(trim(substring(trim(name) from position(' ' in trim(name)) + 1)), '')
WHERE first_name IS NULL;
-- Special case: a single term (no spaces) → it is all given name
UPDATE employees
SET first_name = trim(name), surname = NULL
WHERE position(' ' in trim(name)) = 0;
-- Bidirectional sync trigger: it does not matter which app version writes
CREATE OR REPLACE FUNCTION sync_employee_name() RETURNS TRIGGER AS $$
BEGIN
IF TG_OP = 'INSERT' THEN
IF NEW.first_name IS NULL AND NEW.name IS NOT NULL THEN
-- App v1 wrote (old column): derive the new ones
NEW.first_name := split_part(trim(NEW.name), ' ', 1);
NEW.surname := NULLIF(trim(substring(trim(NEW.name)
from position(' ' in trim(NEW.name)) + 1)), '');
ELSIF NEW.name IS NULL AND NEW.first_name IS NOT NULL THEN
-- App v2 wrote (new columns): derive the old one
NEW.name := trim(NEW.first_name || ' ' || COALESCE(NEW.surname, ''));
END IF;
ELSIF TG_OP = 'UPDATE' THEN
IF NEW.name IS DISTINCT FROM OLD.name THEN
NEW.first_name := split_part(trim(NEW.name), ' ', 1);
NEW.surname := NULLIF(trim(substring(trim(NEW.name)
from position(' ' in trim(NEW.name)) + 1)), '');
ELSIF NEW.first_name IS DISTINCT FROM OLD.first_name
OR NEW.surname IS DISTINCT FROM OLD.surname THEN
NEW.name := trim(NEW.first_name || ' ' || COALESCE(NEW.surname, ''));
END IF;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_sync_employee_name
BEFORE INSERT OR UPDATE ON employees
FOR EACH ROW EXECUTE FUNCTION sync_employee_name();
-- Manual review view: the cases the heuristic probably got wrong
CREATE OR REPLACE VIEW v_employees_name_review AS
SELECT id, name, first_name, surname,
CASE
WHEN first_name IN ('José','Jose','María','Maria','Juan','Ana','Luis','Francisco')
AND surname LIKE '% %' THEN 'possible compound given name'
WHEN surname ~ '^(de|del|la|las|los|van|von|di|da) ' THEN 'surname with particle'
WHEN surname IS NULL THEN 'no surname'
ELSE 'review'
END AS reason
FROM employees
WHERE first_name IN ('José','Jose','María','Maria','Juan','Ana','Luis','Francisco')
OR surname ~ '^(de|del|la|las|los|van|von|di|da) '
OR surname IS NULL;Phase 2 — Migrate. Application v1.5.0 uses the new columns.
@Entity
public class Employee {
@Column(name = "first_name", length = 80)
private String firstName;
@Column(name = "surname", length = 120)
private String surname;
/**
* The old column still exists and the trigger maintains it.
* insertable/updatable false: JPA NEVER writes it.
* It is kept mapped only so that it can be read if ever needed.
*/
@Column(name = "name", insertable = false, updatable = false)
private String legacyFullName;
public String fullName() {
return surname == null ? firstName : firstName + " " + surname;
}
}Phase 3 — Contract. Only once v1.4.x no longer exists in any environment.
-- V11__split_employee_name_phase3.sql
-- Pre-check: if anything ended up inconsistent, ABORT
DO $$
DECLARE inconsistent INTEGER;
BEGIN
SELECT count(*) INTO inconsistent
FROM employees
WHERE first_name IS NULL
OR trim(name) IS DISTINCT FROM trim(first_name || ' ' || COALESCE(surname, ''));
IF inconsistent > 0 THEN
RAISE EXCEPTION 'There are % employees with an inconsistent name. Check v_employees_name_review before contracting.', inconsistent;
END IF;
END $$;
DROP TRIGGER IF EXISTS trg_sync_employee_name ON employees;
DROP FUNCTION IF EXISTS sync_employee_name();
DROP VIEW IF EXISTS v_employees_name_review;
ALTER TABLE employees ALTER COLUMN first_name SET NOT NULL;
ALTER TABLE employees DROP COLUMN name;
CREATE INDEX idx_employees_surname ON employees(surname, first_name);Deployment plan with the points of no return:
| Step | Action | Rollback | Duration |
|---|---|---|---|
| 1 | Full backup | — | 10 min |
| 2 | Apply V9 (phase 1) | Yes: drop the columns and the trigger | 2 min |
| 3 | Check the review view and fix by hand | Yes | 1-2 h |
| 4 | Deploy app v1.5.0 | Yes: go back to v1.4.x | 5 min |
| 5 | Watch for 48 hours | Yes | 2 days |
| 6 | Apply V11 (phase 3) | NO. Point of no return | 1 min |
Testing the migration with hard data:
@Tag("integration")
class EmployeeNameMigrationIT extends PostgresTestBase {
@Test
void splitsTheKnownNamesCorrectly() {
// Initial state: up to V8, with the old column
flywayUpTo("8");
jdbc.update("""
insert into employees (name, email, start_date) values
('Marta Ruiz', '[email protected]', '2024-03-01'),
('Diego Alonso', '[email protected]', '2025-01-15'),
('Nuria Vidal', '[email protected]', '2023-09-10'),
('José María Pérez Gómez', '[email protected]', '2022-05-20'),
('Ana de la Torre', '[email protected]', '2021-11-02'),
('Prince', '[email protected]', '2020-01-01')
""");
flywayUpTo("9"); // apply phase 1
assertThat(namesOf("[email protected]"))
.containsExactly("Marta", "Ruiz");
assertThat(namesOf("[email protected]"))
.containsExactly("Nuria", "Vidal");
// Hard cases: the heuristic splits them wrongly, and that is EXPECTED
assertThat(namesOf("[email protected]"))
.containsExactly("José", "María Pérez Gómez"); // needs manual review
assertThat(namesOf("[email protected]"))
.containsExactly("Ana", "de la Torre");
// A single term
assertThat(namesOf("[email protected]"))
.containsExactly("Prince", null);
// And they all appear in the review view, which is what matters:
// the migration does not aim to always get it right, it aims NOT TO LOSE DATA
// and to flag what needs reviewing.
assertThat(jdbc.queryForList("select email from v_employees_name_review", String.class))
.contains("[email protected]", "[email protected]",
"[email protected]");
}
@Test
void theTriggerSyncsInBothDirections() {
flywayUpTo("9");
// App v1 writes the old column
jdbc.update("insert into employees (name, email, start_date) values (?,?,?)",
"Carlos Sanz", "[email protected]", Date.valueOf("2026-01-01"));
assertThat(namesOf("[email protected]")).containsExactly("Carlos", "Sanz");
// App v2 writes the new columns
jdbc.update("""
insert into employees (first_name, surname, email, start_date)
values (?,?,?,?)""",
"Elena", "Ferrer Rico", "[email protected]", Date.valueOf("2026-01-02"));
assertThat(jdbc.queryForObject(
"select name from employees where email = ?", String.class,
"[email protected]"))
.isEqualTo("Elena Ferrer Rico");
}
@Test
void phase3AbortsIfInconsistenciesRemain() {
flywayUpTo("9");
// Force an inconsistency by bypassing the trigger
jdbc.update("alter table employees disable trigger trg_sync_employee_name");
jdbc.update("insert into employees (name, email, start_date) values (?,?,?)",
"Not Split", "[email protected]", Date.valueOf("2026-01-03"));
jdbc.update("alter table employees enable trigger trg_sync_employee_name");
assertThatThrownBy(() -> flywayUpTo("11"))
.hasMessageContaining("inconsistent name");
// And most important of all: the old column IS STILL THERE. Nothing has been lost.
assertThat(columnExists("employees", "name")).isTrue();
}
}Solution 3
Kubernetes manifests:
# k8s/blue-green/service.yaml
# The Service points at ONE colour. Switching = changing the selector.
apiVersion: v1
kind: Service
metadata:
name: bibliotech
labels: { app: bibliotech }
spec:
selector:
app: bibliotech
colour: blue # ← this is what changes on the switch
ports:
- port: 80
targetPort: 8080
---
# Auxiliary Service to test the idle colour WITHOUT real traffic
apiVersion: v1
kind: Service
metadata:
name: bibliotech-preview
spec:
selector:
app: bibliotech
colour: green # adjusted before the smoke tests
ports:
- port: 80
targetPort: 8080
---
# k8s/blue-green/deployment-template.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: bibliotech-COLOUR
labels: { app: bibliotech, colour: COLOUR }
spec:
replicas: 3
selector:
matchLabels: { app: bibliotech, colour: COLOUR }
template:
metadata:
labels: { app: bibliotech, colour: COLOUR }
spec:
terminationGracePeriodSeconds: 60
containers:
- name: bibliotech
image: IMAGE
ports: [{ name: http, containerPort: 8080 }]
envFrom:
- configMapRef: { name: bibliotech-config }
- secretRef: { name: bibliotech-secrets }
resources:
requests: { memory: "512Mi", cpu: "250m" }
limits: { memory: "768Mi", cpu: "1500m" }
startupProbe:
httpGet: { path: /actuator/health/liveness, port: http }
failureThreshold: 20
periodSeconds: 5
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: http }
periodSeconds: 5The workflow:
# .github/workflows/cd-blue-green.yml
name: Blue-green deployment
on:
push:
tags: ['v*']
env:
NAMESPACE: production
REGISTRY: ghcr.io
IMAGE: ${{ github.repository }}
jobs:
# ---------------------------------------------------------------
# 1. Determine the colours
# ---------------------------------------------------------------
colours:
runs-on: ubuntu-latest
outputs:
active: ${{ steps.detect.outputs.active }}
idle: ${{ steps.detect.outputs.idle }}
steps:
- name: Set up kubectl
uses: azure/k8s-set-context@v4
with:
kubeconfig: ${{ secrets.KUBECONFIG }}
- name: Detect the active colour
id: detect
run: |
ACTIVE=$(kubectl get service bibliotech -n $NAMESPACE \
-o jsonpath='{.spec.selector.colour}')
if [ "$ACTIVE" = "blue" ]; then IDLE="green"; else IDLE="blue"; fi
echo "active=$ACTIVE" >> $GITHUB_OUTPUT
echo "idle=$IDLE" >> $GITHUB_OUTPUT
echo "::notice::Active: $ACTIVE — will deploy to: $IDLE"
# ---------------------------------------------------------------
# 2. Deploy to the idle colour (no traffic)
# ---------------------------------------------------------------
deploy-idle:
runs-on: ubuntu-latest
needs: colours
steps:
- uses: actions/checkout@v4
- uses: azure/k8s-set-context@v4
with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }
- name: Render and apply the idle colour's Deployment
run: |
COLOUR=${{ needs.colours.outputs.idle }}
IMG=${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ github.ref_name }}
sed -e "s|COLOUR|$COLOUR|g" -e "s|IMAGE|$IMG|g" \
k8s/blue-green/deployment-template.yaml | kubectl apply -n $NAMESPACE -f -
kubectl rollout status deployment/bibliotech-$COLOUR -n $NAMESPACE --timeout=10m
- name: Point the preview Service at the idle colour
run: |
kubectl patch service bibliotech-preview -n $NAMESPACE \
-p '{"spec":{"selector":{"app":"bibliotech","colour":"${{ needs.colours.outputs.idle }}"}}}'
# ---------------------------------------------------------------
# 3. Smoke tests against the idle colour
# ---------------------------------------------------------------
smoke:
runs-on: ubuntu-latest
needs: [colours, deploy-idle]
steps:
- uses: azure/k8s-set-context@v4
with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }
- name: Run the tests against the idle colour
run: |
kubectl port-forward service/bibliotech-preview 18080:80 -n $NAMESPACE &
PF=$!
sleep 8
set -e
BASE=http://localhost:18080
echo "→ Health"
curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"'
echo "→ Deployed version"
VERSION=$(curl -fsS "$BASE/actuator/info" | jq -r '.build.version')
test "v$VERSION" = "${{ github.ref_name }}" \
|| { echo "::error::Unexpected version: $VERSION"; exit 1; }
echo "→ Catalogue"
curl -fsS "$BASE/api/materials?size=1" | jq -e '.content | length >= 0'
echo "→ Expected errors (404 and 400)"
test "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/api/materials/978-9999999999")" = "404"
test "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/api/materials/not-an-isbn")" = "400"
kill $PF
echo "Smoke tests passed"
# ---------------------------------------------------------------
# 4. Switch the traffic (with manual approval)
# ---------------------------------------------------------------
switch:
runs-on: ubuntu-latest
needs: [colours, smoke]
environment:
name: production # with required reviewers
url: https://bibliotech.nexussoftware.com
steps:
- uses: azure/k8s-set-context@v4
with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }
- name: Switch the Service to the new colour
run: |
kubectl patch service bibliotech -n $NAMESPACE \
-p '{"spec":{"selector":{"app":"bibliotech","colour":"${{ needs.colours.outputs.idle }}"}}}'
echo "::notice::Traffic switched to ${{ needs.colours.outputs.idle }}"
# ---------------------------------------------------------------
# 5. Watch for 5 minutes; roll back if the error rate rises
# ---------------------------------------------------------------
monitor:
runs-on: ubuntu-latest
needs: [colours, switch]
steps:
- uses: azure/k8s-set-context@v4
with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }
- name: Watch the error rate for 5 minutes
id: monitoring
run: |
BASE=https://bibliotech.nexussoftware.com
ERROR_THRESHOLD=0.02 # 2% of 5xx
for i in $(seq 1 10); do
sleep 30
TOTAL=$(curl -fsS "$BASE/actuator/metrics/http.server.requests" \
| jq '.measurements[] | select(.statistic=="COUNT") | .value')
ERRORS=$(curl -fsS "$BASE/actuator/metrics/http.server.requests?tag=outcome:SERVER_ERROR" \
| jq '.measurements[] | select(.statistic=="COUNT") | .value // 0')
RATE=$(echo "scale=4; $ERRORS / ($TOTAL + 1)" | bc)
echo "Check $i/10 — total=$TOTAL errors=$ERRORS rate=$RATE"
if (( $(echo "$RATE > $ERROR_THRESHOLD" | bc -l) )); then
echo "::error::Error rate $RATE above the threshold $ERROR_THRESHOLD"
exit 1
fi
if ! curl -fsS "$BASE/actuator/health/readiness" | jq -e '.status == "UP"' > /dev/null; then
echo "::error::The readiness probe is failing"
exit 1
fi
done
echo "Monitoring passed"
- name: Automatic ROLLBACK
if: failure()
run: |
echo "::warning::Going back to colour ${{ needs.colours.outputs.active }}"
# The rollback is INSTANT: the previous colour is still on and healthy
kubectl patch service bibliotech -n $NAMESPACE \
-p '{"spec":{"selector":{"app":"bibliotech","colour":"${{ needs.colours.outputs.active }}"}}}'
curl -X POST "${{ secrets.TEAM_WEBHOOK }}" \
-H 'Content-Type: application/json' \
-d '{"text":"🔴 BiblioTech ${{ github.ref_name }}: automatic rollback to ${{ needs.colours.outputs.active }}"}'
exit 1
# ---------------------------------------------------------------
# 6. Retire the old colour, one hour later
# ---------------------------------------------------------------
retire-old:
runs-on: ubuntu-latest
needs: [colours, monitor]
steps:
- uses: azure/k8s-set-context@v4
with: { kubeconfig: '${{ secrets.KUBECONFIG }}' }
- name: Wait an hour before retiring
run: sleep 3600 # safety window: instant rollback available for 1 hour
- name: Scale the old colour down to zero replicas
run: |
# The Deployment is not DELETED: it is scaled to 0.
# That way the manifest survives and bringing it back is one command.
kubectl scale deployment/bibliotech-${{ needs.colours.outputs.active }} \
--replicas=0 -n $NAMESPACE
curl -X POST "${{ secrets.TEAM_WEBHOOK }}" \
-H 'Content-Type: application/json' \
-d '{"text":"✅ BiblioTech ${{ github.ref_name }} stable. Colour ${{ needs.colours.outputs.active }} retired."}'Advantages of blue-green over a rolling update, which is what the exercise assesses:
| Aspect | Rolling | Blue-green |
|---|---|---|
| Rollback | Roll forward in reverse: minutes | Instant: one patch |
| Versions coexisting | Yes, unavoidably | No: all the traffic goes to one |
| Testing before exposing | No | Yes, with the preview Service |
| Resources needed | 1× + 1 pod | 2× during the transition |
| Schema compatibility | Mandatory | Still advisable, because of rollback |
And the point that closes the circle with section 21: instant rollback only works if the database is compatible with both versions. If the new version applied a destructive migration, patching the Service back to the old colour fixes nothing: the old application will find a schema it does not understand. That is why the expand-contract pattern is not optional but the condition that makes blue-green mean anything.
Conclusion
BiblioTech is in production.
You understand what really separates your laptop from a real environment — Java version, configuration, schema, memory, the consequences of a failure — and the three principles that govern a healthy deployment: one artefact for every environment, configuration from the outside and everything must be undoable.
You package into an executable jar, knowing why it displaced the war and how it is built inside. And you use the layered jar, which is not a detail: it turns a 60 MB deployment into a 1 MB one, from forty seconds to three, and with that it changes the team's behaviour — because a three-second deployment gets done without a second thought and a two-minute one gets stockpiled "for Thursday". With reproducible builds and traceability to the commit, so that "which version is in production?" has an exact answer in one second.
You containerise with a multi-stage Dockerfile that you understand line by line: the build stage with Maven that never reaches the final image; the POMs copied before the code so the dependency cache works; the non-root user; the layers ordered by stability; dumb-init as PID 1 without which SIGTERM never reaches the JVM; and the JVM options with MaxRAMPercentage and ExitOnOutOfMemoryError. With a .dockerignore that stops secrets and the .git directory entering the context, and knowing the alternatives — Buildpacks and Jib — with their real advantages.
You know what almost nobody knows about the JVM in a container: that since Java 10 it is cgroup-aware, that the default 25% wastes memory, and above all that the JVM's memory is not only the heap — metaspace, stacks, code cache, direct buffers — which is why a Java container dies with OOMKilled without the application logging a thing.
You govern the schema with Flyway, with the six concrete reasons why ddl-auto: update is no good in production, the version convention, the golden rules — an applied script is never modified — and the three-phase expand-contract pattern, which is what lets a migration coexist with two versions of the application while keeping rollback possible.
You choose where to deploy with judgement, knowing that systemd on your own server is still a perfectly valid answer and that adopting Kubernetes for three services with no platform team is a decision you pay for every week. And if it is Kubernetes, you have a Deployment with maxUnavailable: 0, exact tags instead of latest, resource limits and a PodDisruptionBudget.
You expose probes distinguishing liveness from readiness — the difference between restarting every instance because the database hiccupped and simply taking them out of the balancer — with custom indicators that report degraded rather than down for non-essential dependencies. You shut down gracefully with server.shutdown: graceful, @PreDestroy and the preStop wait that covers the race window with the balancer. And you know the fast-startup options — CDS, AOT, CRaC, Native Image — with the criteria for when each one pays off.
You handle the four deployment strategies with their costs, the feature flags that separate deployment from activation, and the hard limit of rollback: the code comes back, the data does not. Hence the rule that sums it all up: the database goes first and is always backward compatible.
You have the complete continuous delivery pipeline: multi-architecture builds, a blocking vulnerability scan, image signing, deployment by digest, automatic staging, production with approval, post-deployment health verification and automatic rollback. And you know what horizontal scaling demands of the application — no in-memory state, scheduled tasks coordinated with ShedLock or a CronJob, and the reminder that scaling the application does not scale the database.
BiblioTech works, it is tested, it is deployed and it can be updated without interrupting the service. And it has two holes the size of an entire project.
The first: anybody can do anything. There is no authentication, no authorisation, passwords do not exist, the API is wide open and nobody has checked whether it is vulnerable to the things that put applications in the news.
The second: when something fails, you will find out from a phone call. There are no metrics, no traces, no alerts, and the only log is plain text in an ephemeral container.
The final lesson closes both, and closes the course: security — common vulnerabilities and their concrete prevention, Spring Security with BCrypt and JWT, and the warning about what a course cannot replace —, observability — the three pillars, Micrometer and Actuator, Prometheus and Grafana, distributed tracing and useful alerts —, and evolution — API versioning, technical debt, upgrades and how to grow the system. And at the end, a recap of BiblioTech's complete journey, what you can do now, and where to go next.
Java Programming Course
Module 1: Introduction to Java
- Introduction to Java
- Setting Up the Development Environment
- Basic Syntax and Structure
- Variables and Data Types
- Operators
- Console Input and Output
- Your First Complete Program: BiblioTech
Module 2: Control Flow
- Conditional Statements
- Loops
- Switch Statements
- Break and Continue
- Debugging and Execution Traces
- Project: The BiblioTech Interactive Menu
Module 3: Object-Oriented Programming
- Introduction to OOP
- Classes and Objects
- Methods
- Constructors
- Inheritance
- Polymorphism
- Encapsulation
- Abstraction
- The Object Class: equals, hashCode and toString
Module 4: Advanced Object-Oriented Programming
- Interfaces
- Abstract Classes
- Inner Classes
- Anonymous Classes
- Lambda Expressions
- Functional Interfaces and Method References
- Enums and Records
Module 5: Data Structures and Collections
- Arrays
- The Collections Framework
- ArrayList
- LinkedList
- HashMap
- HashSet
- Queue and Deque
- Stack
- Sorting and Searching Collections
Module 6: Exception Handling
- Introduction to Exceptions
- The Try-Catch Block
- Throw and Throws
- Custom Exceptions
- The Finally Block
- Try-with-resources and AutoCloseable
- Error Handling Strategies and Logging
Module 7: File Input/Output
- Reading Files
- Writing Files
- File Streams
- BufferedReader and BufferedWriter
- Serialization
- The NIO.2 API: Path and Files
- Interchange Formats: CSV and Properties
Module 8: Multithreading and Concurrency
- Introduction to Multithreading
- Creating Threads
- Thread Lifecycle
- Synchronization
- Concurrency Utilities
- Concurrent Collections and Atomic Variables
- Asynchronous Tasks with CompletableFuture
Module 9: Networking
- Introduction to Networking
- Sockets
- ServerSocket
- DatagramSocket and DatagramPacket
- URL and HttpURLConnection
- The Modern HTTP Client
Module 10: Advanced Topics
- Generics
- Annotations
- Reflection
- Java 8 Features: Streams and Optional
- Dates and Times with java.time
- Java 9 and Beyond
- Memory, Garbage Collection and Performance
Module 11: Java Frameworks and Libraries
- Introduction to Java Frameworks
- Spring Framework
- Hibernate
- JUnit
- Maven
- Advanced Testing with Mockito
- Essential Ecosystem Libraries
