Escena Viva now configures itself properly and now says what it is doing: structured logs with per-request traceability, metrics on /metrics, /health/live and /health/ready probes. But all of that is still started by hand. Somebody SSHes into the server, types npm start, watches the first log lines go by and closes the terminal. And at that moment the application dies. This lesson solves the process problem: who starts it, who watches it, who brings it back when it falls over and who reloads it when you deploy a new version without cutting a single purchase in half. The tool will be PM2, but the concepts are what matter: they apply just the same if tomorrow you use systemd or an orchestrator.

1. The problem: npm start in a terminal
2. What a process supervisor is and which one to pick
3. PM2: installation and reference commands
4. The Escena Viva ecosystem file
5. PM2 cluster mode versus our src/cluster.js
6. reload versus restart: zero-downtime reloads
7. Automatic restarts and loop protection
8. Memory restarts as a safety net
9. Starting on machine boot
10. PM2 logs and their rotation
11. Environment variables and secrets in PM2
12. Deployment and monitoring
13. What PM2 does not solve

When you run npm start in an SSH session, the Node process is a child of your shell, and your shell is a child of the SSH daemon. When you disconnect, the kernel sends SIGHUP to the session and the whole tree collapses. Your API disappears. The classic workarounds — nohup npm start &, screen, tmux — solve only that part, and leave five problems untouched:

• If the process falls over with an uncaughtException (which, remember, in 11-02 we deliberately decided terminates the process), nobody brings it back.
• If the server reboots, the application does not come back on its own.
• The logs go to an endless file that will fill the disk.
• There is no way to deploy without downtime: stopping and starting leaves a gap of seconds full of 502s.
• There is no visibility: how many processes there are, how much memory they use, how many times they have restarted.

That is exactly what a supervisor does.

A supervisor is a program whose only job is keeping other programs alive: it starts them, watches them, restarts them if they exit, redirects their output and survives the end of your session.

For Escena Viva on a VPS, PM2 is the reasonable choice: it gives you clustering and zero-downtime reloads without writing a line of systemd. In 11-04 and 11-05 we will see that as soon as you containerize, the supervisor becomes somebody else — and that overlap is normal, not a design flaw.

PM2 is installed globally, not as a dependency. Putting it in dependencies is a common mistake: PM2 starts your application, it is not part of it, and folding it in complicates the Docker image in the next lesson. On the first command, PM2 starts a background daemon (the God daemon) that outlives your session and keeps the list of applications. The CLI commands talk to that daemon.

A note on style: although pm2 start src/server.js -i 4 works, on a real server you never start it that way. The configuration would live only in the bash history of whoever typed it. Everything goes into the ecosystem file.

Escena Viva has two processes to deploy, and this comes straight from Module 10: the HTTP API and the BullMQ queue consumer (src/processes/ticket-consumer.js), which produces the ticket PDFs and the emails. They are two different applications, with different resource profiles, and they must scale separately: during the Festival de Jazz opening you may need more consumers without needing more API.

The decisions behind it:

• exec_mode: 'cluster' for the API only. Cluster mode exists to share a port between several processes. The consumer does not listen on any port: BullMQ distributes the work through Redis. Clustering there would add nothing, so it runs in fork.
• instances: 0 for the API, a fixed 2 for the consumer. The API scales with the cores; the consumer scales with the queue load and with the memory the PDF-generating worker threads consume (Module 10). They are different axes.
• A different kill_timeout for each. The API drains HTTP requests: 15 seconds is plenty. The consumer may be halfway through a PDF: 30 seconds keeps you from killing it mid-job.
• time: false. It is subtle and very important: if PM2 prepends its own timestamp to each line, pino's JSON stops being valid JSON and the aggregator cannot index it. With structured logging, PM2 must limit itself to redirecting.
• env_production contains not a single secret. We will come back to this in section 11.

You start it with:

Without --env production, PM2 uses the env block, that is, development. It is one of the most common ways to end up with NODE_ENV=development on a production server, with all the consequences we saw in 11-01.

In Module 10 we wrote src/cluster.js by hand: a primary process that forks N workers with node:cluster, supervises them, replaces them if they die and performs a rolling zero-downtime reload. It works. PM2 in exec_mode: 'cluster' does exactly the same: it uses Node's cluster module underneath, with the same mechanism for sharing the socket between child processes. It is not magic and it is not a different technology.

What you gain: you stop maintaining and testing infrastructure code that is not your business, and you get hot scaling, memory restarts and monitoring for free. What you lose: an external dependency that has to be installed and updated on the server, and less fine-grained control over each worker's lifecycle.

And why writing it by hand was not wasted time: because you now understand that instances: 4 does not create four threads but four processes with separate memory — which is why sessions and rate limiting had to move to Redis in Module 10 — that the Sequelize pool is multiplied by four (and that is why src/db/sequelize.js sizes it according to the worker count), and that one process's in-memory state does not exist for the other three. Somebody who has never written a cluster by hand configures instances: 16 and then wonders why PostgreSQL runs out of connections. In production with PM2, src/cluster.js falls out of use: script points at src/server.js and PM2 supplies the cluster. The file stays in the repository as an alternative for environments where PM2 is absent (inside a container, for instance, though it is rarely needed there either).

This is the main reason to use PM2 on your own server. pm2 restart: it kills every process and starts new ones. Between the two there is a gap of one to three seconds during which nobody is listening on the port. Everything arriving in that gap is a 502.

pm2 reload (cluster mode only): it replaces workers one at a time. It starts the new one, waits until it is ready, takes the old one out of the connection rotation, gives it time to finish what it has in hand and kills it. Somebody is always listening. Zero errors for the user.

For this to really work you need three coordinated pieces, and the first two have existed since Module 6. 1. Graceful shutdown in src/server.js. PM2 sends SIGINT (not SIGTERM) to clustered processes. Our handler already listens for both signals: it stops accepting connections, sets state.acceptingTraffic = false so /health/ready returns 503, closes idle connections with closeIdleConnections, waits for the active ones and exits with a grace timer.

2. The ready signal. With wait_ready: true, PM2 does not consider the worker alive until the worker says so explicitly:

The difference is enormous. Without wait_ready, PM2 accepts the worker as soon as the process exists, and retires the old one while the new one is still connecting to PostgreSQL: the first users see errors. With wait_ready, the replacement window only opens once the new worker can genuinely serve. Note the if (process.send): it only exists when the process has been forked by another one, so starting directly with node src/server.js still works. 3. A sufficient kill_timeout. It is the time PM2 waits from SIGINT to SIGKILL. It must be longer than the longest drain your application expects. If you have requests that take up to 10 seconds, a kill_timeout of 5000 kills responses halfway through and all the Module 6 work goes to waste. Our 15 seconds on the API cover the worst case with room to spare.

And listen_timeout is the flip side: if ready does not arrive within 10 seconds, PM2 assumes startup has failed. It protects against the scenario where a new version cannot connect to the database and hangs forever, leaving the deployment half-done.

With autorestart: true, if a process exits, PM2 starts another. Good. But imagine the new version has a startup bug: an environment variable is missing and src/config/index.js calls process.exit(1) (11-01). PM2 restarts it. It fails again. It restarts it. In a loop, hundreds of times a minute, burning CPU and filling the disk with logs. Three parameters prevent it:

With this combination, a process that dies a second after starting is retried ten times with a growing wait and then moves to the errored state, where it stays. PM2 stops insisting and waits for human intervention, which is the right thing to do: the application is broken and retrying will not fix it. And a warning that is worth the whole lesson: restarting without fixing the cause is papering over a problem. If pm2 list shows 3,400 restarts in the ↺ column, you do not have a resilient system: you have a bug that shows up every few minutes and a supervisor hiding it. Check that column every time you look. A number that grows is a pending investigation, not a reassurance.

max_memory_restart: '700M' restarts the process when it exceeds that usage. It is useful, but you have to be clear about what it is and what it is not.

What it is: a safety net. Faced with a slow memory leak — the kind Module 9 taught us to diagnose with heap snapshots — it keeps the process from growing until the kernel kills it out of memory (the OOM killer) or until garbage collection works all day and the p99 goes through the roof. What it is not: a solution. If the process restarts on memory every two hours, you have a leak and it needs finding. The restart buys time to investigate, it does not close the matter.

To choose the value: look at the steady usage in production with pm2 monit and set it 60-80% above that. Too low and you will restart for no reason under heavy load; too high and the safety net never triggers. Also bear in mind that the limit is per process: with 4 instances at 700 MB you need 2.8 GB for the API alone, plus the consumer. That arithmetic becomes critical inside the container in the next lesson.

A supervisor that does not survive a server reboot solves half the problem. Two commands:

The order matters and forgetting is a classic: pm2 startup makes PM2 start, but with no application list. It is pm2 save that writes the dump (~/.pm2/dump.pm2) that PM2 restores. If you change the ecosystem file and do not run pm2 save again, the old configuration will come back after the next server reboot. Rule: after any change to the applications, pm2 save. A curious and healthy note: underneath, pm2 startup creates a systemd unit. That is, systemd supervises PM2 and PM2 supervises your applications. If that strikes you as one layer too many, the intuition is correct, and it is part of the argument for moving to containers in the next lesson.

PM2 captures each process's stdout and stderr and writes them to the files given in out_file and error_file. Since pino already writes structured JSON (11-02), PM2 must only redirect: no added timestamps, no prefixes.

merge_logs: true puts the four instances into one file. It might look confusing, but every pino JSON line already carries pid and requestId, so splitting by file adds nothing and complicates querying.

Without rotation, those files grow until they fill the disk, and a full disk takes down the application and the database. The PM2 module that solves it:

Even so, remember factor XI of the twelve factors (11-02): ideally the logs should not end up in files on the server but in an aggregator. A common pattern with PM2 is to let it write to files and put a lightweight agent (Vector, Fluent Bit, Promtail) in place to read those files and ship them. The files become a temporary buffer, not the final destination.

The env and env_production blocks in the ecosystem file are convenient... and they are a risk, because that file is versioned. Putting JWT_ACCESS_SECRET there is exactly the failure of the public repository test from 11-01. Only non-secret values belong in the ecosystem file: NODE_ENV, PORT, LOG_LEVEL, TRUST_PROXY. Secrets arrive by another route. Three options, from least to most recommendable:

a) A .env file outside version control, loaded by dotenv. That is what src/config/index.js already does. On the server there is /opt/escena-viva/.env with 600 permissions, owned by the application user. Simple and sufficient for a VPS. b) systemd's EnvironmentFile in the unit generated by pm2 startup. The variables enter the PM2 daemon and every application inherits them. Less flexible if you have several applications with different secrets.

c) Exporting them before starting PM2, pulling them from a secret manager:

--update-env is essential: PM2 remembers the environment an application started with and reuses it on restarts. Without that flag, you change a variable, reload, and it keeps running with the old value. It is one of the most frequent confusions with PM2, and it has cost plenty of people more than an hour of debugging. If you rotated a secret following the procedure in 11-01 and it appears to have had no effect, start here.

PM2 includes an SSH-based deployment system, pm2 deploy, configured in the same file:

With pm2 deploy production it connects over SSH, runs git fetch and checkout, executes post-deploy and reloads without downtime. Notice the order: npm ci --omit=dev (the Module 5 payoff), migrations before the reload, and reload, never restart. It is a decent solution for a VPS and a small team. It has clear limits: it deploys from the repository on each machine, so every server builds on its own and can end up with slightly different dependencies. The alternative — building one artifact and promoting it — is what we will see with Docker in 11-04 and with the CI pipeline in 11-06. In the meantime, pm2 deploy works.

For day-to-day monitoring:

• pm2 monit: an interactive dashboard with CPU and memory per process plus live logs. Useful during a deployment or a load spike like the Festival de Jazz opening.
• pm2 describe escena-viva-api: an application's full record. The first thing to look at when investigating: restart count, uptime, effective environment and log paths.
• pm2 list: the quick view. Keep an eye on the restart column (↺) and on memory.

These tools are for occasional inspection. Continuous monitoring is the 11-02 kind: metrics in Prometheus, dashboards in Grafana and alerts on symptoms. pm2 monit does not wake you up at night.

And here comes the honest closing. PM2 solves the process, not the environment. PM2 guarantees your application is alive, restarts if it falls over, reloads without downtime and starts with the machine. All of that assuming the machine is correct. And that assumption is enormous:

• The Node version has to be the one engines demands (>=24.5.0 <25). If the server has 20, require('node:...') may work and other things may not, and you will find out in production.
• The system libraries have to be present: bcrypt compiles against the system libc, the fonts for generating PDFs have to exist, libpq for PostgreSQL.
• The environment variables have to be set, with the right permissions and up to date.
• The user, the directories, the permissions and the log paths have to exist.

All of that is configured by hand on each server. And as soon as there are two servers, they start to diverge: one has a different OpenSSL version, on the other somebody installed something for debugging and never removed it. The result is the classic "it works on server A and not on server B", which is the grown-up version of "it works on my machine". The answer is to package the environment together with the application, so that what you deploy is not code running on an unknown machine but an artifact that carries its own filesystem inside. That is a container.

• Starting without --env production. PM2 uses the env block (development). Always check with pm2 describe which NODE_ENV is actually in use.
• Forgetting --update-env after changing a variable. PM2 reuses the saved environment and your change has no effect.
• Using restart where reload was called for. A two-second outage on every deployment, avoidable with one different letter.
• A kill_timeout that is too short. PM2 kills mid-response and cancels out the graceful shutdown from Module 6.
• Leaving time: true with JSON logging. The prefix breaks the JSON and the aggregator cannot index anything.
• Secrets in env_production. They are versioned. It is the same mistake as 11-01 wearing different clothes.
• Forgetting pm2 save after changing the configuration. On the next server reboot the previous configuration comes back, sometimes months later, and nobody understands what happened.
• Ignoring the restart column. A counter that grows is a bug hidden by the supervisor.
• Tip: run PM2 as an unprivileged user, never as root. And if the application needed port 80 (Module 4), put a reverse proxy in front instead of granting Node privileges.
• Tip: pin the PM2 version on the server and upgrade it deliberately. It is infrastructure, and a surprise upgrade during a sale makes for a bad night.

With the API running in cluster mode with four instances, fire autocannon (Module 10) at /api/events for 30 seconds and run pm2 reload escena-viva-api halfway through. Check that no error and no 502 appears. Then repeat with pm2 restart and compare.

Deliberately cause a startup failure (remove JWT_ACCESS_SECRET from the environment) and observe PM2's behavior with pm2 logs and pm2 list. Check that after max_restarts attempts the application ends up in the errored state. Document how long it takes to give up with exp_backoff_restart_delay: 200.

The ticket queue piles up 2,000 pending jobs during the Festival de Jazz early sales. Scale the consumer to six instances without stopping the API, verify the effect on the escenaviva_queue_pending metric from 11-02 and explain which other limit could be hit before adding more instances helps.

Exercise 1.

With reload, the autocannon report shows non-2xx: 0 and errors: 0; in the latency you see at most a slight rise in the p99 while one fewer worker serves the traffic. With restart, dozens of connection-refused errors appear, all concentrated at the moment of the cut. The difference is entirely due to the three pieces in section 6: graceful shutdown, wait_ready and a sufficient kill_timeout. If you also see errors when testing with reload, check that process.send('ready') is genuinely being sent: without it, wait_ready: true would make startup time out.

Exercise 2. With exponential retries starting at 200 ms (200, 400, 800, 1600...), ten attempts add up to around 100 seconds before PM2 gives up. In pm2 logs you see the message from src/config/index.js repeated:

And there you can see the value of the validation from 11-01: the reason for the failure appears on the first line of every attempt, instead of a cryptic internal jsonwebtoken error three hours later. In pm2 list the final status is errored and the restart counter stops at 10.

Exercise 3.

The scaling is immediate and does not affect the API, because they are independent applications. escenaviva_queue_pending should fall with a slope roughly three times steeper. The limit you hit first is connections. Each consumer opens connections to Redis (BullMQ uses several per worker) and to PostgreSQL to record the issued tickets. With 6 consumers × concurrency 5, plus the 4 API instances with their pool, it is easy to exceed PostgreSQL's max_connections or Redis's client limit. It is the same arithmetic that forced us in Module 10 to size the Sequelize pool according to the worker count, and the one that will come back in 11-05 with the connection limits of managed plans. Adding instances without recomputing that product turns a performance problem into an outage.

Escena Viva no longer depends on an open SSH session. PM2 supervises it: two applications declared in a versioned ecosystem file — the API in cluster mode and the queue consumer in fork mode, each with its own resource policy — zero-downtime reloads resting on the graceful shutdown from Module 6 and on the ready signal we now emit, protection against restart loops, memory restarts as a safety net against leaks, automatic startup with the machine and rotated logs that do not break pino's JSON. We have also seen why writing src/cluster.js by hand in Module 10 is still valuable even though we no longer use it: understanding process-level distribution is what lets you configure instances without exhausting the database connections. But PM2 supervises the process, not the machine. You still need the server to have the exact Node version, the system libraries to compile bcrypt, the fonts for the PDFs and the right configuration — and you need the second server to be identical to the first, which never happens. In the next lesson, Packaging with Docker, we make the environment travel with the application: a multi-stage Dockerfile explained line by line, a non-root user, CMD in exec form so SIGTERM genuinely reaches the process, a HEALTHCHECK built on /health/ready, and a docker compose setup with the API, the consumer, PostgreSQL, MongoDB and Redis that will finally give you the complete development environment you have been asking for since Module 7.