TechCorp can now see what happens (06-01, 06-02) and survives failures (06-03). What remains is the third question Marta has been asking since 01-05: the Black Friday when the catalog received ×20 traffic and the store went 50 minutes without selling. In the monolith the only lever was "a bigger machine"; on Kubernetes we have replicas, autoscaling and the possibility of scaling only the service that needs it. But scaling is neither free nor automatic: you have to know what makes a service scalable, how to tell the cluster to grow, how to check it beforehand with load tests and, above all, where the bottlenecks are that no replica fixes (the database, a blocked event loop, a badly sized pool). This lesson walks through those layers from the outside in.

1. Vertical and horizontal scaling: what makes a service scalable
2. Autoscaling on Kubernetes: HPA for catalog-service
3. Scaling consumers by queue length: KEDA for inventory-service
4. requests, PodDisruptionBudget and cluster scaling
5. The Black Friday case: sizing and checking with k6
6. Caching: Redis in Catalog and HTTP caching at the gateway
7. Connection pools: the math you have to do
8. Queries, indexes and Node's event loop
9. Scaling messaging: competing consumers and ordering
10. Scaling the database: read replicas and partitioning
11. "Symptom → where to look → remedy" table

A service is horizontally scalable when two replicas behave the same as one and do not get in each other's way. The template from 04-02 already guarantees it, and it is worth remembering why:

• No state in the process. Nothing in memory that a second replica would need: no sessions (the JWT from 07-01 travels with the request), no "real" caches (Redis, section 6), no local files. The template's in-memory cache (Map with TTL) is tolerable only if losing it breaks nothing.
• Limited and known connections. Each replica opens its pg pool (section 7) and its RabbitMQ channel; N replicas = N pools. If you do not do the math, the DB runs out of connections before the service runs out of CPU.
• Fast, clean startup and shutdown. readinessProbe (05-02) so as not to receive traffic too early; SIGTERM (06-03) so as not to lose requests when scaling down.
• Periodic jobs that tolerate replicas. The outbox relay and the watchdog (06-03) use FOR UPDATE SKIP LOCKED so that two replicas do not step on each other; without that, scaling Orders would duplicate events.

The HorizontalPodAutoscaler watches a metric and adjusts the Deployment's replicas between a minimum and a maximum. For Catalog, the service that multiplies ×20:

Line by line:

• scaleTargetRef: the Deployment from 05-02. From now on replicas is not set in the manifest (the HPA would override it); Kustomize leaves the field out and the HPA is in charge.
• minReplicas: 2 for availability (a pod can die at any moment); maxReplicas: 20 is the cost ceiling and also protects MongoDB from receiving 200 connections.
• CPU at 60%: the percentage is computed over resources.requests.cpu (100m in 05-02), not over limits. With requests: 100m, the HPA adds replicas when the average exceeds 60m. That is why requests must reflect real consumption under normal conditions (section 4).
• Custom metric http_requests_per_second: it comes from 06-01's http_requests_total through the Prometheus Adapter, which exposes PromQL queries as metrics of the custom.metrics.k8s.io API. Its configuration (excerpt):

It turns http_requests_total into http_requests_per_second per pod with a 2-minute rate. With two metrics, the HPA computes the replicas needed for each one and takes the larger.

• behavior: without it, the HPA reacts with the default policies. Here we scale up fast (double per minute: on Black Friday there is no time) and down slowly (5 minutes of stabilization, 2 pods per minute) to avoid flapping on short spikes.

It is checked with kubectl get hpa -n techcorp (TARGETS 45%/60%, 80/150) and kubectl describe hpa catalog-service shows every decision. In 05-04 we scaled by hand with kubectl scale; that command is still useful for a one-off adjustment, but the HPA will revert it on the next cycle (15 s).

Inventory does not receive HTTP: it consumes inventory.orders. Its load does not show up in CPU until it is already late; it shows up in rabbitmq_queue_messages_ready. The HPA does not speak RabbitMQ, but KEDA (Kubernetes Event-Driven Autoscaling) does: it creates and manages an HPA underneath, driven by scalers for queues, Prometheus, cron, etc.

• mode: QueueLength, value: 50: KEDA wants replicas = ceil(ready_messages / 50); with 400 messages waiting it goes to 8 replicas.
• cooldownPeriod: 120: two minutes with the queue empty before scaling down.
• authenticationRef reuses the RabbitMQ Secret credentials; they are not put in the YAML.
• KEDA can also scale to zero (minReplicaCount: 0) for sporadic jobs; for Inventory we keep 2 because the first message must not wait for a startup.

Scaling consumers has an effect on message ordering that is covered in section 9, and a clear limit: more Inventory replicas means more load on its PostgreSQL. maxReplicaCount: 10 comes out of the math in section 7.

Three pieces from the Platform team that make the above possible:

• Real resources.requests. The scheduler places pods according to requests; the HPA computes percentages over them. If requests.cpu: 100m but the service uses 300m at rest, the HPA will see "300%" and scale to the maximum for no reason; if it uses 20m, it will never scale. They are tuned by looking at the p50 of container_cpu_usage_seconds_total (06-01) during normal hours, and limits with headroom for spikes (05-02: 100m/500m for Orders; Catalog moves to 200m/1000m after the tests in section 5).
• PodDisruptionBudget: when scaling down, draining a node (kubectl drain) or upgrading the cluster, Kubernetes may kill several pods at once. The PDB sets a floor:

With this, a drain that would leave Catalog with zero pods waits for another node to bring one up. It goes into the Kustomize base of every service with minReplicas ≥ 2.

• Cluster Autoscaler (or Karpenter): when the HPA asks for 20 replicas and they do not fit on the nodes, the pods stay Pending; the cluster autoscaler adds nodes (and removes them when load drops). It is the responsibility of Platform/the provider; for service teams it is enough to know that "replica pending for minutes" means the cluster is growing, and that it takes 2-4 minutes: that is why Catalog goes into Black Friday with minReplicas raised to 6 (a temporary Kustomize overlay), not relying on reaction alone.

Data from 01-05: ~3,000 orders/day on a normal day (≈0.03/s on average, with peaks of 0.5/s); Catalog about 200 requests/s at peak hour. On Black Friday: catalog ×20 (4,000/s) and orders ×3 (peaks of 1.5/s, about 130 orders/minute).

Sizing means measuring how much one replica can take and dividing. That is what load tests with k6 are for, in each service's repository under tests/load/:

• stages describes the shape of the traffic: ramps and plateaus, with a spike that simulates the rush. Each virtual user (VU) runs the function in a loop with a half-second pause: 1,000 VUs ≈ 2,000 requests/s if the server responds instantly.
• thresholds turns the test into pass/fail: they are the same targets that the Grafana panel from 06-01 paints red (300 ms) and that 06-05 will formalize as an SLO. It runs in CI against staging (k6 run -e BASE_URL=https://staging.api.techcorp.example tests/load/catalog.js) from a Job in the 05-03 pipeline, in a manual stage before campaigns.
• X-Request-Id with the k6- prefix: the test's logs can be told apart (and excluded) in Loki.

How to read the k6 summary:

• p(95)=240ms at 1830/s on 2 replicas ⇒ one replica sustains ~900 requests/s within the target. For 4,000/s at least 5 are needed; with ×2 headroom for the HPA and noise, minReplicas: 6 during the campaign and maxReplicas: 20 as the ceiling. With requests.cpu: 200m, CPU at 60% is crossed long before the 150 requests/s per pod, so the CPU metric will be the one that scales in practice.
• An isolated max=2.1s with a healthy p(95) is usually a pod startup or a GC; it is checked in the trace (06-02) before worrying.
• If the p95 does not drop when adding replicas, the bottleneck is behind: MongoDB, the network or the gateway. That is where the following sections come in.
• Orders ×3 is 1.5 orders/s: its 2 replicas are more than enough; what has to be watched in Orders is the saga (RabbitMQ, Inventory, Payments), not HTTP.

The cheapest way to serve 4,000 requests/s is not to compute them. GET /v1/products?ids= in 03-01 already returns Cache-Control: public, max-age=30; now we add caching in two layers:

Cache-aside in productsService with Redis. Before going to MongoDB, Redis is checked; on a miss, it is read, stored with a TTL and returned:

• mget and pipeline batch the operations: a request with 3 ids makes 2 round trips to Redis, not 6.
• The 30 s TTL matches the HTTP max-age: nobody sees a price older than what we were already promising.
• Event-driven invalidation: Catalog consumes its own product.updated event (published when the Shopping Experience team changes a product) and runs redis.del('product:p-501'); that way a price change is visible instantly and not after 30 s. A cache_hits_total{result="hit|miss"} counter (06-01) measures effectiveness: in Catalog, with a 30 s TTL, > 95% hits are expected.
• If Redis does not respond, the cache is skipped (50 ms timeout and a catch that logs warn and goes to MongoDB): the cache can never be a cause of failure (06-03).

Does this break Marta's rule from 04-01 ("two database engines, PostgreSQL and MongoDB, period")? No: Redis here is a cache, not a database: it stores nothing that is not in MongoDB, it can be flushed at any moment (FLUSHDB) and the service keeps working. The rule is about where the truth lives, not about which processes exist in the cluster. Redis is deployed with Helm in techcorp (one instance with a replica, 512 Mi) and configured with REDIS_URL in the ConfigMap.

HTTP caching at the gateway and CDN. Traefik (or a CDN in front) can cache public, max-age=30 responses of GET /v1/products by URL: 90% of the reads of a popular product on Black Friday do not even reach Catalog. It requires the cache key to be stable (the ids sorted: the client from 04-04 already sorts them) and personalized responses (GET /v1/orders, with Authorization) to carry Cache-Control: private, no-store.

Each replica opens a pool of connections to PostgreSQL. The math to do before raising maxReplicas:

For orders-service: pg.Pool({ max: 10 }) × 2 replicas + relay and watchdog on the same pool = 20 connections. With maxReplicas: 10 it would be 100 from Orders alone, and PostgreSQL ships with max_connections = 100 by default for all the services sharing the instance (Customers, Payments, Inventory have separate databases but, in dev, the same server). Options:

• Lower the pool's max: for a Node service with 100 ms per query, 10 connections sustain ~100 queries/s per replica; it is usually plenty. Orders moves to max: 5.
• Put a pooler in front (PgBouncer in transaction mode): 200 application connections are multiplexed onto 20 real ones. It is the norm with many replicas; Platform adds it when any service exceeds 8 replicas.
• Configure pool timeouts (connectionTimeoutMillis: 2000, idleTimeoutMillis: 30000) so that waiting for a connection is a fast error (06-03), not a hang.
• Watch saturation (USE, 06-01): prom-client can expose pg_pool_waiting (pool.waitingCount) and pg_pool_active; a growing wait queue with low CPU is the signature of a small pool.

The same applies to the MongoDB driver (maxPoolSize, 100 by default: too much for 20 replicas → 20) and to RabbitMQ channels (one per consumer and one for publishing, not one per message).

Scaling replicas does not fix a slow query: it multiplies it. Tools in order of use:

EXPLAIN ANALYZE in PostgreSQL. The get(id) query in Orders (04-04) does a LEFT JOIN to customers_ref (the local copy of customer data maintained by the orders.customers consumer):

Two Index Scans: perfect. What to fear is a Seq Scan over orders in the per-customer listing query (WHERE customer_id = $1 ORDER BY created_at DESC), which at 3,000 orders/day walks a million rows within a year: CREATE INDEX orders_customer_created_idx ON orders (customer_id, created_at DESC) and the cursor pagination from 03-01 (WHERE (created_at, id) < ($cursor…)) use the same index and keep the cost constant even as the table grows. The pg_stat_statements extension tells which queries consume the most total time; it is the first thing to look at when the DB is high on CPU.

Indexes in MongoDB. ensureIndexes() from 04-02 creates { _id } (implicit) and { category: 1, name: 1 }; db.products.find({...}).explain('executionStats') must show IXSCAN, not COLLSCAN, and totalDocsExamined close to nReturned.

A single thread. Node serves thousands of connections with one thread because it never blocks it; a 100 ms synchronous operation (the computeRecommendations that the trace in 06-02 uncovered, a 20 MB JSON.parse, synchronous bcrypt, a catastrophic regex) stops all the requests of that replica for 100 ms. The nodejs_eventloop_lag_seconds metric (06-01) gives it away; the remedy is to make it asynchronous, move it out to an event, or, if it is pure and indispensable computation, move it to worker_threads (a mention: a thread pool for CPU-intensive tasks; at TechCorp there is none).

Details that add up: compression() in Express for large responses (product listings: 60-80% fewer bytes), keep-alive in HTTP clients (Node 20's fetch does it by default: it avoids a TCP handshake per request between Orders and Catalog), not serializing huge bodies into events (order.created carries ids and quantities, not the whole catalog) and RabbitMQ prefetch in line with the cost of the message: 10 for Inventory (fast queries), 2-3 for Payments (multi-second calls to the PSP).

Several Inventory replicas consuming inventory.orders is the competing consumers pattern: RabbitMQ distributes messages round-robin among the connected consumers, each message is delivered to one, and throughput grows almost linearly until the DB says enough. It is what KEDA exploits in section 3.

The price is ordering: two messages for the same order (order.created and, a second later, order.cancelled by a customer who changed their mind) can go to different replicas and be processed in reverse order. How it is handled at TechCorp:

• Design consumers for disorder: the Orders state machine ignores impossible transitions (payment.confirmed on CANCELLED does nothing, 06-03) and every event carries occurred_at to discard those older than the current state. It is the solution we already have and it suffices for 99% of cases.
• Partition by key when ordering is indispensable: RabbitMQ's x-consistent-hash exchange (or Kafka with partitions by orderId, 03-02) sends all messages of the same orderId to the same queue/consumer. More complex; TechCorp does not need it today.
• The opposite case: a single consumer guarantees ordering but does not scale; only for very low-volume queues.

Scaling RabbitMQ itself: in production, a 3-node cluster with quorum queues (x-queue-type: quorum, replicated via Raft: a downed node does not lose the queue) instead of classic ones; Platform declares it in the 05-02 chart and the code does not change.

Services scale on their own; the database is the ceiling. Levers, from least to most effort:

1. Fewer queries: cache (section 6), correct pools (7), indexes (8). Solves most cases.
2. Read replicas: PostgreSQL with streaming replication; the OrderRepository uses two pools (ORDERS_DB_URL for writes and the saga, ORDERS_DB_READ_URL for the query views of the CQRS from 02-05: GET /v1/orders?customerId=). Price: replication lag from milliseconds to seconds; a GET /v1/orders/{id} right after the POST must go to the primary or accept Retry-After (03-01 already returns 202 and Location, so the client waits). On a managed provider it is a checkbox.
3. Partitioning (sharding, a mention): splitting orders by date range (PARTITION BY RANGE (created_at), monthly, for archiving) or by hash of customer_id across several instances. TechCorp will evaluate it when the table exceeds tens of millions of rows; today it is unnecessary. MongoDB has native sharding by key, with the same caveat.
4. Vertical scaling of the instance: the legitimate lever for databases: more CPU, RAM and IOPS. It costs money, not complexity.

Rule: measure first (pg_stat_statements, query latency in the traces from 06-02), then choose the cheapest lever that fixes the symptom.

• HPA on a Deployment with replicas pinned in Git. Argo CD (05-03) and the HPA fight: one sets 2, the other 8, every minute. Remove replicas from the manifest (or ignoreDifferences in Argo).
• Made-up requests. With requests.cpu: 1000m for a service that uses 50m, the HPA never exceeds 5% and never scales; with 10m, it scales to the maximum right away. Measure.
• Scaling the service and forgetting the database. 20 replicas × 10 connections = 200 > max_connections. The math from section 7 before raising the maximum.
• Cache without invalidation or TTL. Stale prices for hours. TTL always; invalidation event when it matters.
• A cache that becomes a dependency. If Redis goes down and the service fails, the cache has stopped being a cache. Short timeout and fallback to the DB.
• Load tests against production or from a laptop. The results mean nothing (or take production down). Staging with realistic data and size, load generator inside the cluster or on a dedicated machine.
• Looking at the average. avg=48ms hides a p95 of 240 and a p99 of 900. Percentiles always.
• Adding replicas to fix a query with no index. The DB gets worse. Index first.
• Tip: run the k6 test before every campaign and store the summary next to the image tag; comparing with the previous one catches regressions.
• Tip: every service documents its "capacity sheet": requests/s per replica at the target p95, requests/limits, pool size, HPA min/max and the date of the last load test.

One orders-service replica sustains 40 POST /v1/orders/s at p95 < 500 ms with pg.Pool({ max: 5 }). For Black Friday, HTTP peaks of 1.5 orders/s are expected plus the saga consumer and the relay. Propose minReplicas/maxReplicas, compute the maximum connections to PostgreSQL and say whether PgBouncer is needed.

notifications-service consumes notifications.orders and sends emails through a provider that allows 20 sends/s per connection. Write the ScaledObject with justified values and explain which limit prevents raising maxReplicaCount at will.

During the k6 test, Catalog's p95 goes from 90 ms at 200 VUs to 1.2 s at 1,000 VUs; the HPA has scaled to 12 replicas, the average CPU of the pods is 25% and cache_hits_total{result="hit"} is at 40%. Where is the bottleneck and which two things would you check first?

Exercise 1

1.5 orders/s is 4% of one replica's capacity: minReplicas: 2 (availability, not capacity) and maxReplicas: 4 is more than enough for spikes and for the asynchronous work. Connections: 4 replicas × 5 = 20 to the primary, plus the occasional migrations Job (05-02) and the watchdog/relay that share the pool: ~20-22. With max_connections = 100 shared by Orders, Customers, Payments and Inventory, Orders consumes a fifth: acceptable, PgBouncer is not needed yet. It is noted on the capacity sheet that if maxReplicas exceeds 8 or more services are added to the same instance, it gets reviewed.

Exercise 2

value: 100 because an email takes ~50 ms and 100 messages are drained in 5 s per replica; there is no business urgency. The real limit is the email provider: 20 sends/s per connection and probably a global quota; 5 replicas × 20 = 100/s, which is already close to the contracted quota. Raising maxReplicaCount without raising the quota would only produce 429s from the provider (which the consumer would treat as transient and send to notifications.orders.retry, 06-03). The limit comes from the external dependency, not from the cluster.

Exercise 3

Replicas at 25% CPU and latency through the roof: the bottleneck is not Catalog, it is behind it. The clue is the cache: 40% hits is very low for a 30 s TTL with popular products; 60% of the requests go down to MongoDB. Check first (1) Redis: is it responding, or is the 50 ms timeout firing (warn in Loki "cache unavailable") so that everything goes to the DB?; do the ids arrive in a different order and generate different keys? (here the keys are per product, so it would be the pipeline execution); (2) MongoDB: db.currentOp(), CPU of the instance and explain of findByIds: with _id indexed it should be fast; if the instance is saturated with connections (12 replicas × maxPoolSize 100 = 1,200), that is the problem: lower maxPoolSize to 20 and, if it is Redis, fix it before rerunning the test. Adding more replicas would not help; it would probably make it worse.

Scaling at TechCorp stops being "a bigger machine" and becomes a set of measurable decisions. A template service is scalable because it keeps no state, limits its connections and starts and stops cleanly; the HorizontalPodAutoscaler of catalog-service (2-20 replicas, CPU at 60% and http_requests_per_second via Prometheus Adapter, with asymmetric behavior) and the KEDA ScaledObject for inventory-service on inventory.orders decide the replicas; real requests, PodDisruptionBudget and the Cluster Autoscaler support those decisions; k6 (tests/load/catalog.js, p(95)<300) checks them before Black Friday. Underneath, performance is won with cache-aside caching in Redis (30 s TTL, invalidation via product.updated, never a dependency), connection pools that fit within max_connections, indexes checked with EXPLAIN ANALYZE, an event loop with no blocking, competing consumers that tolerate disorder and read replicas for the query views. We have observability, resilience and capacity; what is missing is agreeing on what "going well" means and what to do when it does not: how much latency and how many errors we accept, when the on-call phone should ring, and how an incident is managed and learned from. That is the close of the module: SLOs, alerts and incident management.