The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Performance Tuning

Next ►
►

The previous lesson closed with an uncomfortable question: is all of it really necessary?

Throughout the module we have taken as given a set of numbers we had been carrying along. That one bookings-api replica sustains 85 requests per second. That we need 30 replicas at the peak. That the correct requests.cpu is 412m. But nobody has asked why one replica only handles 85 requests per second, nor whether it could handle 200 with exactly the same resources.

And that question matters a great deal, because scaling is multiplying inefficiency by the number of replicas. If each replica wastes half its capacity, thirty replicas waste fifteen. You pay for the inefficiency thirty times over, you boot nodes you would not otherwise need, and the real bottleneck — which is almost never where you think it is — stays exactly where it was.

This lesson closes module 9 by looking inward. We are going to look at methodology before tricks, how to run a k6 load test that simulates the May bank holiday and how to read its results properly, tuning the application layer (which is where the problem almost always is), the real mechanics of CPU throttling, fast start-up as an autoscaling requirement, cluster tuning including the ndots: 5 we left pending in 04-03, and a latency budget that turns the business objective into per-component objectives.

Contents

  1. Methodology before tricks
  2. Load testing with k6: the May bank-holiday script
  3. Types of test and when to use each
  4. Running the test inside the cluster
  5. Reading the results correctly
  6. Application tuning: the connection pool
  7. Application tuning: queries, indexes and cache
  8. Application tuning: static assets and persistent connections
  9. Resource tuning: the mechanics of CPU throttling
  10. The debate about removing the CPU limit
  11. Memory and the Node.js garbage collector in a container
  12. The right pod size
  13. Fast start-up as an autoscaling requirement
  14. Cluster tuning: CoreDNS, ndots and kube-proxy
  15. Storage: when the disk is the limit
  16. The latency budget
  17. The measured result of the May bank holiday
  18. Common Mistakes and Tips
  19. Exercises
  20. Conclusion

  1. Methodology before tricks

The internet is full of lists of "20 tricks to speed up Kubernetes". Almost all of them are useless, because performance tuning is not about applying recipes: it is an investigative process.

The four rules

Rule 1: measure before touching anything.

Without a starting measurement, you will not know whether your change improved something, made it worse, or did nothing at all. The subjective feeling that "it's faster now" is completely unreliable.

The Rutas Norte baseline, taken in rutas-norte-pre with a controlled load test:

BASELINE (2026-04-10, rutas-norte-pre, 4 bookings-api replicas)

Sustained requests per second:          340 rps
p50 latency:                             42 ms
p95 latency:                            287 ms
p99 latency:                            891 ms
Error rate:                            0.02%
Average CPU per replica:                310m of 412m (75%)
Active connections to PostgreSQL:        78 of 200

That block is the point of comparison for everything that comes after. Save it, date it and write down the exact conditions.

Rule 2: find the real bottleneck.

In any system there is one resource that runs out before the others. Anything that is not that resource does not matter. Optimising something that is not the bottleneck produces exactly zero improvement.

flowchart LR
    U[User] --> I[Ingress<br/>nginx]
    I --> W[web-store]
    W --> A[bookings-api]
    A --> R[redis-cache]
    A --> P[bookings-postgres]
    A --> M[pricing-engine]

    style P fill:#f88,stroke:#900,stroke-width:3px

If the bottleneck is PostgreSQL, you can optimise nginx all you like: nothing will change. And worse: you can scale bookings-api to thirty replicas and make things worse, because thirty replicas open thirty connection pools against the same saturated database. It is exactly the mistake we warned about in 09-01.

Rule 3: change one thing at a time.

If you change the connection pool, add an index and raise the CPU limit all at once, and performance improves by 40%, you do not know which of the three did it. Worse: perhaps two helped and one hurt, and you are carrying a harmful change without knowing.

One change. Measure. Write it down. Next.

Rule 4: measure again under the same conditions.

Same script, same duration, same environment, same data. If you compare a Tuesday 10:00 run with a Saturday 3:00 one, you are comparing noise.

The full cycle

flowchart TD
    A[1. Define the objective<br/>SLO: p95 < 300 ms at 95%] --> B[2. Measure the baseline<br/>controlled load test]
    B --> C[3. Identify the bottleneck<br/>module 7 metrics]
    C --> D[4. Formulate ONE hypothesis<br/>'the pool is too large']
    D --> E[5. Apply ONE change<br/>in rutas-norte-pre]
    E --> F[6. Measure under the same conditions]
    F --> G{Better?}
    G -->|Yes| H[7. Write it down, take it to production,<br/>verify in Grafana]
    G -->|No| I[Revert. The hypothesis was false.<br/>Write that down too: it is information]
    H --> J{Objective<br/>reached?}
    I --> C
    J -->|No| C
    J -->|Yes| K[END. Document and stop]

    style D fill:#cde,stroke:#369
    style K fill:#cfc,stroke:#393

Note the final step: when the objective is reached, you stop. Performance tuning without a defined objective is a bottomless pit: you can always optimise more, with ever-diminishing returns. The SLO of module 7 is what tells you when you are done.

The tools you already have

The whole toolkit comes from module 7:

Question Tool
What is the real latency by percentile? Prometheus + histogram_quantile (07-03)
Which component eats the latency? Distributed traces, or per-component histograms
How much CPU and memory does each pod use? kubectl top and metrics-server (07-02)
Is there CPU throttling? container_cpu_cfs_throttled_seconds_total
What is the application waiting for? Logs with per-phase timings (07-05)
What is happening in the cluster? Events and kubectl describe (07-06)
Is the SLO being met? Grafana dashboards and the error budget (07-04)

Before touching anything, those dashboards must exist and be populated. Optimising blind is guesswork.

  1. Load testing with k6: the May bank-holiday script

k6 is a load-testing tool whose scenarios are written in JavaScript and executed by a high-performance Go engine. It is the natural choice on Kubernetes: it is a single binary, it works well inside a container and it exports metrics to Prometheus.

A realistic script

The key to a useful test is that it resembles real traffic. A loop hammering a single endpoint tells you nothing: the real Rutas Norte load is a sequence of actions with pauses between them and different proportions.

// tests/load/may-bank-holiday.js
import http from 'k6/http';
import { check, sleep, group } from 'k6';
import { Rate, Trend, Counter } from 'k6/metrics';

// ---------------------------------------------------------------------------
// CUSTOM METRICS
// k6's default metrics measure HTTP requests. These measure
// BUSINESS OPERATIONS, which is what Rutas Norte actually cares about.
// ---------------------------------------------------------------------------
const bookingErrors = new Rate('booking_errors');
const searchDuration = new Trend('route_search_duration', true);
const bookingDuration = new Trend('confirm_booking_duration', true);
const bookingsConfirmed = new Counter('bookings_confirmed');

// ---------------------------------------------------------------------------
// SCENARIO CONFIGURATION
// ---------------------------------------------------------------------------
export const options = {
  scenarios: {
    // SCENARIO 1: the avalanche when the sale opens.
    // It reproduces what happens on 25 April at 10:00.
    sale_opening: {
      executor: 'ramping-arrival-rate',
      // ramping-arrival-rate keeps a constant ARRIVAL RATE,
      // regardless of whether the system responds fast or slow.
      // That is the critical difference from ramping-vus: real users do NOT
      // wait for you to finish serving the previous one before they arrive.
      startRate: 20,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 3000,
      stages: [
        { duration: '2m',  target: 20   },  // Calm before. Baseline.
        { duration: '40s', target: 500  },  // THE AVALANCHE: x25 in 40 seconds
        { duration: '10m', target: 500  },  // Plateau: two hours compressed
        { duration: '5m',  target: 120  },  // Gradual descent
        { duration: '3m',  target: 20   },  // Back to calm
      ],
      gracefulStop: '30s',
    },
  },

  // ---------------------------------------------------------------------------
  // THRESHOLDS: if they are not met, k6 exits with an error code.
  // This turns the test into something that can FAIL in a CI pipeline.
  // ---------------------------------------------------------------------------
  thresholds: {
    // The Rutas Norte SLO: p95 below 300 ms.
    'http_req_duration': ['p(95)<300', 'p(99)<1000'],
    // Less than 1% errors on bookings. That is what costs money.
    'booking_errors': ['rate<0.01'],
    // Route search is the most frequent operation: more demanding.
    'route_search_duration': ['p(95)<200'],
    // Confirming a booking writes to PostgreSQL: it gets more leeway.
    'confirm_booking_duration': ['p(95)<800'],
    // Less than 0.5% HTTP failures overall.
    'http_req_failed': ['rate<0.005'],
  },
};

// ---------------------------------------------------------------------------
// TEST DATA (fictitious)
// ---------------------------------------------------------------------------
const ORIGINS = ['BIL', 'SDR', 'OVD', 'GIJ', 'SAN', 'VIT', 'LOG', 'PAM'];
const DESTINATIONS = ['MAD', 'BCN', 'ZAZ', 'VLC', 'SVQ', 'BIO', 'LCG'];

const BASE = __ENV.BASE_URL || 'http://bookings-api.rutas-norte-pre.svc.cluster.local';

function randomFrom(list) {
  return list[Math.floor(Math.random() * list.length)];
}

function bankHolidayDate() {
  // Bank-holiday bookings concentrate on three specific days.
  const days = ['2026-04-30', '2026-05-01', '2026-05-02', '2026-05-03'];
  return randomFrom(days);
}

// ---------------------------------------------------------------------------
// THE USER JOURNEY
// It reproduces what a real person does: search, look, choose, confirm.
// ---------------------------------------------------------------------------
export default function () {
  const origin = randomFrom(ORIGINS);
  const destination = randomFrom(DESTINATIONS);
  const date = bankHolidayDate();

  let routeId = null;

  // STEP 1: search for routes. It is the MOST FREQUENT operation (everybody searches).
  group('01_search_routes', function () {
    const start = Date.now();
    const res = http.get(
      `${BASE}/routes?origin=${origin}&destination=${destination}&date=${date}`,
      { tags: { operation: 'search_routes' } }
    );
    searchDuration.add(Date.now() - start);

    const ok = check(res, {
      'search returns 200': (r) => r.status === 200,
      'search returns routes': (r) => {
        try {
          return JSON.parse(r.body).routes !== undefined;
        } catch (e) {
          return false;
        }
      },
    });

    if (ok && res.status === 200) {
      try {
        const routes = JSON.parse(res.body).routes;
        if (routes && routes.length > 0) {
          routeId = randomFrom(routes).id;
        }
      } catch (e) { /* malformed body: counted by the check */ }
    }
  });

  // PAUSE: the user looks at the results. Between 2 and 6 seconds.
  // THIS SLEEP IS ESSENTIAL. Without it, the test generates a load pattern
  // that looks nothing like the real one, and the conclusions are worthless.
  sleep(Math.random() * 4 + 2);

  if (!routeId) {
    // No routes for that combination. The user leaves. That is realistic:
    // not everybody finds what they are looking for.
    return;
  }

  // STEP 2: check seat availability.
  // This query SHOULD be served from redis-cache.
  group('02_check_seats', function () {
    const res = http.get(
      `${BASE}/routes/${routeId}/seats`,
      { tags: { operation: 'check_seats' } }
    );
    check(res, { 'seats returns 200': (r) => r.status === 200 });
  });

  sleep(Math.random() * 3 + 1);

  // STEP 3: confirm the booking.
  // Only 18% of searches end in a booking. That is the real Rutas Norte
  // conversion rate, measured in production.
  if (Math.random() > 0.18) {
    return;
  }

  group('03_confirm_booking', function () {
    const start = Date.now();
    const payload = JSON.stringify({
      routeId: routeId,
      seat: Math.floor(Math.random() * 54) + 1,
      passenger: {
        name: `Test Customer ${__VU}-${__ITER}`,
        documentId: `00000000${(__VU % 10)}X`,
        email: `test-${__VU}-${__ITER}@ejemplo.example`,
      },
    });

    const res = http.post(`${BASE}/bookings`, payload, {
      headers: { 'Content-Type': 'application/json' },
      tags: { operation: 'confirm_booking' },
    });

    bookingDuration.add(Date.now() - start);

    const ok = check(res, {
      'booking returns 201': (r) => r.status === 201,
      'booking returns a reference': (r) => {
        try {
          return JSON.parse(r.body).reference !== undefined;
        } catch (e) {
          return false;
        }
      },
    });

    bookingErrors.add(!ok);
    if (ok) bookingsConfirmed.add(1);
  });

  sleep(1);
}

// ---------------------------------------------------------------------------
// FINAL SUMMARY
// ---------------------------------------------------------------------------
export function handleSummary(data) {
  const m = data.metrics;
  const line = (label, value) => `  ${label.padEnd(38)} ${value}\n`;

  let output = '\n=== LOAD TEST: MAY BANK HOLIDAY ===\n\n';
  output += line('Total requests', m.http_reqs.values.count);
  output += line('Requests per second (average)', m.http_reqs.values.rate.toFixed(1));
  output += line('Bookings confirmed', m.bookings_confirmed ? m.bookings_confirmed.values.count : 0);
  output += '\n  --- OVERALL LATENCY ---\n';
  output += line('p50', m.http_req_duration.values['p(50)'].toFixed(1) + ' ms');
  output += line('p95', m.http_req_duration.values['p(95)'].toFixed(1) + ' ms');
  output += line('p99', m.http_req_duration.values['p(99)'].toFixed(1) + ' ms');
  output += line('max', m.http_req_duration.values.max.toFixed(1) + ' ms');
  output += '\n  --- ERRORS ---\n';
  output += line('HTTP failure rate', (m.http_req_failed.values.rate * 100).toFixed(3) + ' %');

  return {
    'stdout': output,
    'results/may-bank-holiday.json': JSON.stringify(data, null, 2),
  };
}

The three details that make this script useful

1. ramping-arrival-rate instead of ramping-vus. The difference is fundamental and very few people know it:

Executor What it keeps constant Behaviour if the system slows down
ramping-vus The number of virtual users The load DROPS on its own. Each user waits longer, so it makes fewer requests
ramping-arrival-rate The arrival rate (requests/s) The load holds. k6 creates more virtual users to keep the pace

With ramping-vus, a degrading system generates less load, which hides the problem: the test throttles itself. Real users do not coordinate to wait. If you open the sale at 10:00, they arrive at 10:00 even if your API takes 4 seconds to answer.

Use ramping-arrival-rate for any test that aims to reproduce real traffic.

2. The sleep()s between steps. A real user searches, looks at the results for 4 seconds, chooses, and confirms. Without those pauses, the test generates a completely different concurrency pattern: much less concurrency per user and many more requests per second with fewer open connections. The results would be optimistic and misleading.

3. The 18% conversion rate. Only 18% of searches end in a booking. If the test made one booking per search, it would overload PostgreSQL with writes far beyond reality and conclude that the database is the bottleneck when it is not in production.

The ratio between operations is as important as the total volume.

  1. Types of test and when to use each

Not every load test answers the same question.

Type What it does What it answers Typical duration
Smoke Minimal load, 1-5 users Does the system work, and does the script? 1-2 min
Load Sustained expected load Does it meet the SLO under normal conditions? 15-30 min
Stress Increasing load until it breaks Where is the limit? How does it break? 20-40 min
Spike A sharp, short jump Does it survive the sale opening? 5-10 min
Soak Normal load for a very long time Are there memory leaks or degradation? 4-24 h

The smoke test

Always the first one. It checks that the script has no errors and that the system responds.

export const options = {
  vus: 3,
  duration: '2m',
  thresholds: {
    'http_req_failed': ['rate<0.01'],
    'http_req_duration': ['p(95)<500'],
  },
};

It costs two minutes and prevents you from discovering fifteen minutes into a stress test that the URL was wrong.

The load test

The reference test. It verifies that the system meets the SLO with the expected load.

export const options = {
  scenarios: {
    normal_load: {
      executor: 'constant-arrival-rate',
      rate: 120,               // 120 requests per second, the usual traffic
      timeUnit: '1s',
      duration: '20m',
      preAllocatedVUs: 100,
      maxVUs: 500,
    },
  },
  thresholds: {
    'http_req_duration': ['p(95)<300'],
    'http_req_failed': ['rate<0.001'],
  },
};

The stress test

The most informative of them all. It raises the load in steps until the system breaks, and what matters is not only where it breaks, but how.

export const options = {
  scenarios: {
    stress: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 500,
      maxVUs: 5000,
      stages: [
        { duration: '3m', target: 100  },
        { duration: '3m', target: 200  },
        { duration: '3m', target: 400  },
        { duration: '3m', target: 600  },
        { duration: '3m', target: 800  },
        { duration: '3m', target: 1200 },
        { duration: '5m', target: 0    },   // Recovery: does it recover on its own?
      ],
    },
  },
  // NO thresholds: we want it to run to the end even if it fails.
};

What to watch for:

Observation What it means
The inflection point The rps beyond which latency explodes. That is the real capacity
The shape of the degradation Gradual (good) or all at once (bad)?
Behaviour under saturation Fast errors (good) or long timeouts (bad)?
Recovery When the load drops, does it return to normal or stay degraded?

That last point is critical and often reveals serious problems. A system that does not recover on its own after a spike has a structural problem: an internal queue that never drains, connections that are never released, an exhausted pool that never regenerates. In production that means a five-minute peak leaves the system broken for hours.

The spike test

It reproduces the sale opening exactly.

export const options = {
  scenarios: {
    spike: {
      executor: 'ramping-arrival-rate',
      startRate: 20,
      timeUnit: '1s',
      preAllocatedVUs: 100,
      maxVUs: 3000,
      stages: [
        { duration: '2m',  target: 20  },   // Calm
        { duration: '30s', target: 800 },   // BRUTAL SPIKE
        { duration: '3m',  target: 800 },   // Sustained
        { duration: '2m',  target: 20  },   // Back to calm
      ],
    },
  },
};

It is the test that validates the whole of module 9. With it you check whether the HPA reacts in time, whether the over-provisioning cushion absorbs the blow, and whether KEDA's cron trigger does its job. Run it twice: with the cron pre-warming enabled and without it, and compare.

The soak test

The most boring one, and the one that catches the most expensive problems.

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: 100,
      timeUnit: '1s',
      duration: '8h',            // EIGHT HOURS
      preAllocatedVUs: 100,
      maxVUs: 300,
    },
  },
};

What you are looking for:

  • Memory leaks: pod memory grows monotonically until the OOMKilled.
  • Connection leaks: connections to PostgreSQL grow and are never released.
  • Gradual degradation: p95 latency grows 5% every hour.
  • Fragmentation: throughput falls without any resource rising.

You run it once before an important event. It is the test that stops the platform holding up nicely for the first two hours of the May bank holiday and falling over in the third.

  1. Running the test inside the cluster

Running k6 from your laptop mostly measures your internet connection. To measure the platform, the test must run inside the cluster, against rutas-norte-pre.

As a Kubernetes Job

# k8s/tests/job-load-may-bank-holiday.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: load-script
  namespace: rutas-norte-pre
data:
  may-bank-holiday.js: |
    // (the contents of the script from section 2)
---
apiVersion: batch/v1
kind: Job
metadata:
  name: load-may-bank-holiday
  namespace: rutas-norte-pre
  labels:
    app: load-test
    app.kubernetes.io/part-of: rutas-norte
spec:
  backoffLimit: 0        # If it fails, do NOT retry: it would skew the results
  ttlSecondsAfterFinished: 86400
  template:
    metadata:
      labels:
        app: load-test
    spec:
      restartPolicy: Never

      # The load generator must NOT compete for resources with what it measures.
      # We send it to a dedicated node with a taint and a toleration (06-05).
      tolerations:
        - key: role
          operator: Equal
          value: tests
          effect: NoSchedule
      nodeSelector:
        role: tests

      containers:
        - name: k6
          image: grafana/k6:0.52.0
          command: ["k6", "run"]
          args:
            - "--out"
            - "experimental-prometheus-rw"     # Sends metrics to Prometheus
            - "--tag"
            - "test=may-bank-holiday"
            - "/scripts/may-bank-holiday.js"
          env:
            - name: BASE_URL
              value: "http://bookings-api.rutas-norte-pre.svc.cluster.local"
            - name: K6_PROMETHEUS_RW_SERVER_URL
              value: "http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/write"
            - name: K6_PROMETHEUS_RW_TREND_STATS
              value: "p(50),p(95),p(99),max"
          resources:
            # DELIBERATELY GENEROUS. If k6 runs out of CPU, it does not generate
            # the requested load and the test LIES: you will measure a system that
            # holds up well because it never received the load you thought.
            requests:
              cpu: "2"
              memory: 2Gi
            limits:
              cpu: "4"
              memory: 4Gi
          volumeMounts:
            - name: scripts
              mountPath: /scripts
      volumes:
        - name: scripts
          configMap:
            name: load-script
kubectl apply -f k8s/tests/job-load-may-bank-holiday.yaml
kubectl logs -f job/load-may-bank-holiday -n rutas-norte-pre

The dedicated-node detail is not a luxury. If k6 runs on the same node as bookings-api, they compete for CPU, and you will not know whether the high latency comes from the system or from the load generator. It is a classic mistake that invalidates entire test runs.

And the warning about k6's resources deserves emphasis: a load test with a throttled generator is worse than no test at all, because it produces a false sense of security. Always check in the k6 output that the actual arrival rate matches the requested one.

With the k6 operator

For tests distributed across several pods:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: load-may-bank-holiday
  namespace: rutas-norte-pre
spec:
  parallelism: 4          # Four pods generating load in parallel
  script:
    configMap:
      name: load-script
      file: may-bank-holiday.js
  runner:
    resources:
      requests: {cpu: "2", memory: 2Gi}
      limits: {cpu: "2", memory: 2Gi}

Necessary when a single pod cannot generate enough load (from around 2,000-3,000 rps, depending on the scenario).

  1. Reading the results correctly

This is where most people get it wrong.

The k6 output

     ✓ search returns 200
     ✓ search returns routes
     ✗ booking returns 201
      ↳  97% — ✓ 8214 / ✗ 254

     checks.........................: 98.42% ✓ 47892     ✗ 768
     data_received..................: 1.2 GB  1.8 MB/s
     data_sent......................: 89 MB   134 kB/s
     route_search_duration..........: avg=98.4ms  min=12ms  med=71ms   max=4.2s   p(95)=241ms  p(99)=892ms
     confirm_booking_duration.......: avg=421ms   min=89ms  med=302ms  max=8.9s   p(95)=1.4s   p(99)=4.1s
     booking_errors.................: 2.96%  ✓ 254       ✗ 8214
     http_req_blocked...............: avg=1.2ms   min=0s    med=2µs    max=1.1s   p(95)=4µs
     http_req_connecting............: avg=0.9ms   min=0s    med=0s     max=892ms  p(95)=0s
   ✗ http_req_duration..............: avg=142ms   min=8ms   med=68ms   max=8.9s   p(95)=487ms  p(99)=2.1s
       { expected_response:true }...: avg=131ms   min=8ms   med=66ms   max=6.2s   p(95)=443ms
   ✗ http_req_failed................: 1.34%  ✓ 892       ✗ 65432
     http_reqs......................: 66324  99.4/s
     iteration_duration.............: avg=6.8s    min=3.1s  med=6.4s   max=22s    p(95)=12.4s
     iterations.....................: 14204  21.3/s
     vus............................: 187    min=20      max=1240
     vus_max........................: 3000   min=3000    max=3000

     bookings_confirmed.............: 8214   12.3/s

ERRO[0942] thresholds on metrics 'http_req_duration, http_req_failed, booking_errors' have been crossed

Percentiles versus the average: the fundamental lesson

Look at these two lines:

http_req_duration: avg=142ms   med=68ms   p(95)=487ms   p(99)=2.1s   max=8.9s

The average is 142 ms. The median is 68 ms. The p95 is 487 ms. The p99 is 2.1 seconds.

If you reported "the average latency is 142 ms", you would be giving a false and reassuring picture. The reality:

  • Half the users have an excellent experience (68 ms).
  • 5 out of every 100 wait almost half a second.
  • 1 out of every 100 waits more than 2 seconds.
  • Somebody waited almost 9 seconds.

With 66,324 requests, that p99 is 663 requests taking more than 2 seconds. And at 100 requests per second, that is one horrible request every second, continuously.

The average hides the tail of the distribution, and the tail is where user dissatisfaction lives.

Metric What it says Usefulness
avg (mean) The arithmetic average Almost none. A single outlier distorts it
med / p(50) The median: half are below The typical user's experience
p(95) 95 out of 100 are below The reference metric for the SLO
p(99) 99 out of 100 The experience of the worst 1%. It matters at scale
max The worst case observed Useful for spotting timeouts and anomalies

An effect that surprises a lot of people: in a session with many requests, the p99 becomes the typical experience. If loading the Rutas Norte page makes 30 requests, the probability that at least one of them lands in the p99 is 1 - 0.99³⁰ = 26%. One in four users suffers the p99 on every page load.

That is why SLOs are defined on p95 and p99, never on the average.

The inflection point

In a stress test, this is the table you have to build:

Load (rps) p50 p95 p99 Errors Average CPU
100 41 ms 78 ms 142 ms 0.00% 24%
200 44 ms 91 ms 178 ms 0.00% 47%
300 48 ms 118 ms 241 ms 0.00% 68%
400 58 ms 187 ms 412 ms 0.01% 84%
500 89 ms 421 ms 1,240 ms 0.4% 94%
600 340 ms 2,100 ms 6,800 ms 8.2% 98%
800 1,900 ms 8,400 ms timeout 34.1% 99%

The inflection point is between 400 and 500 rps. Up to 400, latency grows smoothly and proportionally. From 500 on, it explodes non-linearly: from 400 to 500 rps (25% more load) the p95 multiplies by 2.25.

This has a precise theoretical explanation, and knowing it helps: it is queueing theory. When a resource's utilisation approaches 100%, the waiting time tends to infinity. The approximate formula for a simple queue:

Waiting time ≈ ServiceTime × (utilisation / (1 - utilisation))

Utilisation 50%:  wait = 1.0 x service time
Utilisation 80%:  wait = 4.0 x service time
Utilisation 90%:  wait = 9.0 x service time
Utilisation 95%:  wait = 19.0 x service time
Utilisation 99%:  wait = 99.0 x service time

This is the mathematical justification for why the HPA target should be at 60-70% and not at 90% (09-01). At 90% utilisation, the waiting time is already nine times the service time. The 30% margin is not conservatism: it is staying out of the zone where latency explodes.

The real capacity per replica is defined at the inflection point, not at the breaking point. With 4 replicas and an inflection at 450 rps: 112 rps per replica. Applying the 30% margin, KEDA's threshold would be 78 rps. Very close to the 85 we had been using: the module's numbers were well calibrated.

Errors: which ones matter

http_req_failed: 1.34%  ✓ 892  ✗ 65432
booking_errors:  2.96%  ✓ 254  ✗ 8214

1.34% of HTTP failures overall, but 2.96% of failures on bookings. The difference matters enormously: the failures concentrate on the operation that generates revenue.

A failed search is annoying; the user retries. A failed booking is lost money and, potentially, a charge with no ticket.

Always segment errors by business operation, not just globally. That is why the script defines bookingErrors as its own metric.

  1. Application tuning: the connection pool

And now we get to where the problem almost always is.

The arithmetic that takes the database down

This is the most expensive and most frequent mistake in Kubernetes, and it is purely arithmetic.

"Reasonable" bookings-api configuration (Node.js with pg-pool):
  pool.max = 20 connections

Normal situation: 4 replicas.
  Total connections: 4 x 20 = 80.
  bookings-postgres: max_connections = 200.
  80 < 200. All fine.

MAY BANK HOLIDAY: the HPA scales to 30 replicas.
  Total connections: 30 x 20 = 600.
  max_connections = 200.

  400 connections REJECTED.

What happens next is a cascade:

1. Replicas 11 to 30 cannot get a connection.
2. Their requests fail with "too many connections" or wait until timeout.
3. bookings-api returns 500. CPU GOES UP (error handling, retries).
4. The HPA sees more CPU and wants to scale FURTHER. maxReplicas caps it at 30.
5. The 10 replicas that DO have connections receive all the traffic.
6. PostgreSQL, with 200 active connections, spends more time context-switching
   between processes than executing queries.
7. Queries slow down. Connections are held for longer.
8. Total collapse.

Scaling made things worse. With 4 replicas the platform was slow; with 30 it is down.

The rule

max_replicas x pool_per_replica  <=  max_connections x 0.8

The 0.8 factor reserves connections for maintenance tasks, backups, the metrics exporter and administrative connections.

For Rutas Norte:

bookings-api maxReplicas: 30
bookings-postgres max_connections: 200
20% reserve: 200 x 0.8 = 160

pool_per_replica <= 160 / 30 = 5.33  ->  5 connections per replica

Five connections per replica, not twenty.

But are 5 connections enough?

The natural question is whether a replica can serve the traffic with 5 connections. The answer comes from Little's law:

Concurrency = Arrival rate x Service time

With 5 connections and an average query of 8 ms:
  Queries per second = 5 / 0.008 = 625 queries/s per replica

If each HTTP request makes 2 queries on average:
  Requests per second = 625 / 2 = 312 rps per replica

312 rps per replica, far above the 112 rps of the inflection point. The 5-connection pool is not the limiting factor.

This reveals something important: the 20-connection pool was never justified. It was a default value copied without thinking. And not only was it unnecessary: it was the mechanism that was going to take the platform down at the peak.

Correct configuration

# bookings-api ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: bookings-api-config
  namespace: rutas-norte-pro
data:
  # POSTGRESQL CONNECTION POOL
  #
  # Calculation: maxReplicas (30) x POOL_MAX <= max_connections (200) x 0.8 = 160
  #              POOL_MAX <= 160 / 30 = 5.33  ->  5
  #
  # Capacity check (Little's law):
  #   5 connections / 8 ms per query = 625 queries/s per replica
  #   With 2 queries per request: 312 rps per replica.
  #   Measured inflection point: 112 rps per replica.
  #   The pool is NOT the limiting factor. 5 is plenty.
  #
  # DO NOT RAISE without recalculating. With 30 replicas, each extra connection
  # per replica is 30 more connections against a limit of 200.
  PG_POOL_MAX: "5"
  PG_POOL_MIN: "2"

  # Timeout waiting for a free connection. 3 seconds: we prefer to fail
  # fast (the user retries) rather than pile up requests waiting.
  PG_POOL_ACQUIRE_TIMEOUT_MS: "3000"

  # Close idle connections after 30 s: it frees resources in PostgreSQL
  # when the HPA scales replicas down.
  PG_POOL_IDLE_TIMEOUT_MS: "30000"

  # Recycle connections every 30 minutes: it avoids zombie connections and
  # server-side memory leaks.
  PG_POOL_MAX_LIFETIME_MS: "1800000"

The real solution: a shared pool

Five connections per replica works, but it is fragile: any change to maxReplicas forces a recalculation. The robust solution is PgBouncer, a connection multiplexer.

flowchart LR
    subgraph API["30 bookings-api replicas"]
        A1[replica 1<br/>pool: 20]
        A2[replica 2<br/>pool: 20]
        A3[...]
        A30[replica 30<br/>pool: 20]
    end

    PB[PgBouncer<br/>600 client connections<br/>25 to the server<br/>transaction mode]

    PG[(bookings-postgres<br/>max_connections: 200<br/>25 in use)]

    A1 --> PB
    A2 --> PB
    A3 --> PB
    A30 --> PB
    PB --> PG

    style PB fill:#cde,stroke:#369,stroke-width:2px

PgBouncer in transaction mode assigns a server connection only for the duration of a transaction, not for the duration of the client connection. Since transactions last milliseconds and connections last minutes, the multiplexing is enormous: 600 client connections over 25 server ones.

# k8s/base/deployment-pgbouncer.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbouncer
  namespace: rutas-norte-pro
  labels:
    app: pgbouncer
    app.kubernetes.io/part-of: rutas-norte
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pgbouncer
  template:
    metadata:
      labels:
        app: pgbouncer
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: pgbouncer
      containers:
        - name: pgbouncer
          image: bitnami/pgbouncer:1.22.1
          env:
            - name: PGBOUNCER_POOL_MODE
              # TRANSACTION: the server connection is assigned only during
              # a transaction. That is what enables the massive multiplexing.
              # WARNING: incompatible with session prepared statements,
              # named cursors and LISTEN/NOTIFY. The application must be
              # written with that in mind.
              value: "transaction"
            - name: PGBOUNCER_MAX_CLIENT_CONN
              # 1000 CLIENT connections. With 30 replicas x 20 = 600. Plenty.
              value: "1000"
            - name: PGBOUNCER_DEFAULT_POOL_SIZE
              # Only 25 SERVER connections per database and user.
              # With 3 pgbouncer replicas: 75 total connections to PostgreSQL,
              # far below 200. And stable: they do not depend on the HPA.
              value: "25"
            - name: PGBOUNCER_RESERVE_POOL_SIZE
              value: "5"
          resources:
            requests: {cpu: 100m, memory: 128Mi}
            limits: {cpu: 500m, memory: 256Mi}

The decisive advantage: the number of connections to PostgreSQL stops depending on the number of replicas. The HPA can scale to 30, 50 or 100 replicas without PostgreSQL noticing. It is the only robust way to combine aggressive autoscaling with a relational database.

  1. Application tuning: queries, indexes and cache

The N+1 problem

The pattern that most often turns a fast API into a slow one.

// BAD: one query for the routes, and one MORE for each route. N+1.
const routes = await db.query(
  'SELECT id, origin, destination, departure_time FROM routes WHERE origin=$1 AND destination=$2',
  [origin, destination]
);

for (const route of routes) {
  // If there are 30 routes, this is 30 ADDITIONAL queries.
  route.freeSeats = await db.query(
    'SELECT COUNT(*) FROM seats WHERE route_id=$1 AND status=$2',
    [route.id, 'free']
  );
}
// TOTAL: 31 queries to answer ONE request.

With 30 routes and 5 ms per query: 155 ms just in round trips to the database, not counting the time of the queries themselves. And 31 queries per request means that 100 rps become 3,100 queries per second against PostgreSQL.

// GOOD: a SINGLE query with aggregation.
const routes = await db.query(`
  SELECT
    r.id, r.origin, r.destination, r.departure_time,
    COUNT(s.id) FILTER (WHERE s.status = 'free') AS free_seats
  FROM routes r
  LEFT JOIN seats s ON s.route_id = r.id
  WHERE r.origin = $1 AND r.destination = $2 AND r.date = $3
  GROUP BY r.id, r.origin, r.destination, r.departure_time
  ORDER BY r.departure_time
`, [origin, destination, date]);
// TOTAL: 1 query.

From 31 queries to 1. The measured impact at Rutas Norte: from 155 ms to 12 ms on the route search. A factor of 13.

How to spot it: if the number of queries per request varies with the size of the result, you have an N+1. It is visible in distributed traces (07-03) as a staircase of identical calls.

Missing indexes

-- Diagnosis: how does PostgreSQL execute this query?
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM routes WHERE origin = 'BIL' AND destination = 'MAD' AND date = '2026-05-01';
Seq Scan on routes  (cost=0.00..18432.00 rows=12 width=84) (actual time=0.031..89.412 rows=8 loops=1)
  Filter: ((origin = 'BIL') AND (destination = 'MAD') AND (date = '2026-05-01'))
  Rows Removed by Filter: 847992
  Buffers: shared hit=2841 read=5591
Planning Time: 0.184 ms
Execution Time: 89.487 ms

A Seq Scan with Rows Removed by Filter: 847992. PostgreSQL is reading all 848,000 rows of the table to return 8. Every single time.

-- The composite index that resolves the query
CREATE INDEX CONCURRENTLY idx_routes_search
  ON routes (origin, destination, date)
  INCLUDE (departure_time, base_price);
Index Scan using idx_routes_search on routes  (cost=0.42..8.61 rows=12 width=84)
                                              (actual time=0.024..0.041 rows=8 loops=1)
  Index Cond: ((origin = 'BIL') AND (destination = 'MAD') AND (date = '2026-05-01'))
  Buffers: shared hit=5
Planning Time: 0.211 ms
Execution Time: 0.068 ms

From 89.5 ms to 0.068 ms. A factor of 1,316.

Three important details about the index:

  1. CONCURRENTLY: it builds the index without blocking writes. It takes longer but does not interrupt the service. Mandatory in production.
  2. The column order matters: the most selective equality columns first. An index on (origin, destination, date) serves queries by origin, by origin+destination and by all three; it does not serve a query by date alone.
  3. INCLUDE: it adds columns to the index without making them part of the key, enabling an index-only scan that never touches the table.

How to find the problem queries:

-- Requires the pg_stat_statements extension
SELECT
  substring(query, 1, 80) AS query_text,
  calls,
  round(mean_exec_time::numeric, 2) AS mean_ms,
  round(total_exec_time::numeric / 1000, 1) AS total_s,
  rows / GREATEST(calls, 1) AS rows_per_call
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 15;

Order by total_exec_time, not by mean_exec_time. A 2 ms query executed a million times consumes far more total time than a 500 ms one executed a hundred times. Total time is what saturates the database.

Really using redis-cache

redis-cache has been on the platform since module 6, but we have to check whether it is being used properly.

kubectl exec -n rutas-norte-pro redis-cache-0 -- redis-cli INFO stats | grep keyspace
keyspace_hits:1284729
keyspace_misses:8471203
Hit ratio = 1,284,729 / (1,284,729 + 8,471,203) = 13.2%

A 13% hit ratio is a disaster. 87% of the queries go to PostgreSQL anyway, and on top of that you pay the cost of asking Redis first. The cache is doing harm, not good.

The usual diagnosis: keys that are too specific.

// BAD: the key includes the exact timestamp of the request.
// EVERY request generates a different key. Hit ratio ~0%.
const key = `seats:${routeId}:${Date.now()}`;

// BAD: it includes the user id, which does not affect the result.
const key = `routes:${origin}:${destination}:${date}:${userId}`;

// GOOD: only the parameters that determine the result.
const key = `routes:${origin}:${destination}:${date}`;

The caching strategy by data type at Rutas Norte:

Data TTL Justification
Route and timetable catalogue 1 hour Changes with the seasonal timetable, almost never
Seat availability 10 seconds Changes constantly; 10 s is tolerable and absorbs 95% of the reads
Computed prices 5 minutes The pricing engine recalculates every few minutes
User data 15 minutes Rarely changes; invalidated on edit
The result of a booking Never It is a write; caching a write is a mistake

The 10-second TTL for availability deserves an explanation, because it looks short. With 500 queries per second on the same popular route:

No cache: 500 queries/s to PostgreSQL.
With a 10 s TTL: 1 query every 10 s = 0.1 queries/s to PostgreSQL.

Reduction: 5,000 times.
Cost: the data can be up to 10 seconds out of date.

Is it acceptable to show "12 seats left" when there are really 11? Yes, as long as booking confirmation verifies real availability against the database. The cache is for displaying; the truth is checked on write.

Redis configuration so that it behaves as a cache and not as a database:

# redis-cache ConfigMap
maxmemory 1500mb
maxmemory-policy allkeys-lru
# allkeys-lru: when full, it evicts the least recently used keys.
# WITHOUT this, Redis grows until the OOMKilled (we saw it in 09-02).

  1. Application tuning: static assets and persistent connections

Compression and caching in web-store

# web-store ConfigMap: /etc/nginx/conf.d/rutas-norte.conf

# --- COMPRESSION ---
gzip on;
gzip_vary on;
gzip_min_length 1024;          # Do not compress small things: the CPU cost is not worth it
gzip_comp_level 5;             # 5 out of 9: a good balance. 9 burns a lot of CPU
                               # for 2-3% more compression
gzip_types
    text/plain text/css text/javascript
    application/javascript application/json
    application/xml image/svg+xml;

# Brotli compresses 15-20% better than gzip for text.
# PRECOMPRESSED at build time: ZERO CPU cost at runtime.
brotli_static on;
gzip_static on;

# --- ASSET CACHING ---
# Files with a hash in the name (app.4f8a2c.js) are IMMUTABLE:
# if the content changes, the name changes. They can be cached for a year.
location ~* \.(js|css|woff2|png|jpg|svg)$ {
    expires 1y;
    add_header Cache-Control "public, immutable";
    access_log off;            # Do not log assets: it cuts I/O by 80%
}

# The HTML is NOT cached: it is what points at the hashed files.
location = /index.html {
    expires -1;
    add_header Cache-Control "no-cache, must-revalidate";
}

# --- PERSISTENT CONNECTIONS TOWARDS bookings-api ---
upstream bookings_api {
    server bookings-api.rutas-norte-pro.svc.cluster.local:8080;
    # 64 reused persistent connections. Without this, nginx opens and closes
    # a TCP connection per request: 3 handshake packets plus the close.
    keepalive 64;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}

location /api/ {
    proxy_pass http://bookings_api/;
    # MANDATORY for keepalive to work: HTTP/1.1 and no Connection header
    # inherited from the client.
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

The measured impact of compression:

Resource Uncompressed gzip -5 brotli -11 (precompressed)
app.js 842 KB 218 KB 178 KB
styles.css 156 KB 24 KB 19 KB
/routes response (JSON) 48 KB 6 KB 5 KB

A 75-90% reduction in traffic, at almost no CPU cost (assets ship precompressed; only the JSON is compressed on the fly). On a mobile connection, that is the difference between a 3-second and a 400 ms page load.

Persistent connections: the hidden cost

Without keepalive, every nginx request to bookings-api opens a new TCP connection:

Cost of a new TCP connection (inside the cluster):
  SYN -> SYN/ACK -> ACK:  ~0.4 ms (3 packets)
  Close (FIN/ACK x2):     ~0.2 ms
  Total: ~0.6 ms of added latency per request

At 500 requests/s: 300 ms/s of accumulated latency.

And worse: nginx's ephemeral ports run out.
The typical range is ~28,000 ports. With a TIME_WAIT of 60 s:
  28,000 / 60 = 466 new connections per second AT MOST.
Above that: "cannot assign requested address" errors.

Ephemeral-port exhaustion is a hard limit that appears all at once and is extremely confusing when it happens. Persistent connections remove it at the root.

The same optimisation applies in bookings-api towards pricing-engine and towards the rest of the internal services: always use an HTTP agent with keepAlive enabled.

// bookings-api: HTTP agent with persistent connections
const http = require('http');
const agent = new http.Agent({
  keepAlive: true,
  keepAliveMsecs: 30000,
  maxSockets: 50,
  maxFreeSockets: 10,
  timeout: 5000,
});

  1. Resource tuning: the mechanics of CPU throttling

In module 3 we saw that the CPU limit produces throttling. Now we are going to see exactly how, because the mechanism explains a behaviour that otherwise looks impossible.

The CFS scheduler and the quota

The CPU limit is translated into a quota for the Linux kernel's Completely Fair Scheduler, through two cgroup parameters:

Parameter Default value Meaning
cpu.cfs_period_us 100,000 µs (100 ms) The length of each accounting period
cpu.cfs_quota_us Computed from the limit Microseconds of CPU allowed per period

With limits.cpu: 1:

cfs_period_us = 100,000 (100 ms)
cfs_quota_us  = 100,000 (100 ms of CPU for every 100 ms of wall clock)

With limits.cpu: 500m:

cfs_period_us = 100,000 (100 ms)
cfs_quota_us  =  50,000 (50 ms of CPU for every 100 ms of wall clock)

How it works: every 100 ms the container receives its quota. When it uses it up, ALL its threads stop until the next period.

Why throttling shows up before 100%

Here is the point that surprises everybody.

bookings-api with limits.cpu: 1  ->  a quota of 100 ms per 100 ms period.

Node.js with 4 active threads (the event loop + 3 from the libuv pool
for crypto, compression and DNS):

A 100 ms period:
  t=0 ms:   the 4 threads start working
  t=25 ms:  4 threads x 25 ms = 100 ms of CPU consumed. QUOTA EXHAUSTED.
  t=25 ms:  the kernel STOPS every thread.
  t=100 ms: new period. They resume.

RESULT: the container only works 25 ms out of every 100.
        It is stopped 75% OF WALL-CLOCK TIME.

And yet kubectl top shows:
  CPU: 1000m (100% of the limit)

It used exactly its quota. It looks "at the limit" and not "throttled".
But any request arriving at t=30 ms WAITS 70 ms doing nothing.

A pod can be severely throttled while showing a utilisation that looks reasonable, because utilisation is measured as quota consumed over total quota, not as useful time over wall-clock time.

With limits.cpu: 500m (a 50 ms quota) and the same 4 threads:

t=12.5 ms: quota exhausted.
The container works 12.5 ms out of every 100: 12.5% of the time.
A request arriving at t=20 ms waits 80 ms.

And kubectl top would show 500m: "at 100% of its limit".

The more threads the process has, the sooner it exhausts the quota and the more brutal the throttling. A container with 8 threads and a limit of 1 core exhausts its quota in 12.5 ms and is stopped 87.5 ms out of every 100.

The metric that gives it away

# Percentage of periods in which the container was throttled
rate(container_cpu_cfs_throttled_periods_total{
  namespace="rutas-norte-pro", container="api"
}[5m])
/
rate(container_cpu_cfs_periods_total{
  namespace="rutas-norte-pro", container="api"
}[5m])
* 100
# Seconds of throttling per second: how much REAL time is lost
rate(container_cpu_cfs_throttled_seconds_total{
  namespace="rutas-norte-pro", container="api"
}[5m])

Interpretation:

Percentage of throttled periods Diagnosis
0-1% Healthy
1-5% Acceptable; occasional spikes
5-25% A real problem. p99 latency is suffering
> 25% Serious. The application is stopped most of the time

This metric should be on the main Grafana dashboard of any Kubernetes platform, and it almost never is. It explains most of the mysteries of high p99 latency with apparently normal CPU.

The measurement at Rutas Norte

Before tuning:

bookings-api, limits.cpu: 1, 4 replicas, 340 rps:
  Throttled periods: 18.4%
  Throttled time: 0.31 s/s
  p95 latency: 287 ms
  p99 latency: 891 ms

Almost one in five periods with the container stopped. That is the origin of the 891 ms p99: requests that arrived just after the quota ran out and waited for the next period.

  1. The debate about removing the CPU limit

It is one of the liveliest debates in the Kubernetes community, and it deserves an honest presentation because there are solid arguments on both sides.

The proposal

Remove limits.cpu entirely, leaving only requests.cpu.

resources:
  requests:
    cpu: 412m         # Guaranteed reservation
    memory: 800Mi
  limits:
    # NO limits.cpu
    memory: 1600Mi    # The memory limit IS kept

The arguments in favour

1. requests already guarantees a fair share. requests.cpu is not just a reservation for the scheduler: it translates into cpu.shares (or cpu.weight in cgroups v2), which is the relative weight in the split when there is contention. A pod with 412m of requests is guaranteed its proportional 412m when the node is saturated.

2. CPU is compressible. Unlike memory, CPU can be taken away and given back without killing anything. A pod using extra CPU will simply go slower when contention arrives; it does not corrupt anything and it does not die.

3. Idle CPU is wasted. If the node has free CPU and your pod is throttled, that CPU is lost. Without a limit, the pod takes advantage of what is spare.

4. It eliminates throttling at the root. With no quota, there are no throttled periods. The p99 improves immediately.

5. Start-up is faster. Node.js compiling and loading modules burns a lot of CPU in the first few seconds. Without a limit, it starts in 8 seconds instead of 25. And that is directly relevant to autoscaling (section 13).

The arguments against

1. A faulty pod can affect its neighbours. An infinite loop or a leak would consume all the node's available CPU. The other pods have their requests guaranteed, but they lose their burst headroom.

2. Behaviour becomes unpredictable. A pod's performance depends on what else is on its node. The same request takes 40 ms on an empty node and 90 ms on a full one. That makes diagnosis far harder and measurements non-reproducible.

3. It breaks the Guaranteed QoS class. Without limits equal to requests, the pod cannot be Guaranteed (module 3). For bookings-postgres, which we want to be the last candidate for eviction, that is unacceptable.

4. It can mask problems. A pod that needs three times its requests is badly sized. With a limit, you notice (throttling); without a limit, it works fine until one day the node fills up and everything degrades at once.

5. Load tests stop being reliable. If you measure in pre-production with empty nodes, you will get numbers that will not reproduce in production with full ones.

The recommended position

Workload type limits.cpu Justification
Services on the critical path (bookings-api, web-store) No limits.cpu, or a very generous one Latency rules; throttling is the enemy
Batch jobs (occupancy-reports) With limits.cpu They can go slowly; they must not disturb the rest
Databases (bookings-postgres) With limits.cpu = requests Guaranteed QoS and predictable performance
Sidecars and agents With a low limits.cpu They must never grow; a leaking exporter is a danger
Unknown or third-party workloads With limits.cpu Isolation as a precaution
Multi-tenant environments With limits.cpu, always Isolation is a requirement, not an option

And the decision for Rutas Norte:

# bookings-api: NO CPU limit
resources:
  requests:
    cpu: 412m          # Guaranteed reservation. Weight in the split.
    memory: 800Mi
  limits:
    # NO limits.cpu, DELIBERATELY.
    #
    # Reason: with limits.cpu: 1 we measured 18.4% of throttled periods
    # and a p99 of 891 ms. Node.js uses several threads (event loop + the
    # libuv pool) and exhausts the 100 ms quota in ~25 ms of wall clock.
    #
    # Without a limit:
    #   - Throttling: 0%
    #   - p99: 891 ms -> 340 ms
    #   - Start-up: 25 s -> 9 s (relevant to autoscaling, 09-01)
    #
    # Accepted risk: a faulty pod can consume the node's spare CPU.
    # Mitigations:
    #   - requests guarantees the minimum for all the other pods.
    #   - The namespace LimitRange imposes a ceiling of 4 cores per container.
    #   - An alert if a pod exceeds 3x its requests for 10 minutes.
    #
    # THE MEMORY LIMIT IS KEPT: memory is NOT compressible and
    # a pod with no memory limit can take the whole node down.
    memory: 1600Mi

Note the last line: the memory limit is always kept. The asymmetry is essential. CPU is compressible (it is taken away and you go slower); memory is not (if there is none, the kernel kills processes). A pod with no memory limit and a leak can cause the OOMKilled of innocent neighbouring pods, or leave the node NotReady.

And the alert that goes with the decision:

# A pod consuming more than three times its requests in a sustained way
(
  rate(container_cpu_usage_seconds_total{namespace="rutas-norte-pro"}[10m])
  /
  on(pod, container) kube_pod_container_resource_requests{resource="cpu"}
) > 3

  1. Memory and the Node.js garbage collector in a container

The default heap problem

Node.js decides the maximum size of its JavaScript heap at start-up, and its heuristic looks at the system's memory, not at the container's cgroup.

A 16 GiB node. A container with limits.memory: 1Gi.

Node.js (depending on the version and configuration) may decide a maximum heap
based on the host's 16 GiB, not on the container's 1 GiB.

Result: the garbage collector does not fire urgently because it believes
it has memory to spare. The heap grows until it brushes the container's GiB.

The kernel: OOMKilled.

And the worst part: the process dies WITH no warning and no stack trace. Only
"OOMKilled" appears in the pod status. Node.js never even notices.

Modern Node.js versions are cgroup-aware and adjust better, but the behaviour depends on the version and on the container configuration, so the correct practice is not to rely on the heuristic.

The solution: set the heap explicitly

env:
  # --max-old-space-size sets the maximum size of the old-objects heap
  # in MEGABYTES.
  #
  # Calculation for a limits.memory of 1600Mi:
  #   JavaScript heap:        1024 MB   (--max-old-space-size=1024)
  #   Buffers and ArrayBuffer: ~200 MB (outside the heap)
  #   Native code and libraries: ~150 MB
  #   Stack and internal structures: ~80 MB
  #   Safety margin:                ~146 MB
  #   -------------------------------------
  #   TOTAL:                       1600 MB
  #
  # RULE OF THUMB: heap = 65-70% of limits.memory. The rest is NOT
  # waste: it is memory Node.js needs outside the heap.
  - name: NODE_OPTIONS
    value: "--max-old-space-size=1024"

The 65-70% rule is the one to remember. The usual mistake is setting --max-old-space-size equal to limits.memory, which guarantees an OOMKilled as soon as the heap approaches its maximum: the memory outside the heap does not fit.

Diagnosing memory problems

# Memory usage against the limit
container_memory_working_set_bytes{namespace="rutas-norte-pro", container="api"}
/
on(pod, container) kube_pod_container_resource_limits{resource="memory"}
# Recent OOMKilled events: the unmistakable signal
increase(kube_pod_container_status_restarts_total{namespace="rutas-norte-pro"}[1h]) > 0
and on(pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

And the signature of a memory leak: the working_set grows monotonically and never comes down, not even in the night-time traffic troughs. If it comes down in the troughs, that is normal garbage-collector behaviour.

The container_memory_working_set_bytes detail

It is the metric the kernel uses to decide the OOMKilled, and it is not the same as container_memory_usage_bytes:

Metric What it includes What it is for
container_memory_usage_bytes Everything, including reclaimable page cache Almost nothing; it overestimates
container_memory_working_set_bytes Real usage minus reclaimable cache The one that decides the OOMKilled
container_memory_rss Resident anonymous memory Useful for spotting leaks

Always use working_set for memory alerts. Alerting on usage_bytes produces constant false positives, because it includes file cache the kernel will release without any trouble.

  1. The right pod size

With a fixed amount of resources, many small pods or a few big ones?

Budget: 12 cores and 24 GiB for bookings-api.

Option A: 30 pods of 400m and 800Mi
Option B: 12 pods of 1 core and 2 GiB
Option C:  6 pods of 2 cores and 4 GiB
Criterion Many small (A) Few large (C)
Scaling granularity Fine: +400m per step Coarse: +2 cores per step
Impact of losing one pod 1/30 = 3.3% 1/6 = 16.7%
Per-pod overhead (runtime, sidecars) High: 30 sidecars, 30 pools Low: 6
Spread across nodes and zones Easy Hard: they may not fit
Use of multiple cores Poor if the app is single-threaded Good if the app is multi-threaded
Total start-up time when scaling 30 start-ups 6 start-ups
Connections to PostgreSQL 30 pools 6 pools
In-memory cache per pod Fragmented, ineffective Consolidated, more effective
Bin-packing efficiency on nodes High Low: wasted gaps

The decisive criterion: the concurrency model

Node.js is single-threaded for JavaScript code. A Node.js process cannot use more than one core to run your code, however much CPU you give it.

A bookings-api pod with 4 cores:
  - The event loop uses 1 core at most.
  - The libuv pool (4 threads by default) uses a bit more for file I/O,
    DNS, crypto and compression.
  - In practice, a Node.js process makes use of ~1.3 cores.

The other 2.7 cores ARE WASTED.

For Node.js, many small pods is the right choice, with a size of between 0.5 and 1.5 cores per pod.

For other platforms the answer changes:

Platform Recommended size Reason
Node.js (one process) 0.5-1.5 cores Single-threaded for JavaScript
Node.js with cluster 1 core per worker Several processes in the pod
Java / JVM 2-4 cores Genuinely multi-threaded; the GC benefits from more cores and memory
Go 1-4 cores Efficiently multi-threaded; scales well both ways
Python (WSGI) 0.5-1 core per worker The GIL limits it to one thread per process
nginx 0.2-0.5 cores Very efficient; many small replicas
PostgreSQL 4-8 cores One process per connection; benefits from lots of memory

The final decision for Rutas Norte:

# bookings-api: 12 pods of 1 core (option B), not 30 of 400m nor 6 of 2 cores.
#
# Against option A (30 pods of 400m):
#   - 30 connection pools against PostgreSQL (see section 6).
#   - 30 exporter sidecars: 30 x 6m = 180m on observability alone.
#   - 400m is little for the Node.js START-UP, which burns a lot of CPU
#     compiling: start-up would go from 9 s to more than 30 s.
#
# Against option C (6 pods of 2 cores):
#   - Node.js does not make use of 2 cores with a single process.
#   - Losing 1 of 6 replicas is losing 16.7% of the capacity.
#   - Scaling granularity far too coarse.
#
# 1 core is the sweet spot: enough for the event loop and the libuv
# pool, with headroom for start-up, and fine granularity for the HPA.
resources:
  requests:
    cpu: 412m           # From the VPA (09-02): real steady-state consumption
    memory: 800Mi
  limits:
    memory: 1600Mi      # No limits.cpu (section 10)

A requests of 412m with the ability to burst up to whatever the node allows is the best of both worlds: you reserve what you normally consume, but you can use more during start-up and peaks.

  1. Fast start-up as an autoscaling requirement

This section connects directly with everything else in the module, and it is why performance and scaling are not separate topics.

The calculation

The HPA detects the overload and asks for new replicas.
Time until a new replica serves traffic:

  HPA decision:                           0-15 s (15 s cycle)
  Scheduling:                               1 s
  Image pull:                             0-60 s (0 if cached)
  Container start-up:                    5-30 s
  Readiness probe OK:                    3-10 s
  ----------------------------------------------
  TOTAL:                                 9-116 s

Throughout all that time, the existing replicas are overloaded. A 90-second start-up makes autoscaling almost useless for fast spikes.

Lever 1: a lightweight image

# --- Build stage ---
FROM node:20.15-bookworm AS builder
WORKDIR /app
COPY package*.json ./
# npm ci with --omit=dev: production dependencies only
RUN npm ci --omit=dev
COPY . .
RUN npm run build

# --- Final stage: ONLY what is needed to run ---
FROM node:20.15-alpine
WORKDIR /app
# Unprivileged user (08-02)
RUN addgroup -g 10001 rutasnorte && adduser -u 10001 -G rutasnorte -D rutasnorte
COPY --from=builder --chown=10001:10001 /app/node_modules ./node_modules
COPY --from=builder --chown=10001:10001 /app/dist ./dist
COPY --from=builder --chown=10001:10001 /app/package.json ./
USER 10001
EXPOSE 8080
CMD ["node", "dist/server.js"]

The measured impact:

Image Size Pull (uncached) Pull (cached)
node:20 (full) 1.1 GB 48 s 0 s
node:20-slim 280 MB 14 s 0 s
node:20-alpine multi-stage 94 MB 5 s 0 s

From 48 to 5 seconds. And as an extra benefit from module 8: fewer packages means less attack surface and far fewer vulnerabilities in the Trivy scan (08-06).

Lever 2: preloading images onto the nodes

If the image is already on the node, the pull takes zero. Two ways:

# k8s/environments/pro/daemonset-image-preload.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-preload
  namespace: rutas-norte-pro
  labels:
    app: image-preload
    app.kubernetes.io/part-of: rutas-norte
spec:
  selector:
    matchLabels:
      app: image-preload
  template:
    metadata:
      labels:
        app: image-preload
    spec:
      # Low priority: if the room is needed, let it go.
      priorityClassName: capacity-filler
      # The initContainers PULL the images onto the node and finish.
      # From then on they stay in containerd's local cache.
      initContainers:
        - name: preload-api
          image: registry.rutasnorte.example/bookings-api:1.14.2
          command: ["/bin/true"]
        - name: preload-web
          image: registry.rutasnorte.example/web-store:3.2.0
          command: ["/bin/true"]
        - name: preload-worker
          image: registry.rutasnorte.example/notifications-worker:2.3.0
          command: ["/bin/true"]
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests: {cpu: 5m, memory: 8Mi}
            limits: {cpu: 10m, memory: 16Mi}

Being a DaemonSet, it runs on every node, including the ones the Cluster Autoscaler boots (09-03). Cost: a few megabytes of disk per node and 5m of CPU. Benefit: 5-48 seconds less on every scale-up.

This DaemonSet is one of the best cost-benefit ratios in the whole module.

The cloud alternative is baking the images into the nodes' AMI or machine image, which is even faster but requires rebuilding the machine image on every deployment.

Lever 3: well-tuned probes

From module 7, but with a performance twist:

# BAD: the pod takes 20 s longer than necessary to receive traffic, for no reason.
readinessProbe:
  httpGet: {path: /ready, port: 8080}
  initialDelaySeconds: 30       # <- Waits 30 s even if it is ready in 5
  periodSeconds: 10             # <- Checks every 10 s: loses up to 10 s more
  failureThreshold: 3

# GOOD: a startupProbe for start-up, an aggressive readinessProbe afterwards.
startupProbe:
  httpGet: {path: /health, port: 8080}
  # Checks every second from second 0. As soon as it starts, it is ready.
  periodSeconds: 1
  # 40 attempts x 1 s = 40 seconds of maximum headroom for start-up.
  # If it takes longer, something is wrong and we want to know.
  failureThreshold: 40

readinessProbe:
  httpGet: {path: /ready, port: 8080}
  # NO initialDelaySeconds: the startupProbe already covers start-up.
  periodSeconds: 2              # Frequent checks: it reacts fast
  timeoutSeconds: 1
  failureThreshold: 2
  successThreshold: 1

livenessProbe:
  httpGet: {path: /health, port: 8080}
  periodSeconds: 10
  timeoutSeconds: 3
  # A high failureThreshold: do NOT kill the pod over a temporary latency spike.
  # An aggressive livenessProbe under high load produces cascading restarts:
  # the pod is slow -> the probe fails -> it restarts -> less capacity ->
  # the others get slower -> they restart too. A death spiral.
  failureThreshold: 6

The warning about the livenessProbe deserves emphasis. It is one of the most frequent causes of total outages under load: the liveness probe, designed to detect hung processes, ends up killing healthy pods that are simply slow, and the result is a cascading collapse precisely at the moment of peak traffic. When in doubt, make the livenessProbe far more permissive than the readinessProbe.

The cumulative result

BEFORE:
  Image pull (1.1 GB, uncached):                   48 s
  Node.js start-up (with limits.cpu: 1):           25 s
  readiness initialDelaySeconds:                   30 s
  Probe cycles until it passes:                    10 s
  ----------------------------------------------------
  TOTAL:                                          113 s

AFTER:
  Pull (94 MB, preloaded by the DaemonSet):         0 s
  Node.js start-up (no limits.cpu):                 9 s
  startupProbe (checks every second):               1 s
  readinessProbe (2 s period):                      2 s
  ----------------------------------------------------
  TOTAL:                                           12 s

From 113 to 12 seconds. A factor of 9.4.

And this changes autoscaling qualitatively: with a 12-second start-up, the HPA can react to a spike before the user notices it. With 113 seconds, it cannot.

  1. Cluster tuning: CoreDNS, ndots and kube-proxy

ndots: 5 and outbound-call latency

Here we resolve something we left pending in 04-03.

Every pod has an /etc/resolv.conf generated by Kubernetes:

nameserver 10.96.0.10
search rutas-norte-pro.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means: if the name to resolve has fewer than 5 dots, try all the search suffixes first before trying it as an absolute name.

Let's see what happens when resolving api.external-mail.example (2 dots, fewer than 5):

1. api.external-mail.example.rutas-norte-pro.svc.cluster.local  -> NXDOMAIN
2. api.external-mail.example.svc.cluster.local                  -> NXDOMAIN
3. api.external-mail.example.cluster.local                      -> NXDOMAIN
4. api.external-mail.example                                    -> OK

FOUR DNS queries to resolve one name.
And since the glibc resolver queries A and AAAA: EIGHT queries in total.

With 300 emails per second from notifications-worker, each one resolving the SMTP server:

300 resolutions/s x 8 queries = 2,400 DNS queries/s to CoreDNS.

If each resolution adds 3 ms of latency (4 failed round trips):
  300 x 3 ms = 900 ms/s of accumulated wasted latency.

And CoreDNS with 2,400 queries/s of which 75% are useless NXDOMAINs.

Three solutions, from smallest to largest in scope:

Solution 1: the trailing dot (an absolute FQDN).

// BAD: 4 queries because of ndots:5
const server = 'smtp.external-mail.example';

// GOOD: the trailing dot marks the name as ABSOLUTE.
// The resolver does NOT try the search suffixes. A SINGLE query.
const server = 'smtp.external-mail.example.';

A single character that removes 75% of the DNS queries. It is the cheapest fix there is and almost nobody knows it.

Solution 2: lowering ndots in the pod.

spec:
  dnsConfig:
    options:
      - name: ndots
        # With ndots: 2, any name with 2 or more dots is tried
        # as absolute first. Internal names like
        # "bookings-postgres.rutas-norte-pro" (1 dot) still work
        # through the search suffixes.
        value: "2"
      # Fail fast if DNS does not respond.
      - name: timeout
        value: "1"
      - name: attempts
        value: "2"

Careful: lowering ndots can break the resolution of short internal service names. If your code uses plain bookings-postgres (0 dots), it still works with ndots: 2. If it uses bookings-postgres.rutas-norte-pro (1 dot), it works too. But it is worth testing.

Solution 3: NodeLocal DNSCache.

A DaemonSet that puts a DNS cache on every node. Pods query 169.254.20.10 (a node-local address) instead of going over the network to CoreDNS.

Advantage Detail
Latency From ~2 ms (network) to ~0.1 ms (local)
Load on CoreDNS A 70-90% reduction
Resilience A CoreDNS failure does not break cached queries
Connections It uses TCP towards CoreDNS, avoiding the conntrack problem with UDP

It is the most complete solution and the one recommended for any cluster with serious traffic. I mention it here because it is the natural complement to the ndots tuning.

CoreDNS: replicas and caching

# CoreDNS ConfigMap (kube-system/coredns)
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health { lameduck 5s }
        ready

        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }

        prometheus :9153

        forward . /etc/resolv.conf {
            max_concurrent 1000
            # Distribution policy across external DNS servers.
            policy sequential
        }

        # CACHE: the most important CoreDNS optimisation.
        #   3600 s (1 h) for positive cluster answers
        #     30 s for negative answers (NXDOMAIN)
        # The short TTL for negatives is DELIBERATE: when a new
        # Service is created, we do not want pods to keep getting NXDOMAIN
        # for an hour.
        cache 3600 {
            success 9984 3600
            denial 9984 30
        }

        loop
        reload
        loadbalance
    }

Sizing the replicas:

Rule of thumb: 1 CoreDNS replica per 8-10 nodes, with a minimum of 2.

Rutas Norte cluster: 9-12 nodes at the peak.
Replicas: 3 (one per zone, see 09-05).

If the metrics show high DNS latency:
  histogram_quantile(0.99,
    sum(rate(coredns_dns_request_duration_seconds_bucket[5m])) by (le)
  ) > 0.05

...the fix is usually NodeLocal DNSCache, not more CoreDNS replicas.

kube-proxy in IPVS mode

Recall 04-01: kube-proxy has two main modes.

Aspect iptables (default) IPVS
Structure Sequential rule chains Hash tables in the kernel
Lookup complexity O(n) with the number of services O(1)
Rule update time O(n): with 5,000 services, seconds O(1): milliseconds
Balancing algorithms Random only rr, lc, dh, sh, sed, nq
Practical threshold Up to ~1,000 services More than 1,000 services 
# Enable IPVS in minikube
minikube start -p rutas-norte --extra-config=kube-proxy.mode=ipvs

# Verify
kubectl logs -n kube-system -l k8s-app=kube-proxy | grep -i "proxy mode"
I0501 09:14:22.847 server_others.go: Using ipvs Proxier.

For Rutas Norte, with fewer than 50 services, IPVS adds nothing measurable. It is an optimisation for large clusters. I mention it so that you know when it matters: if your cluster has more than a thousand services and you notice that endpoint changes take a while to propagate, that is the symptom.

Choosing the node size

Size Advantages Drawbacks
Small (2-4 cores) Fine CA granularity; smaller impact per failure; efficient bin packing More system overhead (~0.5 cores and 1.5 GiB per node); more DaemonSets
Large (16-32 cores) Less proportional overhead; fewer DaemonSets; better for large pods Coarse CA granularity; one failure loses a lot; bin packing with gaps

The calculation that decides:

Per-node overhead (kubelet, system, DaemonSets): ~0.95 cores and 2 GiB.

4-core nodes:  0.95 / 4  = 23.8% overhead
8-core nodes:  0.95 / 8  = 11.9%
16-core nodes: 0.95 / 16 =  5.9%

With 12 usable cores needed:
  With 4-core nodes:  12 / 3.05 = 3.93 -> 4 nodes = 16 cores bought (33% excess)
  With 8-core nodes:  12 / 7.05 = 1.70 -> 2 nodes = 16 cores bought (33% excess)
  With 16-core nodes: 12 / 15.05 = 0.80 -> 1 node = 16 cores bought (33% excess)

They coincide by chance in this case, but the 16-core node puts EVERYTHING on one
machine: a failure takes the whole platform. And the CA cannot adjust
with a granularity finer than 16 cores.

Recommendation for Rutas Norte: nodes of 8 cores and 32 GiB. A balance between overhead (11.9%) and granularity, and big enough for bookings-postgres to have headroom in the data group.

  1. Storage: when the disk is the limit

bookings-postgres uses the rutasnorte-fast StorageClass from 05-04. There comes a point when the disk is the real limit, and it is worth knowing how to recognise it.

The IOPS arithmetic

StorageClass rutasnorte-fast: SSD with 3,000 provisioned IOPS.

A booking write in PostgreSQL involves:
  - Writing to the WAL (write-ahead log):         1 IOPS
  - fsync of the WAL (mandatory for durability):  1 IOPS
  - Updating data pages (deferred):              ~0.3 IOPS amortised
  - Updating indexes:                            ~0.5 IOPS
  TOTAL: ~2.8 IOPS per booking

Maximum bookings per second by IOPS:
  3,000 / 2.8 = 1,071 bookings/s

And reads that are NOT in shared_buffers nor in the system cache
also consume IOPS.

With 300 transactions per second at the peak, the disk is not the limit for writes. But reads may be if the cache is not effective.

Diagnosing whether the disk is the bottleneck

-- PostgreSQL cache hit ratio
SELECT
  datname,
  blks_hit,
  blks_read,
  round(100.0 * blks_hit / GREATEST(blks_hit + blks_read, 1), 2) AS hit_ratio
FROM pg_stat_database
WHERE datname = 'bookings';
 datname  | blks_hit  | blks_read | hit_ratio
----------+-----------+-----------+-----------
 bookings | 892471203 |  18472941 |     97.97

97.97% hits: excellent. Only 2% of the reads go to disk.

Hit ratio Diagnosis
> 99% Optimal
95-99% Good
90-95% Increase shared_buffers or the pod's memory
< 90% The disk is probably the bottleneck

And the node metrics:

# Read and write IOPS on the volume
rate(node_disk_reads_completed_total{device="nvme1n1"}[5m])
+ rate(node_disk_writes_completed_total{device="nvme1n1"}[5m])

# THE KEY METRIC: the time the disk is busy.
# If it exceeds 0.8 (80%), the disk is saturated.
rate(node_disk_io_time_seconds_total{device="nvme1n1"}[5m])

node_disk_io_time_seconds_total above 0.8 means the disk is saturated, and no amount of extra CPU or memory will fix it.

PostgreSQL settings related to the disk

# bookings-postgres ConfigMap
data:
  postgresql.conf: |
    # --- MEMORY (the pod's limits.memory: 8 GiB) ---
    # shared_buffers: 25% of the container's memory. It is the
    # classic recommendation and it is still a good one.
    shared_buffers = 2GB
    # effective_cache_size: an estimate of the TOTAL available cache
    # (shared_buffers + the system page cache). It does NOT reserve memory:
    # it only tells the query planner to prefer indexes.
    effective_cache_size = 6GB
    # work_mem per sort or hash operation. CAREFUL: it is multiplied
    # by the number of concurrent operations. With 200 connections and
    # 3 operations each: 200 x 3 x 16MB = 9.6 GB. MORE THAN THE LIMIT.
    # That is why it is 16MB and no more.
    work_mem = 16MB
    maintenance_work_mem = 512MB

    # --- WRITING TO DISK ---
    # Spread the checkpoint fsyncs over 90% of the interval,
    # instead of doing them all at once. Avoids periodic latency spikes.
    checkpoint_completion_target = 0.9
    # 15 minutes between checkpoints: fewer writes, more recovery
    # time after a crash. A reasonable trade-off.
    checkpoint_timeout = 15min
    max_wal_size = 4GB
    min_wal_size = 1GB

    # --- ACCESS COSTS (for the planner) ---
    # random_page_cost = 1.1 on SSD (the default is 4.0, meant for spinning
    # disks). Without this setting, the planner AVOIDS indexes believing
    # that random access is expensive, and does useless sequential scans.
    # IT IS ONE OF THE MOST PROFITABLE SETTINGS ON SSD.
    random_page_cost = 1.1
    effective_io_concurrency = 200

    # --- CONNECTIONS ---
    max_connections = 200

    # --- STATISTICS ---
    shared_preload_libraries = 'pg_stat_statements'
    pg_stat_statements.max = 10000
    track_io_timing = on

random_page_cost = 1.1 deserves to be highlighted: it is a one-line change that can transform the execution plan of many queries. The default of 4.0 assumes spinning disks where a random access costs four times as much as a sequential one. On SSD the difference is minimal, and with the default value the planner rejects perfectly good indexes.

  1. The latency budget

Everything above is technique. The latency budget is what turns it into a plan.

The idea

The business defines an objective: "95% of route searches must respond in under 300 ms". That number is useful for the SLO, but it tells nobody what to do. The latency budget shares it out across the components, turning a global objective into local objectives each team can measure and meet.

The Rutas Norte budget

Objective: route-search p95 < 300 ms.

flowchart LR
    U[User] -->|20 ms| I[Ingress<br/>nginx]
    I -->|5 ms| W[web-store]
    W -->|3 ms| A[bookings-api]
    A -->|4 ms| R[redis-cache]
    A -->|8 ms| P[bookings-postgres]
    A -->|15 ms| M[pricing-engine]

    style A fill:#cde,stroke:#369
Component p95 budget % of total What it includes
User-to-Ingress network 40 ms 13.3% Internet latency, TLS (not controllable)
Ingress (nginx) 8 ms 2.7% Routing, TLS termination
Internal network 6 ms 2.0% 3 cluster network hops
web-store 12 ms 4.0% Reverse proxy, compression
bookings-api (own time) 35 ms 11.7% Logic, serialisation, validation
redis-cache 6 ms 2.0% Availability lookup (95% hits)
bookings-postgres 45 ms 15.0% Indexed route query
pricing-engine 80 ms 26.7% Dynamic price computation
Safety margin 68 ms 22.7% Variability, retries, GC
TOTAL 300 ms 100%

Three things this budget does well:

1. It includes an explicit 22.7% margin. Without margin, any variation (a garbage-collector pause, a retry, a network spike) breaks the objective. A budget that adds up to exactly 300 ms is guaranteed to fail.

2. It identifies the main consumer. pricing-engine takes 26.7% of the budget. If something has to be optimised, that is it. Without the budget, nobody would know where to start.

3. It separates what is controllable from what is not. The 40 ms of user network cannot be reduced with code; only with a CDN or a closer point of presence. That is a different decision and has to be seen as such.

From budget to alerts

Every line of the budget becomes an alert:

# bookings-api (own time, excluding dependencies)
histogram_quantile(0.95,
  sum(rate(rutasnorte_own_duration_seconds_bucket{
    namespace="rutas-norte-pro"
  }[5m])) by (le)
) * 1000 > 35

# bookings-postgres (query time as seen from the API)
histogram_quantile(0.95,
  sum(rate(rutasnorte_db_query_duration_seconds_bucket{
    namespace="rutas-norte-pro"
  }[5m])) by (le)
) * 1000 > 45

# pricing-engine (the biggest consumer: a priority alert)
histogram_quantile(0.95,
  sum(rate(pricing_engine_duration_seconds_bucket{
    namespace="rutas-norte-pro"
  }[5m])) by (le)
) * 1000 > 80

# And the global objective, which is what the business cares about
histogram_quantile(0.95,
  sum(rate(ingress_nginx_request_duration_seconds_bucket{
    ingress="rutas-norte-public", path="/routes"
  }[5m])) by (le)
) * 1000 > 300

When the global alert fires, the per-component alerts tell you immediately which one has blown its budget. It is the difference between "the platform is slow" and "pricing-engine is at 140 ms against a budget of 80". The second sentence is actionable; the first is not.

The margin rule

A heuristic that works well:

Sum of the component budgets  <=  75% of the total objective.
The remaining 25% is margin.

And a sanity check: if, when drawing up the budget, the sum of what the components actually take already exceeds the objective, no amount of tuning will do. You have to redesign: cache more aggressively, compute in the background, or change the objective.

  1. The measured result of the May bank holiday

Let's close with the numbers. This is the whole of module 9, measured.

The before-and-after table

Metric 2025 (before) 2026 (after) Improvement
AVAILABILITY
Minutes of total outage 47 min 0 min —
Requests with errors 12.4% 0.08% 155 times better
Bookings lost to errors ~2,800 11 254 times better
LATENCY (p95, route search)
At calm 287 ms 94 ms 3.1 times
At the peak 4,200 ms 168 ms 25 times
p99 at the peak timeout 412 ms —
CAPACITY
Sustained requests per second 340 2,680 7.9 times
Capacity per replica (rps) 85 224 2.6 times
Replicas needed at the peak 4 (fixed) 12 (auto) —
RESOURCES
CPU throttling 18.4% 0.0% —
Connections to PostgreSQL at the peak 600 (400 rejected) 75 (stable) —
redis-cache hit ratio 13.2% 94.7% 7.2 times
Queries to PostgreSQL per second 8,400 340 24 times fewer
Pod start-up time 113 s 12 s 9.4 times
COST
Nodes at the peak 4 (insufficient) 9 —
Average monthly cost €712 €855 +20%
Cost in the bank-holiday month €712 €912 +28%
Cost per 1,000 bookings €8.90 €1.12 8 times better

The last row is the one to show management. The absolute cost went up 20%, but the cost per booking came down eightfold, because the platform sells eight times more.

What each change contributed

And now the most useful table: the breakdown by change, measured individually in rutas-norte-pre.

Change Contribution to capacity per replica Implementation cost
Fixing the route-search N+1 +65% (85 → 140 rps) 1 day of development
Composite index on routes +28% (140 → 179 rps) 1 hour
Availability cache in Redis (TTL 10 s) +19% (179 → 213 rps) 2 days of development
Removing limits.cpu +5% (213 → 224 rps) 10 minutes
Connection pool to 5 + PgBouncer 0% under normal conditions 1 day
Compression and caching in web-store 0% (it affects bandwidth) 2 hours
random_page_cost = 1.1 Included in the index 1 minute

And this table has to be read carefully, because it contains the most important lesson:

The first three changes are application changes and account for 92% of the improvement. The Kubernetes tuning (removing limits.cpu) accounts for 5%.

The connection-pool change contributes 0% under normal conditions, and yet it is probably the most important change on the list: it is what stops the platform falling over when the HPA scales to 30 replicas. Its value is not in improving average performance, but in eliminating a catastrophic failure mode.

That distinction — between changes that improve performance and changes that eliminate failure modes — is what separates naive tuning from professional tuning.

The conclusion about the whole module

Without performance tuning:
  Capacity per replica: 85 rps.
  For 2,680 rps: 2,680 / 85 = 32 replicas.
  CPU needed: 32 x 412m = 13.2 cores.
  Nodes: 5 nodes of 8 cores for bookings-api alone.

With tuning:
  Capacity per replica: 224 rps.
  For 2,680 rps: 2,680 / 224 = 12 replicas.
  CPU needed: 12 x 412m = 4.9 cores.
  Nodes: 2 nodes of 8 cores.

SAVING: 3 nodes at the peak. And, more importantly, the peak FITS in a cluster
that needs no node-group reconfiguration and no quota renegotiation.

Tuning performance is not an alternative to scaling: it is what makes scaling affordable. Scaling an inefficient application works, but it costs three times as much and hits the capacity limits far sooner.

Common Mistakes and Tips

Mistake 1: optimising without measuring. Changing things "that sound like they should be faster" produces, at best, nothing. At worst, regressions nobody detects. Measure the baseline, date it, and always compare against it.

Mistake 2: reporting the average latency. With avg=142ms and p99=2.1s, the average is a statistically correct lie. The p99 is the experience of 1 in every 100 requests, and on a page with 30 requests that is 1 in every 4 users. Use p95 and p99 always.

Mistake 3: ramping-vus in a load test. When the system slows down, the virtual users generate less load, and the test throttles itself, hiding the problem. Use ramping-arrival-rate: real users do not coordinate to wait.

Mistake 4: a large connection pool with autoscaling. maxReplicas × pool ≤ max_connections × 0.8. With 30 replicas and a pool of 20, you have 600 connections against a limit of 200. It is a bomb that goes off exactly when the HPA scales, that is, at the worst possible moment.

Mistake 5: --max-old-space-size equal to limits.memory. The JavaScript heap is not the whole memory of the process: buffers, native code and internal structures live outside it. The rule is 65-70% of the limit.

Mistake 6: an aggressive livenessProbe. Under load, a slow pod fails the probe, restarts, reduces capacity, makes the others slower, and a cascading collapse follows. Make the livenessProbe far more permissive than the readinessProbe.

Mistake 7: believing throttling appears at 100%. With a limit of 1 core and 4 threads, the quota runs out in 25 ms out of every 100. The pod is stopped 75% of the time while showing 100% utilisation. Watch container_cpu_cfs_throttled_periods_total.

Mistake 8: a high initialDelaySeconds "just in case". Every second of delay is a second in which the new replica is not serving traffic while the existing ones are saturated. Use a startupProbe with a 1-second period and remove initialDelaySeconds from the readiness probe.

Mistake 9: running k6 on the same node as the application. They compete for CPU and you will not know whether the latency comes from the system or from the generator. A dedicated node with a taint, and generous resources for k6.

Tip 1: put throttling on the main dashboard. It is the metric that explains most high-p99 mysteries, and almost nobody has it. A panel with throttled_periods / periods per container saves hours of investigation.

Tip 2: use pg_stat_statements ordered by total_exec_time. A 2 ms query executed a million times saturates more than a 500 ms one executed a hundred times. Total time is what matters.

Tip 3: the trailing dot on external FQDNs. smtp.external-mail.example. with a trailing dot removes 3 out of every 4 DNS queries. One character, 75% less DNS traffic.

Tip 4: automate the load test in the pipeline. With thresholds defined, k6 returns an error code when they are not met. A load test that fails the build stops a performance regression from reaching production.

Tip 5: document the latency budget next to the code. A LATENCY-BUDGET.md file in the repository, with the table and the alerts, turns an abstract objective into a shared, verifiable responsibility.

Tip 6: keep the results of every test with its date and its configuration. The historical series is what catches slow degradations: 3% worse each month goes unnoticed, but 30% in a year is catastrophic.

Tip 7: tune before scaling, always. Before raising maxReplicas, ask why one replica only handles what it handles. It costs less and the benefit is multiplied by every replica.

Exercises

Exercise 1: reading a load test

This is the output of a stress test against rutas-norte-pre with 6 bookings-api replicas:

     ✓ search returns 200
     ✗ booking returns 201
      ↳  91% — ✓ 6142 / ✗ 607

     checks.........................: 94.21% ✓ 38921  ✗ 2391
     route_search_duration..........: avg=142ms  min=18ms  med=94ms   max=8.1s   p(95)=387ms  p(99)=2.4s
     confirm_booking_duration.......: avg=891ms  min=112ms med=612ms  max=14.2s  p(95)=3.1s   p(99)=9.8s
     booking_errors.................: 9.00%  ✓ 607  ✗ 6142
     http_req_duration..............: avg=284ms  min=12ms  med=118ms  max=14.2s  p(95)=982ms  p(99)=4.2s
     http_req_failed................: 3.42%  ✓ 1621  ✗ 45782
     http_reqs......................: 47403  158.0/s
     iterations.....................: 9847   32.8/s
     vus............................: 412    min=50   max=1800

And the system metrics during the test:

Average CPU per bookings-api replica:      680m of a 1000m limit (68%)
Throttled CPU periods:                     31.2%
Average memory:                            620Mi of 1600Mi (39%)
Active connections to PostgreSQL:          118 of 200
redis-cache hit ratio:                     22.4%
Queries to PostgreSQL per second:          3,840
p95 latency of PostgreSQL queries:         41 ms
IOPS on the postgres disk:                 2,847 of 3,000 provisioned
Disk I/O time (io_time):                   0.91

Answer: (a) is an SLO of p95 < 300 ms being met? (b) what is the main bottleneck and how do you justify it? (c) why does 68% CPU coexist with 31.2% throttling? (d) rank the first three actions you would take, with the expected improvement of each; (e) what would you NOT do, and why?

Exercise 2: computing the pool and the latency budget

Rutas Norte is adding business-api, an API for travel agencies that book batches of tickets.

The data:

  • HPA maxReplicas: 20.
  • Each request makes 6 queries to PostgreSQL (it is a batch of 6 tickets).
  • Average duration of each query: 14 ms.
  • bookings-postgres has max_connections: 200, shared with bookings-api (which already uses 75 connections through PgBouncer).
  • Business objective: p95 of a batch booking < 900 ms.
  • Expected traffic: 40 requests per second, up to 180 at the peak.
  • Measured in rutas-norte-pre with 4 replicas: a p95 of 640 ms, of which 310 ms are PostgreSQL queries.

Compute: (a) the maximum pool size per replica; (b) whether that pool is enough for the peak traffic, using Little's law; (c) a complete latency budget for the 900 ms, with margin; (d) whether the budget is achievable with the measured data, and what you would do if it is not; (e) the KEDA threshold you would use and its justification.

Exercise 3: diagnosing a progressive degradation

Rutas Norte has had a problem for three weeks. The symptoms:

  • bookings-api's p95 latency has gone from 94 ms to 280 ms, rising about 10 ms a day.
  • There have been no bookings-api deployments in that period.
  • The replica count has risen from 4 to 9 on average, because KEDA scales more.
  • CPU per replica has not changed: still around 310m.
  • Memory per replica has gone from 620Mi to 1,480Mi, rising every day.
  • There have been occasional OOMKilled events for four days (2-3 a day).
  • Traffic has grown only 8% in that period.
  • The redis-cache hit ratio has fallen from 94.7% to 71.2%.
  • Queries to PostgreSQL per second have gone from 340 to 1,890.
  • The p95 of PostgreSQL queries is still 8 ms.
  • The PostgreSQL disk: io_time at 0.34 (unchanged).

Diagnose the problem: what is the most likely root cause and what is the complete causal chain? Which commands would you run, and in what order, to confirm it? What is the immediate fix and what is the real one? Which of the tests from section 3 would have caught this before it reached production?

Solutions

Solution 1

(a) Is the SLO of p95 < 300 ms met? NO, emphatically.

http_req_duration p95: 982 ms   (objective: 300 ms)   -> 3.3 times worse
route_search_duration p95: 387 ms                     -> 1.3 times worse
confirm_booking_duration p95: 3,100 ms                -> 10.3 times worse

And the errors are alarming:

booking_errors: 9.00%  ->  9 out of every 100 bookings FAIL.

Nine out of every hundred customers trying to buy a ticket do not manage it. That is money lost directly, and it is far worse than the latency.

Note the asymmetry too: search is bad but tolerable (387 ms); booking is catastrophically bad (3.1 s). The problem is on the write path.

(b) The main bottleneck: THE POSTGRESQL DISK.

The decisive evidence:

Disk IOPS: 2,847 of 3,000 provisioned  ->  94.9% of capacity
Disk I/O time (io_time): 0.91          ->  91% of the time busy

An io_time of 0.91 means the disk is busy 91% of the time. By the queueing theory of section 5:

Wait ≈ ServiceTime x (0.91 / (1 - 0.91)) = 10.1 x ServiceTime

Every disk operation waits ten times its own service time. The disk is saturated.

And the reason so much I/O reaches the disk is the cache:

redis-cache hit ratio: 22.4%
-> 77.6% of the availability queries go to PostgreSQL.

Queries to PostgreSQL: 3,840/s with 158 requests/s
-> 24.3 queries per HTTP request.

24 queries per request is outrageous. For a request that should make 1-2 queries, this is the unmistakable signature of an N+1 problem (section 7).

The complete causal chain:

1. The Redis cache is not working (22.4% hits).
2. There is an N+1 problem: 24 queries per request.
3. 3,840 queries/s against PostgreSQL.
4. PostgreSQL's cache cannot keep up: many go to disk.
5. The disk is 91% busy: every operation waits ten times its own time.
6. Queries slow down (the 41 ms p95 already reflects it).
7. Connections are held for longer: 118 of 200 active.
8. Writes (bookings) suffer more than reads: a 3.1 s p95.
9. The timeouts produce the 9% booking error rate.

An important note about what is NOT the bottleneck:

CPU: 680m of 1000m (68%)              -> there is headroom
Memory: 620Mi of 1600Mi (39%)         -> there is plenty of headroom
Connections: 118 of 200 (59%)         -> there is headroom

Scaling bookings-api would NOT help. More replicas would make more queries against the same saturated disk, making things worse. It is the 09-01 mistake in its purest form.

(c) 68% CPU with 31.2% throttling.

This question tests section 9. The apparent contradiction is resolved by understanding that they are two different measurements:

"CPU 680m of 1000m (68%)" is the AVERAGE over the measurement interval.
"31.2% of throttled periods" counts individual 100 ms periods.

What really happens inside each 100 ms period:

  Period A (69% of cases): the pod uses 40 ms of its 100 ms quota.
                            No throttling.
  Period B (31% of cases): a burst arrives. Node.js uses several threads
                            (the event loop + libuv for
                            JSON serialisation and compression).
                            4 threads x 25 ms = 100 ms. QUOTA EXHAUSTED
                            in 25 ms of wall clock.
                            The pod is STOPPED for 75 ms.

Weighted average: 0.69 x 40 + 0.31 x 100 = 27.6 + 31 = 58.6...
Adjusting for the real distribution of bursts: ~68%. It matches.

The 68% average hides the fact that in 3 out of 10 periods the pod is stopped for 75 ms. And those stops are exactly what produces the 4.2-second p99: requests that arrived just after the quota ran out.

Throttling is a problem of the tail of the distribution, and that is why the CPU average never reveals it.

(d) The first three actions, by priority:

Action 1: fix the N+1 (24 queries per request → 2).

It is by far the highest-impact one. It attacks the root cause of the whole chain.

-- Before: 1 routes query + 23 seat and price queries
-- After: 1 query with aggregation
SELECT
  r.id, r.origin, r.destination, r.departure_time, r.base_price,
  COUNT(s.id) FILTER (WHERE s.status = 'free') AS free_seats
FROM routes r
LEFT JOIN seats s ON s.route_id = r.id
WHERE r.origin = $1 AND r.destination = $2 AND r.date = $3
GROUP BY r.id, r.origin, r.destination, r.departure_time, r.base_price
ORDER BY r.departure_time;

Expected improvement:

Queries: 3,840/s -> 316/s (divided by 12)
Disk IOPS: 2,847 -> ~240 (below 10% of capacity)
io_time: 0.91 -> ~0.12
Query p95: 41 ms -> ~6 ms

Estimated global p95: 982 ms -> ~250 ms
Booking errors: 9.0% -> < 0.5%

Cost: 1-2 days of development. Return: the largest of the three.

Action 2: fix the Redis cache (22.4% → > 90% hits).

# Diagnose first
kubectl exec -n rutas-norte-pre redis-cache-0 -- redis-cli INFO stats
kubectl exec -n rutas-norte-pre redis-cache-0 -- redis-cli --scan --count 20

The likely causes, in order:

  1. Keys that are too specific (with a timestamp or a user id).
  2. A TTL that is too short (if it is 1 second, it almost never hits).
  3. maxmemory unconfigured and keys being evicted prematurely.

The typical fix:

// Key: only the parameters that determine the result
const key = `seats:${routeId}`;
const ttl = 10;   // 10 seconds: absorbs 95% of the reads

Expected improvement: an additional 60-70% reduction in the remaining queries.

Cost: 1 day. It is done after the N+1 because the N+1 is what generates the volume.

Action 3: remove limits.cpu (or raise it to 2).

resources:
  requests:
    cpu: 500m
    memory: 800Mi
  limits:
    # No limits.cpu: it eliminates the 31.2% of throttled periods
    memory: 1600Mi

Expected improvement:

Throttling: 31.2% -> 0%
p99: 4.2 s -> ~1.5 s (it removes the 75 ms waits)
p95: a 10-15% improvement

Cost: 10 minutes. It is the cheapest on the list, but not the first, because with the disk at 91% the CPU is not the limiting factor. Once actions 1 and 2 are done, it will be.

(e) What I would NOT do, and why:

I would NOT raise maxReplicas nor scale bookings-api.

It is the first thing many people do and it is exactly the opposite of what is needed:

With 6 replicas: 3,840 queries/s. Disk at 91%.
With 12 replicas: 7,680 queries/s. Disk at... it cannot exceed 100%.

Result: the queries queue up. Latency GOES UP.
Connections run out (12 replicas x pool). More errors.
Cost: double. Performance: worse.

I would NOT provision more IOPS on the StorageClass yet.

Going from 3,000 to 10,000 IOPS would relieve the symptom and cost considerably more per month. But the problem is not a lack of IOPS: it is that 24 queries are made where 2 should be. Paying to absorb an inefficiency is the worst possible decision, because the inefficiency is still there and it grows with the traffic.

After fixing the N+1, the 3,000 IOPS will be more than enough.

I would NOT raise PostgreSQL's max_connections.

With 118 of 200 in use, connections are not the limiting factor. And raising max_connections would make things worse: more PostgreSQL processes competing for the same saturated disk, with more context switching.

I would NOT add more memory to the pods.

39% usage. It is not the problem.

The lesson: the three actions I would not take are the three a team under pressure would take first, because they are the quickest to apply. They all cost money and none of them fixes anything. Diagnosing before acting saves more than any optimisation.

Solution 2

(a) Maximum pool size per replica:

bookings-postgres max_connections:                 200
Reserve for maintenance, backups and metrics:       40 (20%)
Available for applications:                        160
Already used by bookings-api (via PgBouncer):       75
------------------------------------------------------
Available for business-api:                         85

business-api maxReplicas: 20
Maximum pool per replica: 85 / 20 = 4.25  ->  4 connections

4 connections per replica.

(b) Is that pool enough for the peak? Let's apply Little's law.

Concurrency = Arrival rate x Service time

Each request makes 6 queries of 14 ms = 84 ms of database time.

Queries per second one replica sustains with 4 connections:
  4 connections / 0.014 s = 285.7 queries/s

Requests per second per replica:
  285.7 / 6 queries = 47.6 requests/s

With 20 replicas: 20 x 47.6 = 952 requests/s
Peak traffic: 180 requests/s
Capacity with 20 replicas: 952 requests/s

952 >> 180.  THE POOL IS MORE THAN ENOUGH.

Let's check how many replicas are really needed:

180 requests/s / 47.6 per replica = 3.78  ->  4 replicas

4 replicas suffice for the peak. The maxReplicas: 20 is very generous, but that is not a bad thing: it gives headroom if the traffic estimate falls short, and the replicas only exist if the autoscaler asks for them.

An additional check: what if all 20 replicas were active at once?

20 replicas x 4 connections = 80 connections.
Available for business-api: 85.
80 < 85. IT FITS, with 5 to spare.

(c) Latency budget for 900 ms:

Component p95 budget % Justification
Client-to-Ingress network 60 ms 6.7% Agencies use an API, not a browser: less network latency
Ingress (nginx) + TLS 15 ms 1.7% TLS termination and routing
Internal cluster network 10 ms 1.1% 2 hops
business-api (own logic) 90 ms 10.0% Validating 6 tickets, serialisation, business rules
bookings-postgres (6 queries) 300 ms 33.3% 6 × 50 ms of budget per query
redis-cache (availability check) 20 ms 2.2% 2 lookups of 10 ms
pricing-engine (batch prices) 130 ms 14.4% Computation for 6 tickets
Writing and committing the transaction 60 ms 6.7% Commit with WAL fsync
Safety margin 215 ms 23.9% Variability, retries, GC, contention
TOTAL 900 ms 100%

Checking the margin rule:

Sum of the components (excluding margin): 685 ms
685 / 900 = 76.1%  <=  75%... right at the limit.

Acceptable, but tight. Worth keeping an eye on.

(d) Is it achievable with the measured data?

MEASURED in rutas-norte-pre with 4 replicas:
  Total p95:                640 ms
  Of which, PostgreSQL:     310 ms
  The rest (application, network, pricing-engine): 330 ms

BUDGETED:
  PostgreSQL: 300 ms   ->  measured 310 ms. DEVIATION: +10 ms (+3.3%)
  The rest:   385 ms   ->  measured 330 ms. Below budget. GOOD.
  Total:      900 ms   ->  measured 640 ms. MARGIN: 260 ms (28.9%)

YES, it is achievable. The system already meets the objective with a 29% margin.

But there are two important observations:

Observation 1: the PostgreSQL queries are right on budget.

310 ms measured / 6 queries = 51.7 ms per query.
Budget: 50 ms per query.

And the brief says the average duration of each query is 14 ms.
310 ms measured / 6 = 51.7 ms of p95 per query, against 14 ms of average.

RATIO p95/average = 3.7. That is high.

A p95/average ratio of 3.7 suggests high variability: some queries are much slower than others. It deserves investigation:

SELECT
  substring(query, 1, 60),
  calls,
  round(mean_exec_time::numeric, 2) AS mean,
  round(stddev_exec_time::numeric, 2) AS stddev,
  round(max_exec_time::numeric, 2) AS max
FROM pg_stat_statements
WHERE query LIKE '%tickets%'
ORDER BY total_exec_time DESC;

If one of the 6 queries is much slower than the others, it is a candidate for an index.

Observation 2: the measurement is with 4 replicas and low traffic.

The 640 ms p95 was measured under favourable conditions. The measurement has to be repeated with the peak load (180 rps) before signing off on the budget. Queueing theory warns us that latency grows non-linearly with utilisation.

What I would do if it were not achievable:

In order of return:

  1. Reduce the 6 queries to 1 or 2. A batch of 6 tickets should be retrievable with WHERE id = ANY($1). From 300 ms to 60 ms. It is the highest-impact change.
  2. Parallelise the independent queries. If the 6 do not depend on each other, Promise.all() runs them in parallel: from 300 ms sequential to ~60 ms.
  3. Cache the batch prices if the agencies query repeated routes.
  4. Renegotiate the objective with the business. 900 ms to book 6 tickets may be too demanding; 1,500 ms may be perfectly acceptable for a B2B API.

(e) The KEDA threshold:

Measured capacity per replica: 47.6 requests/s (limited by the pool).

But we have to check what the REAL limiting factor is:
  - By pool: 47.6 rps
  - By CPU: it has to be measured with the stress test

We take the more restrictive one. Suppose the stress test gives an inflection
point of 42 rps per replica (CPU limits before the pool does).

We apply the 30% margin to cover the start-up time:
  42 x 0.7 = 29.4  ->  threshold: 29 rps per replica

Check at the peak:
  180 rps / 29 = 6.2  ->  7 replicas.
  7 replicas x 4 connections = 28 connections to PostgreSQL. Very comfortable.
# k8s/environments/pro/scaledobject-business-api.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: business-api
  namespace: rutas-norte-pro
spec:
  scaleTargetRef:
    name: business-api

  # A floor of 3 replicas: one per zone (09-05). It is a B2B API with a service
  # agreement; it cannot have cold starts.
  minReplicaCount: 3

  # A ceiling of 20 replicas.
  # CONNECTION CHECK: 20 x 4 = 80 <= 85 available. It fits.
  # DO NOT RAISE without recalculating the pool (see the ConfigMap).
  maxReplicaCount: 20

  fallback:
    failureThreshold: 3
    replicas: 7        # The peak capacity: a known safe number

  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Pods
              value: 4
              periodSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 600
          policies:
            - type: Percent
              value: 25
              periodSeconds: 120

  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc.cluster.local:9090
        metricName: business_api_requests_per_second
        query: |
          sum(rate(business_api_requests_total{namespace="rutas-norte-pro"}[2m]))
        # 29 requests/s per replica.
        #
        # Calculation:
        #   Measured inflection point (stress test): 42 rps per replica.
        #   A 30% margin to cover start-up: 42 x 0.7 = 29.4 -> 29.
        #
        # Pool check (Little's law):
        #   4 connections / 14 ms = 285.7 queries/s per replica
        #   285.7 / 6 queries per request = 47.6 rps per replica
        #   The POOL supports 47.6 rps; CPU limits earlier at 42.
        #   The pool is NOT the limiting factor. Correct.
        threshold: "29"
        activationThreshold: "2"
      authenticationRef:
        kind: ClusterTriggerAuthentication
        name: prometheus-read

And the ConfigMap with the calculation documented:

apiVersion: v1
kind: ConfigMap
metadata:
  name: business-api-config
  namespace: rutas-norte-pro
data:
  # CONNECTION POOL: 4 per replica.
  #
  # Calculation:
  #   bookings-postgres max_connections:           200
  #   Reserve for maintenance and metrics (20%):   -40
  #   Used by bookings-api (via PgBouncer):        -75
  #   Available for business-api:                    85
  #   business-api maxReplicas:                      20
  #   Pool = 85 / 20 = 4.25  ->  4
  #
  # DO NOT RAISE the pool or maxReplicas without redoing this calculation.
  # With 20 replicas, each extra connection per replica is 20 more connections.
  PG_POOL_MAX: "4"
  PG_POOL_MIN: "1"
  PG_POOL_ACQUIRE_TIMEOUT_MS: "2000"

Solution 3

Diagnosis: a memory leak in bookings-api that is destroying the cache's effectiveness.

Let's go step by step, because the causal chain is the interesting part.

The clues and what they rule out:

Observation What it indicates
No deployments in 3 weeks It is not a code change. It is something cumulative
Memory grows every day and never comes down The unmistakable signature of a memory leak
OOMKilled events for 4 days Memory has reached the limit
CPU unchanged (310m) It is not a compute problem
Traffic only +8% It is not organic growth
PostgreSQL p95 unchanged (8 ms) PostgreSQL is not the bottleneck
Disk io_time unchanged (0.34) Nor is the disk
Redis hits: 94.7% → 71.2% Here is the key
Queries to PostgreSQL: 340 → 1,890 (5.6×) A consequence of the degraded cache

The causal chain:

1. There is a memory leak in bookings-api. Memory grows ~40Mi/day.
   (620Mi -> 1,480Mi in 21 days = 41Mi/day)

2. On reaching ~1,500Mi of a 1,600Mi limit, the pod is OOMKilled.

3. EVERY OOMKilled RESTART EMPTIES THE PROCESS'S IN-MEMORY CACHE.
   If bookings-api keeps a local cache (in addition to Redis), it is lost.

4. And MORE IMPORTANTLY: the restarts break the Redis access pattern.
   A freshly started pod makes "cold" queries: it asks for keys that are
   not cached because the work of populating the cache was done by the
   previous pod with its session data.

5. The Redis hit ratio falls from 94.7% to 71.2%.

6. The 28.8% of misses (against 5.3% before) translates into
   5.4 times more queries to PostgreSQL: 340 -> 1,890/s.

7. More queries -> more latency per request (even though each individual
   query still takes 8 ms, there are now many more per request).

8. The p95 latency rises: 94 ms -> 280 ms.

9. KEDA detects more latency or more requests per replica and scales:
   4 -> 9 replicas.

10. MORE REPLICAS MAKE THE CACHE WORSE: more pods, each with fewer
    requests, more scattered access patterns, and more pods accumulating
    the leak in parallel.

    IT IS A SELF-REINFORCING CYCLE.

The detail that confirms the diagnosis: the CPU has not changed.

If the problem were more real load, CPU would have gone up. If it were a PostgreSQL problem, the query p95 would have gone up. The only variable that has changed monotonically and steadily is memory. Everything else is a consequence.

Diagnostic commands, in order:

# 1. Confirm the OOMKilled events and their frequency
kubectl get pods -n rutas-norte-pro -l app=bookings-api \
  -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason'
NAME                            RESTARTS   LAST
bookings-api-7c9d4f8b6d-2mkjp   3          OOMKilled
bookings-api-7c9d4f8b6d-4nqzx   2          OOMKilled
bookings-api-7c9d4f8b6d-7wtrv   0          <none>
# 2. Confirm the monotonic memory growth (the definitive proof)
# In Prometheus:
#   container_memory_working_set_bytes{namespace="rutas-norte-pro", container="api"}
# Range: 30 days.
#
# If the line GOES UP AND NEVER COMES DOWN, not even in the night-time troughs,
# it is a leak. If it drops overnight, it is normal GC behaviour.
# 3. Check the Node.js heap configuration
kubectl get deployment bookings-api -n rutas-norte-pro \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="NODE_OPTIONS")].value}{"\n"}'
# 4. See the composition of the process's memory
kubectl exec -n rutas-norte-pro bookings-api-7c9d4f8b6d-7wtrv -c api -- \
  node -e "const m = process.memoryUsage();
    console.log('rss:', (m.rss/1048576).toFixed(0), 'MB');
    console.log('heapTotal:', (m.heapTotal/1048576).toFixed(0), 'MB');
    console.log('heapUsed:', (m.heapUsed/1048576).toFixed(0), 'MB');
    console.log('external:', (m.external/1048576).toFixed(0), 'MB');
    console.log('arrayBuffers:', (m.arrayBuffers/1048576).toFixed(0), 'MB');"
rss: 1412 MB
heapTotal: 1180 MB
heapUsed: 1094 MB
external: 89 MB
arrayBuffers: 34 MB

A heapUsed of 1,094 MB and growing: the leak is in the JavaScript heap, not in native buffers. That points at JavaScript objects piling up.

# 5. Investigate Redis: what has changed?
kubectl exec -n rutas-norte-pro redis-cache-0 -- redis-cli INFO stats | grep -E "keyspace|expired|evicted"
kubectl exec -n rutas-norte-pro redis-cache-0 -- redis-cli INFO memory | grep -E "used_memory_human|maxmemory"
kubectl exec -n rutas-norte-pro redis-cache-0 -- redis-cli DBSIZE
# 6. Correlate restarts with drops in the hit ratio.
# In Grafana, overlay:
#   - kube_pod_container_status_restarts_total{container="api"}
#   - redis_keyspace_hits_total / (hits + misses)
#
# If every restart coincides with a drop in the hit ratio,
# the causal chain is CONFIRMED.
# 7. Capture a heap snapshot to find the leak
kubectl exec -n rutas-norte-pro bookings-api-7c9d4f8b6d-7wtrv -c api -- \
  kill -USR2 1
# (requires the application to have the heapdump handler)

The most likely leak causes in a Node.js API:

Cause How it shows up
A global Map or array that grows without bound An in-memory cache with no TTL and no size limit
Event listeners that are never removed emitter.on() with no matching off()
Timers that are never cancelled setInterval per request with no clearInterval
Closures capturing large objects Callbacks holding on to the whole request object
A connection pool that never releases Connections that are not returned to the pool

The main suspect, given the context: an in-memory cache inside bookings-api, added perhaps a month ago, with no size limit and no eviction policy. It fits perfectly:

  • It grows monotonically (every new route queried adds an entry).
  • It grows faster the more varied the traffic is.
  • It never comes down, because nothing evicts entries.
// THE SUSPICIOUS PATTERN
const routeCache = new Map();          // <- No size limit

app.get('/routes', async (req, res) => {
  const key = `${req.query.origin}:${req.query.destination}:${req.query.date}`;
  if (routeCache.has(key)) return res.json(routeCache.get(key));
  const routes = await queryRoutes(req.query);
  routeCache.set(key, routes);         // <- GROWS FOREVER
  res.json(routes);
});

With 8 origins × 7 destinations × 365 dates = 20,440 possible combinations, and each entry taking ~50 KB, the cache can reach 1 GB. Exactly what we are seeing.

Immediate fix (today):

# 1. Raise the memory limit to stop the OOMKilled events while it is
#    investigated. It is NOT a solution: it is a sticking plaster.
resources:
  requests:
    memory: 800Mi
  limits:
    memory: 2560Mi     # From 1600Mi to 2560Mi
env:
  # And adjust the heap accordingly: 65% of 2560 = 1664
  - name: NODE_OPTIONS
    value: "--max-old-space-size=1664"
# 2. An orderly restart of all the replicas to go back to the starting point
kubectl rollout restart deployment/bookings-api -n rutas-norte-pro

This buys a few days: at 41Mi/day and a 2,560Mi limit, the next OOMKilled would arrive in around 6-7 weeks instead of 4 days.

The real fix (this week):

// Replace the unbounded Map with a bounded LRU cache
const { LRUCache } = require('lru-cache');

const routeCache = new LRUCache({
  // HARD CEILING: 500 entries. At ~50 KB per entry, about 25 MB maximum.
  max: 500,
  // A 5-minute TTL: timetables do not change, but it bounds the growth.
  ttl: 1000 * 60 * 5,
  // A size ceiling too, in case some entry is enormous
  maxSize: 50 * 1024 * 1024,   // 50 MB
  sizeCalculation: (value) => JSON.stringify(value).length,
  updateAgeOnGet: false,
});

And the architectural decision worth raising: is this in-memory cache needed at all if redis-cache already exists?

In-process memory cache:
  + Faster (0.001 ms versus 1 ms for Redis)
  - Duplicated in every replica: 9 replicas = 9 copies of the same data
  - Lost on every restart
  - Fragments the hit ratio: each replica caches its own
  - It is the cause of this leak

Cache in Redis:
  + Shared across ALL replicas: a far better hit ratio
  + Survives restarts
  + A single place to tune the TTL and the eviction policy
  - 1 ms of network latency

RECOMMENDATION: remove the in-memory cache and use only Redis.
The millisecond of difference is irrelevant against the 300 ms
latency budget, and it eliminates the leak, the fragmentation and the
restart problem in one go.

And the preventive measures:

# ALERT 1: monotonic memory growth (the one that would have caught this
# on day 3, not on day 21).
# deriv() computes the slope: if it is positive for 6 hours straight,
# memory only ever goes up.
deriv(
  container_memory_working_set_bytes{namespace="rutas-norte-pro", container="api"}[6h]
) > 0
# ALERT 2: any OOMKilled
increase(kube_pod_container_status_restarts_total{namespace="rutas-norte-pro"}[1h]) > 0
and on(pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
# ALERT 3: a drop in the cache hit ratio
(
  rate(redis_keyspace_hits_total[10m])
  /
  (rate(redis_keyspace_hits_total[10m]) + rate(redis_keyspace_misses_total[10m]))
) < 0.85

Which test would have caught it earlier?

The soak test from section 3, without a shadow of a doubt.

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: 100,
      timeUnit: '1s',
      duration: '8h',            // EIGHT HOURS of constant load
      preAllocatedVUs: 100,
      maxVUs: 300,
    },
  },
};

Why this one and not another:

Test Would it have caught the leak?
Smoke (2 min) No. Far too short
Load (20 min) No. The leak is 41Mi/day = 1.7Mi/hour. In 20 minutes that is 0.6Mi: noise
Stress (25 min) No. It measures the limit, not the evolution over time
Spike (8 min) No. Far too short
Soak (8 h) YES. 8 hours × 1.7Mi/h = 14Mi of growth, clearly visible on the graph

And with a more aggressive soak test (peak traffic for 8 hours), the growth would be much larger and obvious within the first hour.

What to watch during the soak test:

# If memory goes up and does NOT come down over 8 hours of CONSTANT load, there is a leak.
container_memory_working_set_bytes{namespace="rutas-norte-pre", container="api"}

# If connections to PostgreSQL grow without being released, there is a connection leak.
pg_stat_activity_count{datname="bookings"}

# If the p95 latency grows 5% every hour, there is progressive degradation.
histogram_quantile(0.95, sum(rate(rutasnorte_request_duration_seconds_bucket[10m])) by (le))

The lesson of the exercise, and of the whole module: an 8-hour soak test run once a month costs a few hours of a test node and catches problems that in production take three weeks to show up, degrade the service progressively, drive up infrastructure cost through needless scaling, and are extraordinarily hard to diagnose once they have built a ten-step causal chain.

Conclusion

Performance tuning is what makes everything else in module 9 affordable. Scaling an inefficient application works, but it multiplies the inefficiency by the number of replicas: you pay thirty times over, you boot nodes you would not need, and you hit the capacity limits far sooner.

The essentials:

  • Methodology before tricks. Measure the baseline, find the real bottleneck, change one thing at a time, measure again under the same conditions, and stop when the objective is reached.
  • Load tests with k6 must use ramping-arrival-rate (real users do not wait), include realistic pauses between steps and respect the proportions between operations. The five types — smoke, load, stress, spike, soak — answer different questions.
  • Percentiles, never the average. With avg=142ms and p99=2.1s, the average is a reassuring lie. And on a page with 30 requests, one user in four suffers the p99.
  • The inflection point defines the real capacity, and queueing theory explains why: at 90% utilisation the wait is already nine times the service time. That is the mathematical justification for the 60-70% HPA target.
  • The connection pool is the most expensive trap in autoscaling. maxReplicas × pool ≤ max_connections × 0.8, or PgBouncer to decouple the number of connections from the number of replicas.
  • The application is where the problem is. The N+1, the missing indexes and a badly used cache accounted for 92% of the Rutas Norte improvement. The Kubernetes tuning accounted for 5%.
  • CPU throttling appears far before 100%: with four threads and a 100 ms quota, it runs out in 25 ms and the pod is stopped 75% of the time while showing 100% utilisation. Watch container_cpu_cfs_throttled_periods_total.
  • Removing limits.cpu is defensible for services on the critical path and a bad idea for batch jobs, databases and multi-tenant environments. The memory limit is always kept: memory is not compressible.
  • Fast start-up is an autoscaling requirement, not a luxury. From 113 to 12 seconds with a lightweight image, a preload DaemonSet and well-tuned probes.
  • The trailing dot on external FQDNs removes three out of every four DNS queries caused by ndots: 5. One character.
  • The latency budget turns a business objective into per-component objectives, with an explicit 25% margin, and every line becomes an alert that says exactly who has blown their share.

The measured result of the 2026 May bank holiday: zero minutes of outage against 47, 0.08% errors against 12.4%, a capacity per replica of 224 requests per second against 85, and a cost per thousand bookings of €1.12 against €8.90. The platform did not just hold up: it held up while costing eight times less per ticket sold.

With this we close module 9. Rutas Norte has a platform that grows when it needs to, with pods of the right size, on nodes that appear on their own, anticipating known events, without breaking during maintenance, and tuned so as not to waste the capacity it pays for.

And yet, there is something that has been piling up over nine modules and can no longer be ignored.

Take a look at the k8s/ directory. There is the bookings-api Deployment, its Service, its ConfigMap, its Secret, its HPA, its VPA, its PDB, its NetworkPolicy, its ServiceMonitor and its ScaledObject. Multiplied by six components. And multiplied again by three environments, because rutas-norte-dev has minReplicas: 1 and rutas-norte-pro has minReplicas: 4, because the images carry different tags, because the resource limits differ, because there is no TLS in development.

That is more than a hundred and twenty YAML files, many of them identical apart from three lines. When the bookings-api port changes, it has to be touched in nine places. When a new version is deployed, somebody runs a sequence of kubectl apply commands from memory, in the right order, hoping not to forget any. When somebody asks "what version is in pre-production?", the honest answer is "let me go and look".

And we have brushed against the problem several times without solving it: the replicas field that has to be removed from the versioned manifest (09-01), the VPA recommendation that has to be carried by hand into a pull request (09-02), the PDB that was left with a value from a previous May bank holiday (09-05). They are all symptoms of the same thing: the source of truth is scattered across a hundred and twenty files and the memory of three people.

In module 10, Kubernetes Ecosystem and Tooling, we attack exactly that: reproducible local environments with minikube and kind, building clusters with kubeadm, packaging and parameterisation with Helm, per-environment customisation without duplication using Kustomize, and the definitive leap to GitOps with Argo CD and Flux, where the repository stops being a folder of files somebody applies by hand and becomes the single source of truth for what is in the cluster. We will finish with managed Kubernetes — EKS, AKS and GKE — and what changes when the control plane is not yours.

◄ Previous
◄
Next ►
►

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved