The Project | About Us | Contribute | Donations | License

HOME

At the end of the previous lesson, Rutas Norte finally had a memory: Prometheus stores thirty days of metrics for every component and we can ask it anything in PromQL. But one problem was left open: all of that lives in a spartan interface where you have to type queries by hand, and nobody is looking at it at three in the morning. Data nobody sees and nobody is warned about has solved nothing.

This lesson takes that leap: from the one-off query to the dashboard that tells a story at a glance, and from there to the alert that wakes somebody up because it genuinely needs to. We will look at Grafana for visualization, PrometheusRule for defining when something is wrong, and Alertmanager for deciding who gets told, when and with what grouping. And above all we will look at the judgement that separates a useful alerting system from a noise generator everybody ends up muting: alert on symptoms the customer perceives, not on causes.

Grafana: what it is and how it connects to Prometheus
The anatomy of a panel and the types that actually get used
Dashboard variables: one dashboard for the three environments
Importing community dashboards
The Rutas Norte dashboard, panel by panel
Provisioning dashboards as code
Alerting rules with PrometheusRule and the importance of for
The Rutas Norte alert catalogue
Alertmanager: routes, receivers, inhibition and silences
The judgement call: alert on symptoms, not on causes
Rutas Norte SLOs and error budget
Common mistakes and tips
Exercises

Grafana: what it is and how it connects to Prometheus

Grafana is a visualization tool that connects to data sources — Prometheus, Elasticsearch, PostgreSQL, dozens more — and draws dashboards. It does not store metrics: it only queries and paints. If Prometheus goes down, Grafana goes blank.

That separation of responsibilities is deliberate and very healthy: Prometheus specialises in collecting, storing and evaluating; Grafana in showing. Each does one thing well.

When we installed the kube-prometheus-stack in 07-03, Grafana came with it and with the data source already configured. The chart automatically creates the connection to the Prometheus Service, so there is no URL to enter by hand.

# Check that Grafana is running
kubectl -n monitoring get pods -l app.kubernetes.io/name=grafana

NAME                                 READY   STATUS    RESTARTS   AGE
monitoring-grafana-6d84b8c7f9-w2xnk  3/3     Running   0          2d4h

Note the 3/3: as well as the Grafana container there are two sidecars, one watching the dashboard ConfigMaps and another the data source ones. The first will be key in section 6.

Access

kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
# Open http://localhost:3000

The initial credentials are in a Secret created by the chart:

kubectl -n monitoring get secret monitoring-grafana \
  -o jsonpath='{.data.admin-user}' | base64 -d; echo

kubectl -n monitoring get secret monitoring-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

admin
change-in-production

Before going any further. We set that value in the Helm values file in 07-03, and in rutas-norte-pro it is unacceptable for two reasons: it is in plain text in a file versioned in Git, and it is a weak password. In production you have to (a) generate the password as an external Secret and reference it with admin.existingSecret, and (b) better still, delegate authentication to the company's identity provider (OAuth or LDAP), disabling the local user. Grafana gives read access to all the platform's metrics, including the business ones.

In rutas-norte-pro we would publish Grafana with an Ingress and TLS managed by cert-manager, exactly as we did in 04-04 and 04-05:

# k8s/environments/pro/ingress-grafana.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/auth-type: basic          # an extra layer
    nginx.ingress.kubernetes.io/auth-secret: grafana-basic-auth
spec:
  ingressClassName: nginx
  tls:
    - hosts: [metrics.rutasnorte.example]
      secretName: grafana-tls
  rules:
    - host: metrics.rutasnorte.example
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: monitoring-grafana
                port:
                  number: 80

Verifying the data source

In the interface: Connections → Data sources → Prometheus. It should appear configured and with the "Save & test" button coming back green. The internal URL it uses is the Service's DNS name (04-03):

http://monitoring-kube-pr-prometheus.monitoring.svc.cluster.local:9090

One configuration detail worth adjusting: the Scrape interval field must match the interval of your ServiceMonitors (30 s in our case). Grafana uses it to calculate the $__rate_interval variable, which we will see shortly.

The anatomy of a panel and the types that actually get used

A Grafana panel has four parts, and understanding them avoids 90 % of illegible dashboards.

The query

One or more PromQL expressions. Grafana offers two modes: the visual builder (useful for learning) and code mode (the one you will use as soon as you know PromQL).

Two special variables Grafana substitutes automatically:

Variable	What it is	When to use it
`$__rate_interval`	A window adapted to the zoom level and the scrape interval	Always inside `rate()`
`$__interval`	The graph's resolution based on its width in pixels	In `increase()` over variable windows

Always use rate(metric[$__rate_interval]) rather than rate(metric[5m]). With a fixed window, zooming out to 7 days makes the graph noisy or produces gaps; $__rate_interval adapts and guarantees there are always at least four samples in the window.

The legend with label templates

By default, Grafana shows the legend with all the series labels:

{__name__="rutasnorte_requests_total", code="200", component="bookings-api", environment="pro", instance="10.244.2.17:9090", job="bookings-api", method="GET", namespace="rutas-norte-pro", pod="bookings-api-7d9f8c4b5-x2klm", route="/api/routes", version="2.8.1"}

Illegible. With a template in the Legend field:

{{route}} · {{code}}

/api/routes · 200
/api/bookings · 201
/api/bookings · 422

Rule of thumb: the legend must fit on one line and contain only what distinguishes that series from the others in the panel.

Units and thresholds

Under Standard options → Unit. This setting matters more than it looks:

Metric	Correct unit	Without it you see
Latency in seconds	`seconds (s)`	`0.412` instead of `412 ms`
Error rate (0–1)	`Percent (0.0-1.0)`	`0.023` instead of `2.3 %`
Memory in bytes	`bytes (IEC)`	`536870912` instead of `512 MiB`
Requests per second	`requests/sec (rps)`	A bare number with no context

Thresholds colour the panel according to the value. For the bookings-api error rate: green up to 0.01, amber from 0.01 to 0.05, red above that. A well-painted dashboard lets you spot the problem from the other side of the office, without reading a number.

The visualization type

Of the dozens Grafana offers, in practice four get used:

Type	When to use it	Example in Rutas Norte
Time series	The evolution of a value over time. 70 % of panels	Requests/s, p95 latency, memory
Stat	A big number, the current state of an indicator	Bookings confirmed in the last hour
Table	Comparing the state of several entities at once	The state of every pod, with restarts and age
Heatmap	The complete distribution of a histogram over time	The distribution of latencies, not just the p95

About the heatmap: it is the most under-used type and the one that tells you most about latency. A p95 of 400 ms can hide two very different populations (a mass at 20 ms and a tail at 3 s) or a uniform distribution. The heatmap tells them apart at a glance, and it feeds directly off the histogram buckets:

sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment="$environment"}[$__rate_interval]))

With the query format set to Heatmap and the Format: Heatmap option enabled.

Types worth avoiding: gauge dials take up a lot of room and say little, and pie charts are almost always a bad choice for time-based data.

Dashboard variables: one dashboard for the three environments

Without variables, you would have to duplicate the dashboard three times: one for rutas-norte-dev, one for pre and one for pro. Three copies that fall out of sync with the first modification.

Variables turn the dashboard into a template with drop-downs at the top.

Defining them

Under Dashboard settings → Variables.

Variable environment (type Query):

Name:          environment
Type:          Query
Data source:   Prometheus
Query:         label_values(rutasnorte_requests_total, environment)
Sort:          Alphabetical (asc)
Multi-value:   No

label_values(metric, label) is a Grafana-specific function that returns every distinct value that label has. The drop-down fills itself with dev, pre and pro, and if we add a fourth environment tomorrow it will appear automatically.

Variable namespace (dependent on the previous one):

Name:          namespace
Query:         label_values(kube_pod_info{namespace=~"rutas-norte-$environment"}, namespace)

Variable component (multi-value):

Name:          component
Query:         label_values(kube_pod_info{namespace="$namespace"}, created_by_name)
Multi-value:   Yes
Include All:   Yes
All value:     .*

With Multi-value you can select several at once; with Include All, the "All" option.

Variable interval (type Interval, to adjust the rate window):

Name:    interval
Type:    Interval
Values:  1m,5m,15m,30m,1h,6h,24h

Using them in queries

# With a simple variable
sum(rate(rutasnorte_requests_total{environment="$environment"}[$__rate_interval]))

# With a multi-value variable: you MUST use =~ and the regex format
sum by (pod) (rate(container_cpu_usage_seconds_total{
  namespace="$namespace",
  pod=~"$component.*"
}[$__rate_interval]))

A detail that causes plenty of headaches: with multi-value variables, Grafana substitutes $component with (bookings-api|web-store). That only works with the =~ operator, never with =. If you use =, the query returns nothing and there is no error message to explain why.

To force the format explicitly: ${component:regex} or ${component:pipe}.

The result

A single dashboard, versioned once, that serves all three environments. You switch the drop-down from pro to pre and see the same panels with the other environment's data. And if we add a new component with the Rutas Norte labelling convention, it appears in the drop-down by itself.

Importing community dashboards

Grafana has a public catalogue with thousands of dashboards. They are imported by their numeric identifier: Dashboards → New → Import → paste the ID.

The ones that genuinely matter for a Kubernetes cluster:

ID	Dashboard	What it shows
315	Kubernetes cluster monitoring	Cluster overview: nodes, pods, network
1860	Node Exporter Full	Everything node-exporter exposes, very complete
6417	Kubernetes Cluster (Prometheus)	Resources by namespace and by workload
9628	PostgreSQL Database	For the `bookings-postgres` exporter
11835	Redis Dashboard	For `redis-cache`
7645	NGINX Ingress Controller	Traffic through the 04-04 Ingress

On top of that, the kube-prometheus-stack already installs a couple of dozen very good dashboards (usage by namespace, by pod, by node, control plane state). Before importing anything, look at what you already have.

Why they are worth reviewing before trusting them

A community dashboard is code written by a stranger for their cluster, not yours. Mandatory checks:

Do the metrics exist in your Prometheus? Many dashboards use names from old versions of kube-state-metrics or cAdvisor. An empty panel does not always mean "everything is fine": it can mean "this metric does not exist here".
Does the data source name match? On import, Grafana asks you to map the data source. If the dashboard expects one called Prometheus and yours is named differently, every panel fails.
Are the queries reasonable at your scale? A panel with rate(...[1m]) over 50,000 series can take 20 seconds and punish Prometheus every time somebody opens the page.
Does it reflect your reality? A generic Kubernetes dashboard has no idea what bookings-api is or what a confirmed booking is. It is fine for infrastructure; it does not replace your own dashboard.
Has it been updated recently? A dashboard from 2019 will use metrics that no longer exist.

The recommended strategy for Rutas Norte:

Community dashboards for generic infrastructure (nodes, PostgreSQL, Redis, Ingress). We add nothing by reinventing them.
Our own dashboard for the platform: the four golden signals of our components and the business metrics. Nobody in the community can write that for us.

The Rutas Norte dashboard, panel by panel

We design the main dashboard. Structure: a summary row at the top and one row per component below, all collapsible.

flowchart TB
    subgraph DASH["Dashboard: Rutas Norte Platform — [environment] [namespace]"]
        subgraph F0["Row 0: Executive summary"]
            A["Bookings/min<br/>Stat"]
            B["Error rate<br/>Stat"]
            C["p95 latency<br/>Stat"]
            D["Pods not ready<br/>Stat"]
        end
        subgraph F1["Row 1: bookings-api"]
            E["Traffic"] --- F["Errors"] --- G["Latency p50/p95/p99"] --- H["Saturation"]
        end
        subgraph F2["Row 2: bookings-postgres"]
            I["Connections"] --- J["Transactions/s"] --- K["Size on disk"] --- L["Deadlocks"]
        end
        subgraph F3["Row 3: remaining components"]
            M["web-store"] --- N["redis-cache"] --- O["notifications-worker"] --- P["occupancy-reports"]
        end
    end

Row 0 — Executive summary

Four Stat panels that answer "is the platform doing well?" at a glance.

Panel 1: Bookings confirmed per minute. The business metric.

sum(rate(rutasnorte_bookings_confirmed_total{environment="$environment"}[$__rate_interval])) * 60

Type: Stat, with a trend graph in the background (Graph mode: Area).
Unit: short, suffix bkg/min.
Thresholds: red below 1, amber up to 5, green above.
Why it is the dashboard's first panel: if this drops to zero, it makes absolutely no difference that the pods are Running. The platform exists to sell tickets.

Panel 2: Error rate.

sum(rate(rutasnorte_requests_total{environment="$environment", code=~"5.."}[$__rate_interval]))
  /
sum(rate(rutasnorte_requests_total{environment="$environment"}[$__rate_interval]))

Unit: Percent (0.0-1.0).
Thresholds: green < 0.5 %, amber < 2 %, red ≥ 2 %.

Panel 3: Overall p95 latency. We use the recording rule we created in 07-03:

max(bookingsapi:latency_p95:5m{environment="$environment"})

Unit: seconds (s). Thresholds: green < 0.3 s, amber < 1 s, red ≥ 1 s.

Panel 4: Pods not ready. Straight from kube-state-metrics:

sum(kube_deployment_status_replicas_unavailable{namespace="$namespace"})
  +
sum(kube_statefulset_status_replicas_current{namespace="$namespace"}
    - kube_statefulset_status_replicas_ready{namespace="$namespace"})

Thresholds: green 0, red ≥ 1. Any non-zero value is an anomaly.

Row 1 — `bookings-api`: the four golden signals

Traffic (Time series):

sum by (route) (rate(rutasnorte_requests_total{environment="$environment"}[$__rate_interval]))

Legend: {{route}}. Unit: reqps.

Errors (Time series, with two queries overlaid):

# A: our platform's error rate
sum(rate(rutasnorte_requests_total{environment="$environment", code=~"5.."}[$__rate_interval]))
  / sum(rate(rutasnorte_requests_total{environment="$environment"}[$__rate_interval]))

# B: the external gateway's failure rate
sum(rate(rutasnorte_payment_gateway_calls_total{environment="$environment", result=~"error|timeout"}[$__rate_interval]))
  / sum(rate(rutasnorte_payment_gateway_calls_total{environment="$environment"}[$__rate_interval]))

Overlaying the two is a deliberate design decision: it lets you see in one glance whether our errors coincide with the external provider's. That visual correlation saves twenty minutes of investigation during an incident.

Latency (Time series, three percentiles):

histogram_quantile(0.50, sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment="$environment"}[$__rate_interval])))
histogram_quantile(0.95, sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment="$environment"}[$__rate_interval])))
histogram_quantile(0.99, sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment="$environment"}[$__rate_interval])))

Legends: p50, p95, p99. Seeing all three together is what distinguishes "everything is slow" (all three rise) from "there is a tail of pathological requests" (only the p99 rises).

Saturation (Time series, two axes):

# CPU consumed against the limit
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="$namespace", pod=~"bookings-api-.*", container="api"}[$__rate_interval]))
  / sum by (pod) (kube_pod_container_resource_limits{namespace="$namespace", pod=~"bookings-api-.*", resource="cpu"})

# Fraction of throttled periods: the confirmation of throttling
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="$namespace", pod=~"bookings-api-.*"}[$__rate_interval]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="$namespace", pod=~"bookings-api-.*"}[$__rate_interval]))

# The connection pool: the 07-01 phenomenon, now visible
max(rutasnorte_pool_connections_active{environment="$environment"}) / 20

Row 2 — `bookings-postgres`

All from the 06-04 exporter sidecar, connected back in 07-03:

# Connections against the maximum
pg_stat_database_numbackends{datname="bookings"} / pg_settings_max_connections

# Transactions per second (committed and rolled back)
rate(pg_stat_database_xact_commit{datname="bookings"}[$__rate_interval])
rate(pg_stat_database_xact_rollback{datname="bookings"}[$__rate_interval])

# Database size, with projection
pg_database_size_bytes{datname="bookings"}

# Free space on the PVC (this comes from the kubelet, not the exporter)
kubelet_volume_stats_available_bytes{persistentvolumeclaim="data-bookings-postgres-0"}
  / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="data-bookings-postgres-0"}

# Deadlocks: zero is normal, anything else is worth a look
increase(pg_stat_database_deadlocks{datname="bookings"}[$__interval])

Row 3 — The remaining components

notifications-worker — the most important panel is the queue depth:

max(rutasnorte_queue_pending{environment="$environment"})
max(rutasnorte_queue_wait_seconds{environment="$environment"})
sum(rate(rutasnorte_emails_sent_total{environment="$environment", result="ok"}[$__rate_interval])) * 60

occupancy-reports — the 06-03 CronJob. It has no classic golden signals:

# Seconds since the last successful run
time() - kube_job_status_completion_time{job_name=~"occupancy-reports.*"}

# Duration of the last run
kube_job_status_completion_time{job_name=~"occupancy-reports.*"}
  - kube_job_status_start_time{job_name=~"occupancy-reports.*"}

General status panel (Table), very useful as a summary:

kube_pod_container_status_restarts_total{namespace="$namespace"}

With transformations to show pod, container, restarts and age in sortable columns.

A design tip

A dashboard must answer one question, not show everything. Ours answers "is the platform serving customers well and, if not, where is the problem?". Nobody looks at dashboards with forty panels; eight well-chosen panels get looked at every day.

Provisioning dashboards as code

If you build the dashboard by clicking around the interface, it lives in Grafana's internal SQLite database. And that database, in a pod with no persistent volume, disappears when the pod is recreated. It is a disaster that happens to everybody once.

Besides, dashboards are configuration: they belong in Git, reviewed in a pull request and deployed like the rest of the manifests.

The sidecar mechanism

The chart deploys a sidecar container alongside Grafana that watches every ConfigMap in the cluster looking for a specific label. When it finds one, it writes its contents into Grafana's dashboards directory, which loads it automatically.

# Fragment of the chart values (07-03) that enables the mechanism
grafana:
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard      # <-- the label it looks for
      labelValue: "1"
      searchNamespace: ALL          # searches in every namespace
      folderAnnotation: grafana_folder
      provider:
        foldersFromFilesStructure: true

The dashboard ConfigMap

# k8s/base/monitoring/dashboard-rutas-norte.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dashboard-rutas-norte
  namespace: monitoring
  labels:
    grafana_dashboard: "1"          # the sidecar detects it by this label
    app.kubernetes.io/part-of: rutas-norte
  annotations:
    grafana_folder: "Rutas Norte"   # the folder it will appear in
data:
  rutas-norte-general.json: |
    {
      "title": "Rutas Norte Platform — Overview",
      "uid": "rutasnorte-general",
      "tags": ["rutas-norte", "production"],
      "timezone": "Europe/Madrid",
      "refresh": "30s",
      "time": { "from": "now-6h", "to": "now" },
      "templating": {
        "list": [
          {
            "name": "environment",
            "type": "query",
            "datasource": { "type": "prometheus", "uid": "prometheus" },
            "query": "label_values(rutasnorte_requests_total, environment)",
            "current": { "text": "pro", "value": "pro" },
            "sort": 1
          },
          {
            "name": "namespace",
            "type": "query",
            "datasource": { "type": "prometheus", "uid": "prometheus" },
            "query": "label_values(kube_pod_info{namespace=~\"rutas-norte-$environment\"}, namespace)",
            "sort": 1
          }
        ]
      },
      "panels": [
        {
          "id": 1,
          "title": "Bookings confirmed per minute",
          "type": "stat",
          "gridPos": { "h": 5, "w": 6, "x": 0, "y": 0 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "sum(rate(rutasnorte_bookings_confirmed_total{environment=\"$environment\"}[$__rate_interval])) * 60",
              "legendFormat": "bookings/min"
            }
          ],
          "fieldConfig": {
            "defaults": {
              "unit": "short",
              "decimals": 1,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  { "color": "red",   "value": null },
                  { "color": "yellow","value": 1 },
                  { "color": "green", "value": 5 }
                ]
              }
            }
          },
          "options": { "graphMode": "area", "colorMode": "background" }
        },
        {
          "id": 2,
          "title": "5xx error rate",
          "type": "stat",
          "gridPos": { "h": 5, "w": 6, "x": 6, "y": 0 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "sum(rate(rutasnorte_requests_total{environment=\"$environment\", code=~\"5..\"}[$__rate_interval])) / sum(rate(rutasnorte_requests_total{environment=\"$environment\"}[$__rate_interval]))",
              "legendFormat": "error rate"
            }
          ],
          "fieldConfig": {
            "defaults": {
              "unit": "percentunit",
              "decimals": 2,
              "thresholds": {
                "mode": "absolute",
                "steps": [
                  { "color": "green",  "value": null },
                  { "color": "yellow", "value": 0.005 },
                  { "color": "red",    "value": 0.02 }
                ]
              }
            }
          }
        },
        {
          "id": 3,
          "title": "bookings-api latency",
          "type": "timeseries",
          "gridPos": { "h": 9, "w": 12, "x": 0, "y": 5 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "histogram_quantile(0.50, sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment=\"$environment\"}[$__rate_interval])))",
              "legendFormat": "p50"
            },
            {
              "expr": "histogram_quantile(0.95, sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment=\"$environment\"}[$__rate_interval])))",
              "legendFormat": "p95"
            },
            {
              "expr": "histogram_quantile(0.99, sum by (le) (rate(rutasnorte_request_duration_seconds_bucket{environment=\"$environment\"}[$__rate_interval])))",
              "legendFormat": "p99"
            }
          ],
          "fieldConfig": {
            "defaults": { "unit": "s", "custom": { "fillOpacity": 10 } }
          }
        }
      ]
    }

The recommended workflow

Do not write that JSON by hand. The practical procedure:

Build the dashboard in the Grafana interface, which is comfortable.
Dashboard settings → JSON Model → Copy.
Paste it into the ConfigMap, inside data, with the right indentation.
kubectl apply and review in a pull request.
In Grafana, mark that dashboard as read-only so that nobody edits it in the interface and loses the changes on the next deployment.

kubectl apply -f k8s/base/monitoring/dashboard-rutas-norte.yaml

# The sidecar detects it within seconds; check it in its log
kubectl -n monitoring logs deploy/monitoring-grafana -c grafana-sc-dashboard --tail=10

{"time": "2026-08-06T11:42:03", "msg": "Working on configmap monitoring/dashboard-rutas-norte"}
{"time": "2026-08-06T11:42:03", "msg": "Writing /tmp/dashboards/rutas-norte-general.json"}

When we get to 10-04 (Kustomize) and 10-05 (GitOps), this ConfigMap will be one more artefact in the repository, deployed automatically on merge.

Alerting rules with `PrometheusRule` and the importance of `for`

Grafana has its own alerting engine, but in an environment with the Prometheus Operator the natural thing is to define the alerts in Prometheus through the PrometheusRule resource. The advantages: they live in Git alongside the application, they are evaluated by Prometheus (which is where the data is) and they are routed by Alertmanager.

The anatomy of a rule

- alert: BookingsApiHighErrorRate
  expr: |
    sum(rate(rutasnorte_requests_total{environment="pro", code=~"5.."}[5m]))
      /
    sum(rate(rutasnorte_requests_total{environment="pro"}[5m]))
    > 0.05
  for: 5m
  labels:
    severity: critical
    component: bookings-api
    team: platform
  annotations:
    summary: "bookings-api is returning more than 5% errors"
    description: >
      The 5xx error rate of bookings-api in production is
      {{ $value | humanizePercentage }} over the last 5 minutes.
      Customers cannot complete bookings.
    runbook_url: "https://runbooks.rutasnorte.example/bookings-api-5xx-errors"
    dashboard_url: "https://metrics.rutasnorte.example/d/rutasnorte-general"

Field	Function
`alert`	The alert's name. In `PascalCase`, descriptive, no spaces
`expr`	The PromQL expression. The alert is "active" for every series it returns
`for`	How long it must hold continuously before firing
`labels`	Metadata for routing in Alertmanager. The severity goes here
`annotations`	Text for the human who receives it. They do not affect routing

The importance of `for`

This is the field that separates a usable alerting system from an unbearable one.

Without for, the alert fires as soon as the expression holds a single time. A 30-second spike from a deployment, a one-off restart or a missed scrape generates a notification. Multiplied by twenty alerts and three environments, that is a Slack channel nobody reads.

With for: 5m, Prometheus checks the expression on every evaluation cycle (every 30 s by default) and only fires if it has held on every evaluation in those 5 minutes. A single cycle in which the condition does not hold resets the counter.

The states of an alert:

stateDiagram-v2
    [*] --> Inactive: the expression does not hold
    Inactive --> Pending: the expression holds
    Pending --> Inactive: it stops holding before for elapses
    Pending --> Firing: it holds throughout the for period
    Firing --> Inactive: it stops holding (the resolution is sent)

Only in Firing is anything sent to Alertmanager. Pending is visible in the Prometheus interface (Alerts), which is useful for debugging.

Criteria for choosing for:

Type of alert	Recommended `for`	Reason
Total outage (`up == 0`)	`2m`	Fast, but tolerates a restart or a deployment
Error rate	`5m`	A short spike should not wake anybody
High latency	`10m`	Very noisy if shorter
Disk filling up	`15m`	It is a trend, not an event
Certificate expiring	`1h`	There is no rush at all
Pod in `CrashLoopBackOff`	`10m`	Tolerates a legitimately slow start-up

Annotations and templates

Annotations support Go templates with access to the value and the labels:

Expression	Result
`{{ $value }}`	`0.0734829`
`{{ $value \| humanizePercentage }}`	`7.35%`
`{{ $value \| humanize }}`	`73.5m`
`{{ $value \| humanizeDuration }}`	`1h 12m 30s`
`{{ $labels.pod }}`	`bookings-api-7d9f8c4b5-x2klm`
`{{ $labels.namespace }}`	`rutas-norte-pro`

The runbook_url is not optional. An alert that arrives at three in the morning to somebody who did not write the code and that contains only "BookingsApiHighErrorRate" is useless. With a link to a written procedure — what to check, in what order, what to do, who to escalate to — the alert is actionable. Full runbooks are the subject of 11-06, but the link goes in from day one.

The Rutas Norte alert catalogue

# k8s/base/monitoring/rutas-norte-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: rutas-norte-alerts
  namespace: monitoring
  labels:
    app.kubernetes.io/part-of: rutas-norte
    prometheus: monitoring
spec:
  groups:
    # =====================================================================
    # GROUP 1: SYMPTOMS. What the customer perceives. Top priority.
    # =====================================================================
    - name: rutasnorte.symptoms
      interval: 30s
      rules:

        - alert: PlatformNotSelling
          expr: |
            sum(rate(rutasnorte_bookings_confirmed_total{environment="pro"}[10m])) == 0
            and
            sum(rate(rutasnorte_requests_total{environment="pro"}[10m])) > 0.5
          for: 10m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "Rutas Norte has confirmed NO bookings for 10 minutes"
            description: >
              There is inbound traffic ({{ $value | humanize }} requests/s) but
              zero confirmed bookings. The platform is not selling.
              This is the most important alert in the system.
            runbook_url: "https://runbooks.rutasnorte.example/no-sales"

        # Rationale: the "there is traffic" condition stops it firing in the
        # small hours, when selling nothing for ten minutes is perfectly normal.

        - alert: BookingsApiHighErrorRate
          expr: |
            sum(rate(rutasnorte_requests_total{environment="pro", code=~"5.."}[5m]))
              / sum(rate(rutasnorte_requests_total{environment="pro"}[5m])) > 0.05
          for: 5m
          labels:
            severity: critical
            component: bookings-api
            team: platform
          annotations:
            summary: "bookings-api is returning more than 5% errors"
            description: >
              Error rate of {{ $value | humanizePercentage }} for
              5 minutes. Customers cannot complete their purchases.
            runbook_url: "https://runbooks.rutasnorte.example/api-5xx-errors"

        - alert: BookingsApiHighLatency
          expr: bookingsapi:latency_p95:5m{environment="pro"} > 1
          for: 10m
          labels:
            severity: warning
            component: bookings-api
            team: platform
          annotations:
            summary: "The bookings-api p95 latency is above 1 second"
            description: >
              The 95th percentile is at {{ $value | humanizeDuration }} on
              route {{ $labels.route }}. The Rutas Norte SLO is 500 ms.
            runbook_url: "https://runbooks.rutasnorte.example/api-latency"

        - alert: WebStoreDown
          expr: |
            sum(kube_deployment_status_replicas_available{
              namespace="rutas-norte-pro", deployment="web-store"}) == 0
          for: 2m
          labels:
            severity: critical
            component: web-store
            team: platform
          annotations:
            summary: "No web-store replica is available"
            description: "www.rutasnorte.example is down for every customer."
            runbook_url: "https://runbooks.rutasnorte.example/web-store-down"

    # =====================================================================
    # GROUP 2: CAUSES AND CAPACITY. They warn before there is a symptom.
    # =====================================================================
    - name: rutasnorte.capacity
      interval: 60s
      rules:

        - alert: PostgresDiskWillFill
          expr: |
            predict_linear(
              kubelet_volume_stats_available_bytes{
                persistentvolumeclaim="data-bookings-postgres-0"}[6h],
              4 * 3600
            ) < 0
          for: 30m
          labels:
            severity: critical
            component: bookings-postgres
            team: platform
          annotations:
            summary: "The bookings-postgres disk will fill up in under 4 hours"
            description: >
              Based on the trend of the last 6 hours, the volume
              data-bookings-postgres-0 will run out of space in less than
              4 hours. There are {{ $value | humanize1024 }}B free.
              Expand the PVC (the 05-05 procedure) BEFORE it happens:
              a database with a full disk stops accepting writes.
            runbook_url: "https://runbooks.rutasnorte.example/expand-postgres-pvc"

        # predict_linear fits a regression line over the [6h] window and
        # extrapolates 4*3600 seconds forward. If the result is negative,
        # it means the line crosses zero within 4 hours.
        # It is the difference between warning of a problem and warning of the future.

        - alert: PostgresConnectionsRunningOut
          expr: |
            pg_stat_database_numbackends{datname="bookings"}
              / pg_settings_max_connections > 0.85
          for: 10m
          labels:
            severity: warning
            component: bookings-postgres
            team: platform
          annotations:
            summary: "bookings-postgres at {{ $value | humanizePercentage }} of its connections"
            description: >
              When they run out, bookings-api will fail readiness and leave
              the Endpoints. Review slow queries and the API's pool.
            runbook_url: "https://runbooks.rutasnorte.example/postgres-connections"

        - alert: CertificateExpiringSoon
          expr: |
            (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 15
          for: 1h
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "Certificate {{ $labels.name }} expires in {{ $value | humanize }} days"
            description: >
              cert-manager should renew it automatically (04-05). The fact that it
              has not points to a problem with the issuer or with the ACME challenge.
            runbook_url: "https://runbooks.rutasnorte.example/certificates"

        - alert: ContainerSeverelyThrottled
          expr: |
            sum by (namespace, pod, container) (
              rate(container_cpu_cfs_throttled_periods_total{namespace=~"rutas-norte-.*"}[5m]))
            / sum by (namespace, pod, container) (
              rate(container_cpu_cfs_periods_total{namespace=~"rutas-norte-.*"}[5m]))
            > 0.30
          for: 15m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "{{ $labels.container }} throttled {{ $value | humanizePercentage }} of the time"
            description: >
              The container is hitting its limits.cpu in a sustained way.
              Review the resource recalibration from 07-02.
            runbook_url: "https://runbooks.rutasnorte.example/cpu-throttling"

    # =====================================================================
    # GROUP 3: OBJECT HEALTH. Source: kube-state-metrics.
    # =====================================================================
    - name: rutasnorte.workloads
      interval: 60s
      rules:

        - alert: PodInCrashLoop
          expr: |
            increase(kube_pod_container_status_restarts_total{
              namespace=~"rutas-norte-.*"}[15m]) > 3
          for: 10m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "{{ $labels.pod }} has restarted more than 3 times in 15 min"
            description: >
              Container {{ $labels.container }} in {{ $labels.namespace }}.
              Frequent causes: OOMKilled, start-up failure, or a
              livenessProbe that is too aggressive (07-01).
              Gather evidence with 'kubectl logs --previous' BEFORE touching anything.
            runbook_url: "https://runbooks.rutasnorte.example/crashloop"

        - alert: ReportsCronJobNotRun
          expr: |
            time() - max(kube_job_status_completion_time{
              job_name=~"occupancy-reports.*"}) > 100000
          for: 30m
          labels:
            severity: warning
            component: occupancy-reports
            team: data
          annotations:
            summary: "The occupancy-reports CronJob has not run successfully"
            description: >
              {{ $value | humanizeDuration }} have passed since the last
              successful run. The nightly CronJob should run every
              24 hours. Without reports, the sales department is blind.
            runbook_url: "https://runbooks.rutasnorte.example/reports-cronjob"

        # 100000 seconds ≈ 27.8 h: a margin over the cron's 24 h that
        # tolerates a one-off delay without generating false positives.

        - alert: JobFailed
          expr: kube_job_status_failed{namespace=~"rutas-norte-.*"} > 0
          for: 5m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "Job {{ $labels.job_name }} has failed"
            runbook_url: "https://runbooks.rutasnorte.example/job-failed"

        - alert: NotificationsQueueGrowing
          expr: |
            max(rutasnorte_queue_wait_seconds{environment="pro"}) > 900
          for: 10m
          labels:
            severity: warning
            component: notifications-worker
            team: platform
          annotations:
            summary: "Confirmation emails more than 15 minutes behind"
            description: >
              The oldest message has been {{ $value | humanizeDuration }}
              in the queue. There are customers who have paid and received nothing.
            runbook_url: "https://runbooks.rutasnorte.example/notifications-queue"

    # =====================================================================
    # GROUP 4: META. Alerts about the monitoring system itself.
    # =====================================================================
    - name: rutasnorte.meta
      interval: 60s
      rules:

        - alert: PrometheusTargetDown
          expr: up{namespace=~"rutas-norte-.*|monitoring"} == 0
          for: 5m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "Prometheus cannot scrape {{ $labels.job }}"
            description: >
              Target {{ $labels.instance }} unreachable. Check the
              NetworkPolicies, the /metrics endpoint and the pod's readiness.
            runbook_url: "https://runbooks.rutasnorte.example/target-down"

        - alert: HPAWithoutMetrics
          expr: |
            kube_horizontalpodautoscaler_status_condition{
              condition="ScalingActive", status="false"} == 1
          for: 5m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "HPA {{ $labels.horizontalpodautoscaler }} cannot scale"
            description: >
              Autoscaling is inactive, probably because
              metrics-server is not responding (07-02). During a traffic peak
              the platform will NOT scale and nobody will notice.
            runbook_url: "https://runbooks.rutasnorte.example/hpa-inactive"

This last alert is the preventive measure we promised in 07-02: the HPA's silent failure can no longer go unnoticed.

Applying and verifying:

kubectl apply -f k8s/base/monitoring/rutas-norte-alerts.yaml

# Has the operator loaded the rules?
kubectl -n monitoring get prometheusrule rutas-norte-alerts

# Is Prometheus evaluating them?
curl -s localhost:9090/api/v1/rules | jq -r '
  .data.groups[] | select(.name | startswith("rutasnorte")) |
  .rules[] | "\(.name)\t\(.state // "recording")"'

PlatformNotSelling	inactive
BookingsApiHighErrorRate	inactive
BookingsApiHighLatency	pending
WebStoreDown	inactive
PostgresDiskWillFill	inactive

Testing an alert before trusting it is essential. A safe way in rutas-norte-dev:

# Deliberately make a target go down
kubectl -n rutas-norte-dev scale deployment bookings-api --replicas=0
# Wait out the for and check that the alert moves to firing

Alertmanager: routes, receivers, inhibition and silences

Prometheus decides what is wrong. Alertmanager decides who is told, when and how.

The problem it solves

Imagine rutas-norte-worker-2 goes down at 03:14. Within seconds, Prometheus fires:

6 PodInCrashLoop alerts (the pods that lived there).
4 PrometheusTargetDown alerts.
1 WebStoreDown.
1 BookingsApiHighErrorRate.
1 NodeNotReady (from the rules the stack ships with).

Without Alertmanager, the on-call person receives thirteen notifications in two minutes and has to work out mentally that they are all the same problem. With Alertmanager configured properly they receive one, grouped, with the downed node highlighted and the knock-on alerts silenced.

The routing tree

# k8s/base/monitoring/alertmanager-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-monitoring-kube-pr-alertmanager
  namespace: monitoring
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
      smtp_smarthost: 'smtp.rutasnorte.example:587'
      smtp_from: '[email protected]'

    # -----------------------------------------------------------------
    # ROUTING TREE: evaluated top to bottom; the first match wins,
    # unless continue: true is set.
    # -----------------------------------------------------------------
    route:
      receiver: 'platform-team-slack'        # default receiver

      # Which alerts are grouped into a single notification.
      # Grouping by alertname + namespace means: "every
      # PodInCrashLoop in rutas-norte-pro arrives in one message".
      group_by: ['alertname', 'namespace', 'component']

      # After the FIRST alert of a new group, wait 30 s in case
      # more arrive and send them together. It is what turns 13 messages
      # into 1 when a node goes down.
      group_wait: 30s

      # If NEW alerts join an already notified group, wait 5 min
      # before sending the update.
      group_interval: 5m

      # If the alert is still active, repeat the warning every 4 hours.
      # Neither so short that it saturates nor so long that it is forgotten.
      repeat_interval: 4h

      routes:
        # 1. Test and development alerts: to a separate channel, without
        #    waking anybody. It stops dev noise reaching the on-call rota.
        - matchers:
            - namespace =~ "rutas-norte-(dev|pre)"
          receiver: 'development-channel'
          group_wait: 5m
          repeat_interval: 24h

        # 2. Critical production alerts: PagerDuty (it wakes somebody up)
        #    AND ALSO Slack, thanks to continue: true.
        - matchers:
            - severity = "critical"
            - namespace =~ "rutas-norte-pro|monitoring"
          receiver: 'oncall-pagerduty'
          group_wait: 10s          # critical ones, with less waiting
          repeat_interval: 1h      # and reminded more often
          continue: true

        - matchers:
            - severity = "critical"
          receiver: 'platform-team-slack'

        # 3. Data team alerts: to their own channel.
        - matchers:
            - team = "data"
          receiver: 'data-team-slack'
          repeat_interval: 12h

    # -----------------------------------------------------------------
    # INHIBITION: a serious alert silences the knock-on ones.
    # -----------------------------------------------------------------
    inhibit_rules:
      # If the web store is completely down, there is no need to warn
      # that its latency is high as well.
      - source_matchers:
          - alertname = "WebStoreDown"
        target_matchers:
          - severity = "warning"
          - component = "web-store"
        equal: ['namespace']

      # If the node is down, do not warn about every pod on it.
      - source_matchers:
          - alertname = "NodeNotReady"
        target_matchers:
          - alertname =~ "PodInCrashLoop|PrometheusTargetDown"
        equal: ['node']

      # General rule: if there is a critical alert for the same component,
      # that component's warnings go quiet.
      - source_matchers:
          - severity = "critical"
        target_matchers:
          - severity = "warning"
        equal: ['component', 'namespace']

    # -----------------------------------------------------------------
    # RECEIVERS
    # -----------------------------------------------------------------
    receivers:
      - name: 'platform-team-slack'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack/url
            channel: '#platform-alerts'
            send_resolved: true
            title: '{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ .CommonLabels.alertname }}'
            text: |
              {{ range .Alerts }}
              *{{ .Annotations.summary }}*
              {{ .Annotations.description }}
              Environment: `{{ .Labels.namespace }}` · Severity: `{{ .Labels.severity }}`
              <{{ .Annotations.runbook_url }}|📖 Runbook> · <{{ .Annotations.dashboard_url }}|📊 Dashboard>
              {{ end }}

      - name: 'oncall-pagerduty'
        pagerduty_configs:
          - routing_key_file: /etc/alertmanager/secrets/pagerduty/key
            description: '{{ .CommonAnnotations.summary }}'
            severity: 'critical'
            details:
              runbook: '{{ .CommonAnnotations.runbook_url }}'
              namespace: '{{ .CommonLabels.namespace }}'

      - name: 'development-channel'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack/url
            channel: '#dev-alerts'
            send_resolved: false

      - name: 'data-team-slack'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack/url
            channel: '#data-alerts'
            send_resolved: true

      - name: 'owners-email'
        email_configs:
          - to: '[email protected]'
            send_resolved: true

The four timings

Parameter	What it controls	Typical value	If too short	If too long
`group_wait`	Wait before a group's first notification	30 s (10 s critical)	Messages arrive one by one, not grouped	Detection is delayed
`group_interval`	Wait before notifying about new alerts in the group	5 m	Spam during evolving incidents	You find out late that it is getting worse
`repeat_interval`	How often an active alert is repeated	4 h (1 h critical)	Fatigue and mass muting	An open problem gets forgotten
`resolve_timeout`	How long to wait without data before calling it resolved	5 m	False resolutions	Phantom alerts

Inhibition

Inhibition is what turns thirteen messages into one. The syntax has three parts:

source_matchers: the alert that silences.
target_matchers: the alerts that get silenced.
equal: the labels that must match between the two. Without this, a WebStoreDown in dev would silence the warnings in pro.

The equal field is the one most often forgotten and the one that makes inhibition safe.

Silences during maintenance

A silence is a temporary suppression created by a person, usually before a planned intervention: expanding the PostgreSQL PVC, migrating a node, doing a large deployment.

From the Alertmanager interface (Silences → New Silence), or through the API:

kubectl -n monitoring port-forward svc/monitoring-kube-pr-alertmanager 9093:9093 &

# Silence every bookings-postgres alert for 2 hours
curl -s -X POST http://localhost:9093/api/v2/silences \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "component", "value": "bookings-postgres", "isRegex": false},
      {"name": "namespace", "value": "rutas-norte-pro", "isRegex": false}
    ],
    "startsAt": "2026-08-07T02:00:00Z",
    "endsAt":   "2026-08-07T04:00:00Z",
    "createdBy": "joan.costa",
    "comment": "Scheduled expansion of the bookings-postgres PVC (ticket OPS-1842)"
  }'

Good practice with silences:

Always with an end date. An indefinite silence is a deleted alert in practice, and everybody forgets about it.
Always with a comment and a ticket. In six weeks nobody will remember why it exists.
As specific as possible. Silencing the whole namespace=rutas-norte-pro during a database intervention leaves you blind to an unrelated problem.
Review active silences regularly. A forgotten silence has hidden more than one serious incident.

# List active silences
curl -s http://localhost:9093/api/v2/silences | \
  jq -r '.[] | select(.status.state=="active") |
  "\(.id)\t\(.comment)\tuntil \(.endsAt)"'

The judgement call: alert on symptoms, not on causes

Everything above is mechanics. This is judgement, and it determines whether the system is any use.

The alert fatigue problem

A team receiving forty notifications a day stops reading them within two weeks. When the one that really matters arrives, it is lost in the noise. A system with too many alerts is worse than one with none, because it creates a false sense of coverage.

Rule 1: alert on symptoms, not on causes

A symptom is something the customer perceives. A cause is something in the system that may or may not turn into a symptom.

Cause (bad alert)	Symptom (good alert)	Why
"Pod `bookings-api-x2klm` restarted"	"The `bookings-api` error rate is above 5 %"	With 6 replicas and readiness, one restart affects nobody
"Node CPU is at 85 %"	"The p95 latency is above 1 s"	A node at 85 % can be perfectly healthy
"There are 4 replicas instead of 6"	"The web store is down"	4 replicas may be plenty in the small hours
"`redis-cache` memory is at 70 %"	"The cache hit rate has dropped"	70 % may be the normal state

The definitive test: if this alert fires at three in the morning and nobody does anything, does something bad happen? If the answer is no, it should not wake anybody.

This does not mean causes are not monitored: they are visualised on the dashboard and consulted during diagnosis. They simply do not wake anybody.

A legitimate exception: predictive capacity alerts. PostgresDiskWillFill is a cause, not a symptom. It is justified because it warns hours in advance of a catastrophic and irreversible symptom that can still be avoided. That is the criterion for making exceptions: enough advance notice to act and serious consequences if you do not.

Rule 2: every alert must be actionable

Before creating an alert, answer three questions in writing:

What should whoever receives it do? If the answer is "see whether it fixes itself", it is not an alert: it is a panel.
Is that procedure written down? If not, write it before enabling the alert and put it in runbook_url.
Can the on-call person act, or does it always have to be escalated? If it always has to be escalated, route it directly to the team that can act.

Rule 3: review the alerts after every incident

After every incident, two questions:

Did any alert warn us? If not, one is missing. Create it.
Did alerts fire that contributed nothing? If so, they are surplus. Delete them or raise their threshold.

Without this periodic review, the catalogue only grows and ends up as noise.

Rutas Norte severity levels

Severity	Meaning	Channel	Example
`critical`	Affects customers now. Requires immediate action, day or night	PagerDuty + Slack	`PlatformNotSelling`
`warning`	It will have an effect if nothing is done. Handled during working hours	Slack	`PostgresConnectionsRunningOut`
`info`	Context. No action required	Dashboard only	A deployment has finished

A hard rule: if a critical alert does not justify a phone call at four in the morning, it is not critical.

Rutas Norte SLOs and error budget

SLOs (Service Level Objectives) formalise the question "what does it mean for the platform to work well?" with a number agreed with the business.

Defining the SLOs

An SLO has three parts: an indicator (SLI, what is measured), a target (what value it must reach) and a window (over what period it is evaluated).

The Rutas Norte SLOs, agreed with management:

Service	Indicator (SLI)	Target (SLO)	Window
`bookings-api` availability	% of requests with no 5xx error	99.5 %	30 days
`bookings-api` latency	% of requests under 500 ms	99.0 %	30 days
`web-store` availability	% of requests with no 5xx error	99.9 %	30 days
Confirmation emails	% delivered in under 5 min	99.0 %	30 days

An important nuance: 99.5 % is not 100 %, and that is deliberate. Chasing 100 % is infinitely expensive and stops any change. The SLO acknowledges that a percentage of failure is acceptable.

The error budget

The error budget is the SLO's complement: the failure you can afford.

bookings-api availability SLO: 99.5 % over 30 days
Error budget = 100 % − 99.5 % = 0.5 %

In time:      30 days × 24 h × 0.5 % = 3 h 36 min of downtime per month
In requests:  with ~4 million requests/month → 20,000 requests may fail

That budget is a decision-making tool, not a curiosity:

If budget remains, the team can deploy, experiment and take risks. The pace of change is not restricted.
If the budget is exhausted, changes other than reliability fixes are frozen until the next period. You stop adding features and fix what is failing.

It is an objective mechanism that replaces the eternal argument between "we have to ship the new feature" and "we have to stabilise the platform".

Measuring the budget in PromQL

- name: rutasnorte.slo
  interval: 60s
  rules:
    # Success ratio over the 30-day window
    - record: bookingsapi:slo_availability:30d
      expr: |
        1 - (
          sum(increase(rutasnorte_requests_total{environment="pro", code=~"5.."}[30d]))
          /
          sum(increase(rutasnorte_requests_total{environment="pro"}[30d]))
        )

    # Error budget CONSUMED, as a fraction of one.
    # 0 = untouched, 1 = exhausted, >1 = SLO breached.
    - record: bookingsapi:error_budget_consumed:30d
      expr: |
        (
          sum(increase(rutasnorte_requests_total{environment="pro", code=~"5.."}[30d]))
          /
          sum(increase(rutasnorte_requests_total{environment="pro"}[30d]))
        ) / 0.005

    # Latency SLO: fraction of requests under 500 ms
    - record: bookingsapi:slo_latency:30d
      expr: |
        sum(increase(rutasnorte_request_duration_seconds_bucket{environment="pro", le="0.5"}[30d]))
        /
        sum(increase(rutasnorte_request_duration_seconds_count{environment="pro"}[30d]))

Alerts on budget consumption

Instead of alerting on a fixed error threshold, you alert on the speed at which the budget is being consumed (the burn rate). It is smarter: it tolerates a short spike but quickly detects a haemorrhage.

- alert: ErrorBudgetBurningFast
  expr: |
    (
      sum(rate(rutasnorte_requests_total{environment="pro", code=~"5.."}[1h]))
      / sum(rate(rutasnorte_requests_total{environment="pro"}[1h]))
    ) > (14.4 * 0.005)
  for: 5m
  labels:
    severity: critical
    team: platform
  annotations:
    summary: "The error budget is being consumed 14 times faster than is sustainable"
    description: >
      At this rate, the 30-day budget will run out in about 2 days.
      Current error rate: {{ $value | humanizePercentage }}.
    runbook_url: "https://runbooks.rutasnorte.example/error-budget"

- alert: ErrorBudgetNearlyExhausted
  expr: bookingsapi:error_budget_consumed:30d > 0.90
  for: 30m
  labels:
    severity: warning
    team: platform
  annotations:
    summary: "Less than 10% of this month's error budget remains"
    description: >
      {{ $value | humanizePercentage }} of the budget consumed.
      Consider freezing deployments unrelated to reliability.

The factor 14.4 is not arbitrary: it is the rate at which the whole 30-day budget would be consumed in about 2 days. The usual factors are 14.4 (1 h, critical), 6 (6 h, critical) and 1 (3 d, warning), combined to detect both fast haemorrhages and slow bleeds.

A Stat panel with bookingsapi:error_budget_consumed:30d, as a percentage and with thresholds at 50/80/100, is probably the most useful panel on the dashboard for talking to the business.

Common Mistakes and Tips

1. Dashboards that do not persist. Built in the interface, with no ConfigMap and no persistent volume, they disappear when the pod is recreated. Always provision them as code.

2. Using rate(metric[5m]) instead of rate(metric[$__rate_interval]). With a fixed window, zooming out to 30 days fills the graph with gaps or noise.

3. A multi-value variable with = instead of =~. Grafana substitutes it with (a|b|c), which only works with the regular expression operator. The panel is left empty with no visible error.

4. Alerts with no for. The mistake that turns an alerting system into noise fastest. Every 30-second spike generates a notification.

5. Alerts with no runbook_url. At three in the morning, an alert name with no associated procedure forces you to improvise. Write the runbook before enabling the alert.

6. Alerting on causes instead of symptoms. "The pod restarted" is not a problem if there are six replicas and readiness. "Customers cannot buy" is.

7. Forgetting equal in the inhibition rules. Without it, an alert in dev can silence the warnings in pro. A silent and dangerous mistake.

8. Silences with no end date. That is deleting an alert and forgetting it. Review the active silences regularly.

9. Thresholds copied from another system. A p95 of 1 s can be excellent for a report and catastrophic for an autocomplete. Thresholds come from your data and your SLO.

10. Too many critical alerts. If everything is critical, nothing is. Reserve critical for what justifies a call in the middle of the night.

11. Not testing the alerts. An alert with a badly written query never fires and provides false peace of mind. Trigger the condition deliberately in rutas-norte-dev and verify that the notification reaches the right channel.

12. The Grafana password in Git. The adminPassword value in the Helm file is versioned plain text. Use an external Secret or delegated authentication.

Exercises

Exercise 1 — Fix a faulty alert catalogue

A colleague has written these alerts. Identify the problems with each one and rewrite the corrected PrometheusRule.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: team-alerts
  namespace: monitoring
spec:
  groups:
    - name: alerts
      rules:
        - alert: HighCPU
          expr: |
            sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="rutas-norte-pro"}[5m])) > 0.5
          labels:
            severity: critical
          annotations:
            summary: "High CPU"

        - alert: PodRestarted
          expr: kube_pod_container_status_restarts_total{namespace="rutas-norte-pro"} > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "A pod has restarted"

        - alert: HighMemory
          expr: container_memory_working_set_bytes{namespace="rutas-norte-pro"} > 400000000
          for: 30s
          labels:
            severity: critical
          annotations:
            summary: "High memory in {{ $labels.pod }}"

Exercise 2 — Design the predictive alert for `redis-cache`

redis-cache has maxmemory set to 1 GiB with the allkeys-lru policy. When it fills up, it starts evicting keys and the hit rate collapses, which makes bookings-api query PostgreSQL far more and slows everything down.

The Redis exporter exposes:

redis_memory_used_bytes
redis_memory_max_bytes
redis_keyspace_hits_total
redis_keyspace_misses_total
redis_evicted_keys_total

Write the PromQL query for the cache hit rate (the proportion of hits over the total).
Write an alert that warns before the problem affects customers, using predict_linear.
Write a second alert on the symptom, for when it is already happening, and explain why both are needed.
What inhibition rules would you add between them?

Exercise 3 — Configure routing for a night-time incident

Rutas Norte defines this on-call policy:

From 08:00 to 20:00 on weekdays: every alert goes to the #platform-alerts Slack channel.
Outside those hours: only critical alerts from rutas-norte-pro go to PagerDuty; the rest waits until the following morning.
Data team alerts (team: data) never go to PagerDuty.
On the Saturday of the May bank-holiday weekend there is a scheduled bookings-postgres migration from 02:00 to 05:00.

Can Alertmanager route by time of day? If so, write the configuration.
Write the complete routing tree that implements the policy.
Write the command that creates the silence for the scheduled migration, with the good practice from section 9.

Solutions

Solution 1

Problems with HighCPU:

No for: it fires as soon as a pod goes above 0.5 cores for an instant. A JVM start-up or a one-off compaction generates a notification.
It alerts on a cause, not a symptom. 0.5 cores can be perfectly normal for bookings-postgres. Nobody knows what to do when they receive it.
An absolute threshold with no context. 0.5 cores means different things depending on each component's limits.
Unjustified critical severity: it wakes nobody who can do anything useful.
No runbook_url and no description.

Problems with PodRestarted:

> 0 on a cumulative counter: kube_pod_container_status_restarts_total never goes back down. A pod that restarted three weeks ago keeps the alert firing forever. You have to use increase(...[window]).
An isolated restart is not a problem with six replicas and readiness (07-01).
for: 1m is far too short and the severity is overblown.

Problems with HighMemory:

for: 30s: extremely noisy.
A raw, absolute threshold in bytes: 400 MB is a lot for web-store and next to nothing for bookings-postgres. It has to be compared with the container's limits.
No container!="" filter: it includes the pod's aggregate series and the pause container, duplicating the alerts.
It alerts on a cause. What matters is whether the container is about to be OOMKilled, not a number of bytes.

The corrected version:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: team-alerts
  namespace: monitoring
  labels:
    app.kubernetes.io/part-of: rutas-norte
spec:
  groups:
    - name: rutasnorte.resources
      interval: 60s
      rules:

        # Replaces HighCPU: it alerts on the real effect (throttling),
        # relative to the container's own limit, not an absolute figure.
        - alert: ContainerThrottled
          expr: |
            sum by (namespace, pod, container) (
              rate(container_cpu_cfs_throttled_periods_total{namespace="rutas-norte-pro"}[5m]))
            / sum by (namespace, pod, container) (
              rate(container_cpu_cfs_periods_total{namespace="rutas-norte-pro"}[5m]))
            > 0.30
          for: 15m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "{{ $labels.container }} throttled {{ $value | humanizePercentage }} of the time"
            description: >
              Container {{ $labels.container }} of pod {{ $labels.pod }}
              is hitting its limits.cpu in a sustained way and that degrades latency.
              Review the resource recalibration from 07-02.
            runbook_url: "https://runbooks.rutasnorte.example/cpu-throttling"

        # Replaces PodRestarted: it uses increase() over a window,
        # requires several restarts and lowers the severity.
        - alert: PodInCrashLoop
          expr: |
            increase(kube_pod_container_status_restarts_total{
              namespace="rutas-norte-pro"}[15m]) > 3
          for: 10m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "{{ $labels.pod }} restarted more than 3 times in 15 minutes"
            description: >
              Container {{ $labels.container }}. Frequent causes: OOMKilled,
              a configuration failure or a livenessProbe that is too aggressive (07-01).
              Collect 'kubectl logs --previous' BEFORE touching anything.
            runbook_url: "https://runbooks.rutasnorte.example/crashloop"

        # Replaces HighMemory: relative to the limit, with a sensible for
        # and filtering out the pod's aggregate series.
        - alert: MemoryNearLimit
          expr: |
            container_memory_working_set_bytes{namespace="rutas-norte-pro", container!=""}
            / on (namespace, pod, container)
            kube_pod_container_resource_limits{namespace="rutas-norte-pro", resource="memory"}
            > 0.90
          for: 15m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "{{ $labels.container }} at {{ $value | humanizePercentage }} of its memory limit"
            description: >
              The container is close to its limits.memory and will be OOMKilled
              if it exceeds it. Check whether there is a leak or the limit is too low.
            runbook_url: "https://runbooks.rutasnorte.example/memory-limit"

        # An alert on the real SYMPTOM: the container has ALREADY been killed.
        - alert: ContainerOOMKilled
          expr: |
            increase(kube_pod_container_status_last_terminated_reason{
              namespace="rutas-norte-pro", reason="OOMKilled"}[15m]) > 0
          for: 1m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "{{ $labels.container }} has been killed for lack of memory"
            description: >
              The kernel killed the container for exceeding its limits.memory
              (exit code 137). Raise the limit or fix the leak.
            runbook_url: "https://runbooks.rutasnorte.example/oomkilled"

The substantive changes, beyond syntax: we have gone from three alerts on absolute causes to alerts relative to the container's own limit, with sensible windows, proportionate severities and runbooks. The only critical one is the one that reflects an accomplished and actionable fact.

Solution 2

1. Cache hit rate.

sum(rate(redis_keyspace_hits_total{environment="pro"}[5m]))
  /
(
  sum(rate(redis_keyspace_hits_total{environment="pro"}[5m]))
  +
  sum(rate(redis_keyspace_misses_total{environment="pro"}[5m]))
)

Rates are used rather than absolute values because counters accumulated since the pod started dilute any recent degradation.

2. The predictive alert (the cause, in advance).

- alert: RedisCacheWillFill
  expr: |
    predict_linear(
      (redis_memory_max_bytes{environment="pro"} - redis_memory_used_bytes{environment="pro"})[2h:],
      2 * 3600
    ) < 0
  for: 20m
  labels:
    severity: warning
    component: redis-cache
    team: platform
  annotations:
    summary: "redis-cache will fill up in under 2 hours"
    description: >
      Based on the trend of the last 2 hours, redis-cache will reach its
      1 GiB maxmemory in less than 2 hours and will start evicting keys.
      Current free memory: {{ $value | humanize1024 }}B.
      Action: raise maxmemory or review the TTL of the availability keys.
    runbook_url: "https://runbooks.rutasnorte.example/redis-memory"

A note on the syntax: predict_linear needs a range vector. When applying it to the subtraction of two gauges you have to use a subquery ([2h:]), because the resulting expression is not a simple series. It is a detail that confuses a lot of people.

A simpler alternative, over a single metric:

predict_linear(redis_memory_used_bytes{environment="pro"}[2h], 2*3600)
  > avg(redis_memory_max_bytes{environment="pro"})

3. The alert on the symptom.

- alert: RedisHitRateDegraded
  expr: |
    (
      sum(rate(redis_keyspace_hits_total{environment="pro"}[10m]))
      / (sum(rate(redis_keyspace_hits_total{environment="pro"}[10m]))
         + sum(rate(redis_keyspace_misses_total{environment="pro"}[10m])))
    ) < 0.80
  for: 15m
  labels:
    severity: warning
    component: redis-cache
    team: platform
  annotations:
    summary: "The redis-cache hit rate has dropped to {{ $value | humanizePercentage }}"
    description: >
      95% is normal. With a low rate, bookings-api queries
      bookings-postgres far more than expected and latency rises.
      Check whether redis-cache is evicting keys for lack of memory.
    runbook_url: "https://runbooks.rutasnorte.example/redis-hit-rate"

- alert: RedisEvictingKeys
  expr: sum(rate(redis_evicted_keys_total{environment="pro"}[5m])) > 10
  for: 10m
  labels:
    severity: warning
    component: redis-cache
    team: platform
  annotations:
    summary: "redis-cache is evicting {{ $value | humanize }} keys/s for lack of memory"
    runbook_url: "https://runbooks.rutasnorte.example/redis-memory"

Why both are needed. They serve different purposes in time:

The predictive one warns two hours ahead, while there is still time to act calmly and with no customer impact. It is a capacity alert.
The symptom one covers the case where the prediction fails: an abrupt change of pattern (for example, a deployment that caches much larger objects) can fill Redis in minutes without any trend anticipating it. Linear regression only predicts well what behaves linearly.

Relying only on the predictive one is assuming the future resembles the past. Relying only on the symptom is giving up on prevention.

4. Inhibition rules.

inhibit_rules:
  # If Redis is already evicting keys, predicting that it will fill up
  # adds nothing: the event has happened.
  - source_matchers:
      - alertname = "RedisEvictingKeys"
    target_matchers:
      - alertname = "RedisCacheWillFill"
    equal: ['component', 'namespace']

  # If bookings-api latency is through the roof (a symptom the customer
  # suffers), the redis-cache warnings are the cause: there is no need for two
  # separate notifications about the same incident.
  - source_matchers:
      - alertname = "BookingsApiHighLatency"
      - severity = "critical"
    target_matchers:
      - component = "redis-cache"
      - severity = "warning"
    equal: ['namespace']

The first rule is a perfect example of the inhibition principle: when the prediction comes true, the prediction is redundant.

Solution 3

1. Can Alertmanager route by time of day?

Yes. Since Alertmanager 0.22 there are time intervals (time_intervals), referenced in routes with active_time_intervals (the route only applies within the interval) or mute_time_intervals (the route is muted within the interval).

time_intervals:
  - name: working-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '08:00'
            end_time: '20:00'
        location: 'Europe/Madrid'

  - name: out-of-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '20:00'
            end_time: '24:00'
          - start_time: '00:00'
            end_time: '08:00'
        location: 'Europe/Madrid'
      - weekdays: ['saturday', 'sunday']
        location: 'Europe/Madrid'

The location field is essential: without it, Alertmanager uses UTC and in summer the on-call window would start two hours earlier than intended.

2. The complete routing tree.

route:
  receiver: 'platform-team-slack'
  group_by: ['alertname', 'namespace', 'component']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # ------------------------------------------------------------------
    # 1. Data team: ALWAYS to their channel, NEVER to PagerDuty.
    #    It goes first because the first match wins.
    # ------------------------------------------------------------------
    - matchers:
        - team = "data"
      receiver: 'data-team-slack'
      repeat_interval: 12h

    # ------------------------------------------------------------------
    # 2. dev and pre: they never wake anybody.
    # ------------------------------------------------------------------
    - matchers:
        - namespace =~ "rutas-norte-(dev|pre)"
      receiver: 'development-channel'
      group_wait: 5m
      repeat_interval: 24h

    # ------------------------------------------------------------------
    # 3. OUT OF HOURS: only production criticals wake anybody.
    # ------------------------------------------------------------------
    - matchers:
        - severity = "critical"
        - namespace =~ "rutas-norte-pro|monitoring"
      active_time_intervals: ['out-of-hours']
      receiver: 'oncall-pagerduty'
      group_wait: 10s
      repeat_interval: 1h
      continue: true          # so that Slack keeps a record too

    # 3b. The rest, out of hours, to Slack only: read in the morning.
    - matchers:
        - namespace =~ "rutas-norte-pro|monitoring"
      active_time_intervals: ['out-of-hours']
      receiver: 'platform-team-slack'
      group_wait: 5m
      repeat_interval: 12h     # without nagging in the small hours

    # ------------------------------------------------------------------
    # 4. WORKING HOURS: everything to Slack, with criticals moving faster.
    # ------------------------------------------------------------------
    - matchers:
        - severity = "critical"
      active_time_intervals: ['working-hours']
      receiver: 'platform-team-slack'
      group_wait: 10s
      repeat_interval: 1h

    - active_time_intervals: ['working-hours']
      receiver: 'platform-team-slack'

Points worth highlighting about the design:

Order matters: the data team route goes first so that their critical alerts do not end up in PagerDuty via route 3.
continue: true on route 3 lets the alert reach both places: PagerDuty wakes somebody and Slack keeps a record for the following day's review.
Non-critical alerts out of hours have repeat_interval: 12h so as not to fill the channel in the small hours.

A robustness consideration: this time-based routing assumes the on-call person is always the same. In teams with a rota, the usual approach is to delegate the shift logic to the on-call tool itself (PagerDuty, Opsgenie), which handles calendars, escalations and cover far better than Alertmanager. Here the time rules are mainly there to decide what deserves to wake somebody, not who.

3. The silence for the scheduled migration.

kubectl -n monitoring port-forward svc/monitoring-kube-pr-alertmanager 9093:9093 &

curl -s -X POST http://localhost:9093/api/v2/silences \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "component", "value": "bookings-postgres", "isRegex": false},
      {"name": "namespace", "value": "rutas-norte-pro",   "isRegex": false}
    ],
    "startsAt": "2026-05-02T00:00:00Z",
    "endsAt":   "2026-05-02T03:15:00Z",
    "createdBy": "[email protected]",
    "comment": "Scheduled bookings-postgres migration 02:00-05:00 CEST. Ticket OPS-1842. Approved at the change committee on 28/04. On-call owner: Marta Ruiz."
  }' | jq -r '.silenceID'

The good practice applied and why:

Times in UTC. 02:00-05:00 CEST is 00:00-03:00 UTC. It is the commonest mistake when creating silences and it leaves alerts unsilenced during exactly the window in question.
15 minutes of margin over the planned window (until 03:15 UTC), because migrations run long.
Specific matchers: only bookings-postgres in pro. A silence covering the whole namespace would hide an unrelated problem in web-store for three hours.
A comment with ticket, approval and owner: in six weeks, anybody can reconstruct why it existed.
endsAt is mandatory, never indefinite.

And what should not be silenced during the migration:

# Check WHICH alerts the silence covers before applying it:
# the platform-wide ones (PlatformNotSelling, WebStoreDown) do NOT
# carry the component=bookings-postgres label, so they stay active.
# That is deliberate: if the migration leaves the platform unable to sell,
# we need to know immediately even though the migration is the cause.

curl -s http://localhost:9093/api/v2/alerts | \
  jq -r '.[] | select(.status.silencedBy | length > 0) | .labels.alertname'

Verification after the window:

# Confirm the silence has expired and none has been left forgotten
curl -s http://localhost:9093/api/v2/silences | \
  jq -r '.[] | select(.status.state=="active") |
  "\(.id)\t\(.createdBy)\t\(.comment)\tuntil \(.endsAt)"'

This last command should be run as part of the team's weekly review: a forgotten silence is an alert that no longer exists.

Conclusion

Rutas Norte no longer just remembers: now it shows and warns. In this lesson we have:

Connected Grafana to Prometheus and learned the anatomy of a panel: the query with $__rate_interval, the legend with label templates, the units and thresholds that make a number legible, and the four types that actually get used, including the heatmap for seeing the complete distribution of latencies.
Used dashboard variables to have a single dashboard that serves dev, pre and pro, avoiding three copies that drift apart.
Built the Rutas Norte dashboard panel by panel: an executive summary row headed by bookings confirmed per minute — because if that drops to zero it makes no difference that every pod is Running — and one row per component with the four golden signals.
Provisioned the dashboards as code in ConfigMaps with the grafana_dashboard label, so that they survive the pod being recreated and are reviewed in a pull request.
Written the alert catalogue with PrometheusRule, understanding that for is the field that separates a usable system from noise, that predict_linear lets you warn that the PostgreSQL disk will fill up before it fills up, and that runbook_url is not optional.
Configured Alertmanager: the routing tree with its four timings, the receivers, the inhibition that turns thirteen notifications into one when a node goes down, and the silences with an end date and a ticket for scheduled maintenance.
And above the mechanics, the judgement: alert on symptoms the customer perceives, not on causes; make every alert actionable; and use the SLOs and the error budget as an objective tool for deciding when to stop adding features and start stabilising.

But there is a question that neither the metrics nor the dashboards can answer. When BookingsApiHighErrorRate fires at 03:14, we know that 7 % of requests are failing, we know on which route and with what latency. What we do not know is why. The specific exception, the database's message, the stack trace pointing to the line of code: that does not live in a metric. It lives in the logs, which today are scattered across the nodes, are lost when a pod is recreated and cannot be searched.

In 07-05 we will set up the complete centralized logging stack we announced in 06-02 when we deployed the collector as a DaemonSet: Elasticsearch, Fluentd and Kibana. We will see how to go from kubectl logs to a search that correlates every component, why structured JSON logs are the single decision that improves the whole system most, and — very important for a platform that stores its customers' ID numbers and phone numbers — what must never end up written in a log.

Visualization and Alerting with Grafana and Alertmanager

Contents