In Module 5 you learned to measure. The USE method, vmstat, iostat, free, sar and pidstat tell you how much: the CPU is at 80%, the disk has 22 ms of write latency, 300 MB of memory is still available. With that you solved the incident of the website being slow in the mornings, because the counters pointed at I/O and the I/O had an identifiable culprit.

But there is a class of problem the counters are no use for: when they are all green and the service is performing badly anyway. The CPU idle, the disk quiet, memory plentiful, and the application taking ten times longer than it should. At that point the question stops being how much and becomes what exactly is this process doing, and answering it needs tools that do not measure aggregates but observe individual events.

This lesson gives you three of them, in increasing order of sophistication and decreasing order of cost: strace to see the system calls one by one, perf to profile where the cycles really go, and eBPF to instrument the kernel in production with no appreciable penalty. And you are going to run head-on into a consequence of your own work from Module 6.

1. When counters are not enough
2. strace: the system calls one by one
3. The obstacle you put there yourself: ptrace_scope
4. ltrace and the library level
5. perf: sampling-based profiling
6. Flame graphs
7. eBPF: instrumenting the kernel in production
8. The bpfcc tools and the question each one answers
9. A decision table: which tool for which question
10. The Tramontana case: from 400 ms to 40 ms

The difference between the two families of tools is the unit of observation:

That last point is important and it orders the whole lesson: tracing tools do not replace the counters, they continue from them. You use them once you already have a hypothesis you want to confirm or refute. Firing strace at a process without knowing what you are looking for produces a hundred thousand unreadable lines and slows the service down.

The methodology, then, has three steps and the order is not negotiable:

1. A measured symptom. "The application takes 400 ms to respond; the baseline says 40 ms." Not "it is slow".
2. Hypotheses ruled out with counters. CPU? Disk? Memory? Network? It is free and it eliminates most cases.
3. A targeted measurement with the right tool. One specific hypothesis, one tool, one answer.

Remember the layers from Module 1: hardware → kernel → system calls → shell and libraries → applications. A system call is the only way a program has of asking the kernel for something: open a file, read from a socket, allocate memory, wait. Everything a process does towards the outside world goes through there.

strace intercepts those calls and prints them. It is literally watching the conversation between the program and the kernel.

Every line has the same structure: the call's name, the arguments in brackets, and the returned value after the =. If the value is negative, the error's symbolic name (ENOENT, EACCES, EAGAIN) and its description appear. That third part is usually where the answer is.

Look at the /etc/ld.so.preload line: it returns ENOENT. That is not an error, it is normal — that file rarely exists — and it illustrates something you have to internalise: strace shows an enormous number of expected errors. Knowing which ones are normal is half the craft.

The -e trace= groups save you memorising names:

This is the first step of any diagnosis with strace, and for a practical reason: it fits on one screen and it immediately says where the time goes. Here 89% of the time is in connect, with 300 calls at 8 ms each. That is an enormous clue: the process is constantly opening new connections and each one costs 8 milliseconds.

The columns: % time is the proportion of the total time inside system calls (not of the process's total time: if the program burns CPU in its own code, it does not show up here); usecs/call is the average per call; errors counts the returns that failed.

The most profitable use of strace, and the one to have memorised. A program fails saying it cannot find a file, and you are certain the file exists:

The file is there. The question is what the program is really looking for:

There it is, and it was not what it looked like. The configuration file opens fine; the .local that does not exist is optional. The real error is the EACCES on the credential file: the program looks for it in /etc/tramontana/secrets/db_password, but since 06-05 the credential is delivered by systemd in $CREDENTIALS_DIRECTORY, and when run by hand that variable does not exist. The program works correctly under systemd and fails when run directly.

That is the general pattern, and it is why it is worth memorising:

strace works through ptrace, which stops the process at every system call, transfers control to the tracer, and resumes it. That means two context switches per call.

A process making 50,000 calls a second can become a hundred times slower. The operational consequences:

• Never run strace without -e trace= or -c on a production process under load. If you have to, do it with a time limit: timeout 5 strace -f -c -p PID.
• A process you are attached to whose strace you kill with SIGKILL can be left stopped. Always exit with Ctrl+C, which does a clean detach.
• For production, perf trace and eBPF do the same thing at a cost one or two orders of magnitude lower. That is the reason they exist.

Try attaching to the application's process as your own user:

It is not a fault. It is the kernel.yama.ptrace_scope = 1 you set in /etc/sysctl.d/60-hardening.conf in 06-06, which that day came with a warning written beside it for precisely this reason:

The four possible values:

And why the measure is correct, which is the part to understand rather than merely work around: with ptrace_scope = 0, a compromised process can read all the memory of any other process belonging to the same user. That includes the database credential you took such trouble to encrypt in 06-05: it is in the clear in the memory of the process using it. An attacker who managed to run code as svc-tramontana in any process at all could extract it from the application's process without touching a single file. The value 1 closes exactly that route.

The two legitimate ways out:

For way out B, the disciplined form is a script with a trap, applying what you saw in 04-06, because a Ctrl+C halfway through would leave the system with the protection turned off:

In practice, way out A — sudo — is the right one 95% of the time. Way out B only makes sense with tools that do not work well under sudo, and on a production server the honest answer is that you do not lower ptrace_scope: you use eBPF, which does not need ptrace at all. That is another argument in favour of the third section of this lesson.

strace sees the boundary between the program and the kernel. There is another boundary one level up: between the program and the shared libraries it uses.

It is useful for understanding the behaviour of a program of your own — memory leaks, use of a cryptographic library, calls into libcurl — but it has two serious limits: it only sees calls into dynamic libraries (a static binary is opaque), and its cost is even higher than strace's. In practice it gets little use, and on a server almost none. strace for the boundary with the kernel, perf for the inside of the process.

perf is the Linux kernel's own performance tool, and it operates on a radically different model:

The key difference is the penultimate row. strace cannot tell you anything about a process burning CPU in its own loop, because there are no system calls there. perf can.

On a VM, some hardware counters are unavailable because the hypervisor does not expose them. The software events (task-clock, context-switches, page-faults) always work.

How to read this, line by line, because each one answers a different question:

And the conclusion from this particular measurement: 0.128 CPUs utilized. The process is not working, it is waiting. The 3,412 context switches in 10 seconds confirm it: it goes on and off the CPU constantly because it blocks. This rules out the CPU as the cause of the latency problem, which was the point of step 2 of the methodology.

The IPC of 0.61 and the 39% cache misses are mediocre, but irrelevant here: with the process at 12.8% of one core, improving its CPU efficiency would not change the latency.

-F 99 fixes the sampling frequency at 99 Hz. It is a convention with a reason: using 100 Hz risks synchronising with periodic system events that also run at 100 Hz, and skewing the sample. A nearby prime number avoids that.

The [k] and [.] markers distinguish kernel space from user space. Here 82% of the CPU time is in connect and tcp_v4_connect, both in the kernel, and tramontana_db_connect appears in user space. Three different tools pointing at the same place: the process spends its life opening connections.

A perf report with call stacks is hard to read because the information is hierarchical and the output is flat. The flame graph solves that visually, and it has become the standard in the field.

How to read one, which is the thing to learn:

• The horizontal axis is NOT time. It is the alphabetical grouping of the stacks. A block's width is the proportion of samples in which that function was on the stack.
• The vertical axis is the stack depth. At the bottom the entry point, at the top the function that was executing at that instant.
• What you are looking for are wide plateaux, especially near the top: a wide function at the top is a function where the process spends a lot of time actually executing. A wide function at the bottom with lots of thin towers on top of it is only a place things pass through.

The result is an interactive SVG: you open it in a browser, and you can click to zoom into a branch and search by function name.

There is a very useful variant almost nobody knows about: the off-CPU flame graph. The normal one shows where CPU is consumed; the off-CPU one shows where the process is blocked waiting. For a latency problem like ours — a process at 12% CPU — the second is far more informative, and it is built with eBPF rather than with perf:

And perf trace, which deserves a mention because it solves strace's cost problem:

The same information as strace -T, at a much lower cost because it uses the kernel's event infrastructure instead of ptrace — and, incidentally, ptrace_scope does not affect it.

eBPF is the most important change in Linux observability of the last decade. The idea: allow programs of your own to be loaded inside the kernel, which run when an event occurs, with three guarantees that make it safe:

1. A verifier analyses the program before loading it and rejects anything that could hang the kernel: unbounded loops, arbitrary memory accesses, disallowed calls.
2. It is compiled to native code with a JIT, so it runs at kernel speed.
3. It cannot block or modify the kernel's flow; only observe and aggregate.

Why that changes everything: previously, to know the latency of each disk operation you had to choose between an aggregate counter (iostat, which gives the average and hides the tail) or tracing everything (strace, prohibitive). With eBPF you can compute the complete histogram inside the kernel and export only the result. The cost is fractions of a percentage point.

And here the second sysctl from 06-06 shows its consequences:

This stops an unprivileged user loading eBPF programs, and it is correct: the BPF subsystem has had privilege escalation vulnerabilities, and its surface is large. The practical consequence is that every tool in this section is run with sudo, which is exactly what you want on a server.

bpftrace is a one-line language for eBPF, with syntax inspired by awk — which you already know from Module 3: event { action }.

The histogram in example 2 is the kind of information no other tool gives you easily: 300 calls to connect, with the bulk between 4 and 16 microseconds... hold on. That histogram says microseconds, and strace -c said 8 milliseconds per call. The discrepancy is the clue: the connect call into the kernel is fast; the time goes somewhere else in the connection process. We will come back to this in the case study.

Note the mechanics of example 2, because it is the pattern for measuring latency with eBPF: store nsecs in a map indexed by tid on entry, subtract on exit, and aggregate with hist(). The entire calculation happens inside the kernel; the only thing that leaves for user space is the final histogram.

bpfcc-tools installs about a hundred and fifty ready-written tools. These are the ones that really get used:

Two of them deserve a comment for what they add over the classic counters:

Those three events at 256-511 ms appear in no average at all. If they coincide with the slow requests, they are the cause.

The table that summarises the lesson, and the one to come back to when you have a problem in front of you:

And the rule of order: counters → perf stat → eBPF → strace. From lowest to highest cost, with strace last precisely because it is the most expensive. The opposite intuition — starting with strace because it is the best known — is what produces diagnoses that degrade the very service they are trying to fix.

A note on dumps: if the problem is a crash rather than slowness, the tool is a different one. Remember that in 06-06 you set * hard core 0 in limits.conf so that dumps would not expose secrets; to debug a crash you would have to reverse that temporarily, and systemd's coredumpctl is the modern route.

health_check.sh starts returning 1. The 05-07 baseline says the application responds in 40 ms; now it takes 400.

Confirmed and reproducible: ~400 ms, not an isolated spike.

The CPU 95% idle, the disk at 0.4% utilisation, 2.4 GB of memory available. No resource is saturated. This is exactly the scenario the close of 07-01 announced: the counters green and the service performing badly.

12.8% of one core and 3,412 context switches. The process is blocking constantly. The hypothesis becomes: it is waiting for something external. The candidates are disk (ruled out by iostat) and network — that is, PostgreSQL.

With sudo, because of ptrace_scope, and with a time limit because of the cost:

300 calls to connect in 10 seconds, at 8 ms each. With about 75 requests in that interval, that works out at roughly 4 new connections per request. An application with a connection pool should not be opening any.

This is where the discrepancy we left hanging gets resolved:

Three hundred connections to PostgreSQL, each with 8 ms of life and a few KB. They open, run one query and close. That is a connection pool that is not working.

And the histogram explains the 8 ms that the connect call alone did not account for:

The system call takes microseconds. The 8 milliseconds strace was seeing include everything around the connection: the TCP handshake, and above all PostgreSQL's authentication and session startup. perf report was already hinting at it with SSL_connect in the stack.

18.4 of 20 seconds blocked inside db_connect. Confirmed from a fourth independent tool.

The missing piece is a fact you already had. Remember the incident from 05-07: errors.log with active_connections=200 and db_timeout. The I/O saturation was resolved then, but the max_connections=200 stayed where it was:

There is the root cause, and it is configuration, not code: the application believes it can have 200 connections, PostgreSQL only accepts 100. When the pool tries to grow beyond 100, the new connections are refused, the application gives up on the pool and opens direct connections per request, and each one costs 8 ms of setting up and authenticating.

From 400 ms to 40 ms. From 300 connections every 10 seconds to 80 in total, which is the size of the pool establishing itself once. And health_check.sh is back to 0.

What makes this diagnosis valid is not any tool in particular, but the convergence of five independent measurements — perf stat, strace -c, tcplife, bpftrace and offcputime — all pointing at the same place, plus a final hypothesis verified by a before-and-after comparison. And the note for the runbook: changing a limit on one side of a client-server relationship without checking the other side is the mistake that caused this, and it happened three modules ago.

• Starting with strace. It is the best-known tool and the most expensive. The order is counters → perf stat → eBPF → strace.
• strace without -e trace= or -c in production. A hundred thousand unreadable lines and a service a hundred times slower. If you have to trace in production, timeout 5 strace -f -c -p PID.
• Forgetting -f. Without following children and threads, you lose almost everything in a multithreaded application.
• Confusing strace -c's time with the process's total time. % time is the proportion inside system calls. If the program burns CPU in its own code, it does not show up there: that is what perf is for.
• Killing strace with SIGKILL. It can leave the traced process stopped. Exit with Ctrl+C.
• Reading every ENOENT as an error. A normal startup generates dozens of them looking for libraries and locales. Filter out the noise before drawing conclusions.
• Lowering ptrace_scope and forgetting to restore it. It leaves memory readable across processes belonging to the same user, and with it the secrets. Use sudo, or a script with a trap.
• Trusting the average latency. iostat can report a 1 ms average while 1% of the operations take 500 ms. biolatency-bpfcc shows the distribution, and the tail is what the user notices.
• Sampling at 100 Hz. It can synchronise with periodic system events and skew the sample. Use a nearby prime: -F 99.
• Looking for a latency problem in a normal flame graph. The CPU one shows where time is consumed; for a process that is waiting you need the off-CPU one (offcputime-bpfcc).
• Concluding from a single tool. This case was solved with five converging measurements. One alone would have given a plausible and probably incomplete answer.
• A tip on method. Store the diagnostic measurements alongside the 05-07 baseline: what you measured, with what command, and the normal value. Next time, step 2 is thirty seconds instead of twenty minutes.

backup_tramontana.sh used to take 40 minutes and now takes 3 hours, which means the 02:30 backup does not finish before the morning peak — the 05-07 incident all over again. iostat shows the disk at 45% utilisation, a long way from saturation. Design the diagnostic procedure, stating which tool you would use at each step and why, knowing that the backup uses restic over a LUKS-encrypted volume.

Write a latency_diagnostics.sh script that, given the name of a systemd service, automatically runs steps 2, 3 and 4 of the methodology — counters, perf stat and strace -c — and produces a readable report. It must respect the course's conventions, handle ptrace_scope safely, and leave no residue behind.

A colleague proposes setting kernel.yama.ptrace_scope = 0 permanently "so we can diagnose without sudo". Write the technical reply: what is gained, what is lost exactly, and what alternative you propose.

Solution 1

The key to the question is "45% utilisation", which rules out saturation but does not rule out latency: they are two different things and confusing them is the usual mistake. A disk at 45% can have a queue of slow requests.

With a w_await of 38 ms and a %util of 45%, there is already an anomaly: there is latency without saturation, which points to individually slow operations or to a bottleneck somewhere in the stack.

And here is the reasoning specific to this exercise: /srv/tramontana/backups is a LUKS volume on LVM on sda. That is three layers, and the latency has to be attributed to one of them:

The interpretation, which is what is being asked for:

• CPUs utilized close to 1.00 and profile-bpfcc showing AES functions: the bottleneck is software encryption. Possible causes: the VM does not expose AES-NI to the guest, or the CPU does not have it. You check with /proc/cpuinfo and resolve it by enabling host-passthrough in the VM's configuration (you will see this in 07-04) or by changing the LUKS cipher.
• Low CPUs utilized and offcputime showing waits on I/O: the bottleneck is the disk. Then biolatency -D says whether the latency already comes from sda, and the next step is ext4slower-bpfcc to see which specific operations are slow.
• Very high context switches with both of the above low: contention between restic and another process. runqlat-bpfcc would confirm it.

And two alternative hypotheses to rule out before touching anything, because they are more likely than a hardware problem:

A restic repository with no forget --prune accumulates data and slows every operation down. If the GFS 14/8/12 retention stopped running, that is the simplest explanation — and checking it is free. Before you diagnose the hardware, rule out the boring stuff.

A final note on method: the ionice set in 05-07 still applies and it is correct that the process should yield I/O. But if the backup no longer fits in its window, ionice stops being enough and you have to attack the cause, not the symptom.

Solution 2

The design decisions that make this script usable in production rather than a hazard:

1. MainPID instead of pgrep. systemctl show -p MainPID gives the process systemd considers the main one. pgrep -f tramontana could pick up the script itself, a grep, or a process from another release.
2. The order of the steps is one of increasing cost, and strace goes last. If the problem is visible in step 2, you have the answer before paying anything.
3. sudo for strace, never touch ptrace_scope. It is way out A from the lesson, and it is the only acceptable one in a script anybody might run.
4. A compulsory timeout on strace, with || true because timeout's exit code 124 is the expected result, not a failure. Without it, set -e would abort the script just before the conclusions get written.
5. Graceful degradation. If perf or eBPF are absent, the report says so and carries on. A diagnostic script that fails because an optional tool is missing is useless precisely when you need it most.
6. A mktemp -d with a trap, following 04-06: it leaves no residue even when interrupted.
7. The final section points to the next step. The script automates the mechanical steps; the interpretation remains human, and handing the operator the decision tree is more useful than trying to conclude automatically.

Solution 3

Technical reply: the proposal to set kernel.yama.ptrace_scope = 0

What is gained. Being able to run strace, gdb and perf against processes belonging to the same user without sudo. In practice this saves typing four characters, because on srv-tramontana the application runs as svc-tramontana and we run as operator: with ptrace_scope = 0 we would still need sudo, since the value 0 only allows tracing processes belonging to your own user. The real benefit of the proposal, in our specific case, is zero.

What is lost, exactly. ptrace allows reading and writing all the memory of another process. The memory of the application's process contains, in the clear and necessarily:

• The database password, which in 06-05 we took out of app.conf and encrypted with systemd-creds precisely so that it would not be readable.
• The TLS private key, if the process loads it.
• The personal data of guests in the requests being served.

With ptrace_scope = 0, any compromised process running as the same user can extract all of that without touching a single file: with no writes for AIDE to detect, no accesses for auditd to record on the paths we watch, and no trace left in the logs. In other words, it would in practice undo the work of two entire lessons from the previous module.

Being specific about our threat model from 06-06: the second most likely threat is a leaked credential, and the fourth is abuse of legitimate access. This proposal opens a direct route for both and closes neither.

Alternatives, in order of preference.

1. Use sudo. It is the right answer 95% of the time. CAP_SYS_PTRACE ignores Yama's restriction, it is recorded in auth.log — which is an advantage, not a drawback — and it does not change the system's posture. Cost: five characters.
2. Use eBPF, which is the better tool. perf trace, tcplife-bpfcc, biolatency-bpfcc, offcputime-bpfcc and bpftrace do not use ptrace at all, so ptrace_scope does not affect them. And they are between ten and a hundred times cheaper, which makes them the only ones genuinely suitable for production. If the underlying motivation is "diagnosing comfortably", this is the technical answer, not lowering a protection.
3. If tracing without privileges really is needed, there is a middle route: grant CAP_SYS_PTRACE to the diagnostic binary alone, rather than opening up the whole system:

   $ sudo setcap cap_sys_ptrace+ep /usr/bin/strace
   

   Picking up the capabilities from 05-02. It is still an increase in surface — anybody who can run strace will be able to trace — but bounded to one binary instead of the entire system. Even so, I do not recommend it here: it does not solve our real case (different users) and it adds a privileged binary to audit.
4. Lower it temporarily, only during a diagnostic session, with restoration guaranteed by a trap (~/scripts/trace_temporarily.sh). Acceptable in the lab; in production it is unnecessary given point 2.

Recommendation. Keep kernel.yama.ptrace_scope = 1. The proposal brings no benefit at all in our configuration — we would still need sudo — and it opens a route for extracting credentials and personal data that leaves no trace. If the underlying problem is friction when diagnosing, the solution is to install and learn bpfcc-tools and bpftrace, which will also let us diagnose in production and under load, something strace never allows.

I am adding two notes for the runbook: the value 1 is documented in /etc/sysctl.d/60-hardening.conf with a comment that already warned of this side effect; and it is worth recording here that the diagnosis of this week's latency incident was resolved entirely with eBPF and sudo strace, with no need to touch the parameter.

You now know how to look inside a running process. You have three families of tools and — most importantly — the criterion for choosing between them: the counters from Module 5 answer how much and are free, perf stat says whether a process is working or waiting at a negligible cost, eBPF instruments the kernel in production with histograms that reveal the tails the averages hide, and strace gives the exact answer at the price of slowing the process down by up to a hundred times. The order counters → perf stat → eBPF → strace is the practical lesson to take away, along with the one line that solves half of all configuration mysteries: strace -e trace=%file | grep ENOENT.

You have solved a complete case following the methodology: a measured, reproducible symptom, hypotheses ruled out with free counters, and five independent measurements converging on the same cause — a max_connections=200 against a max_connections=100, a mismatch that had been sitting there for three modules — with the fix verified from 400 ms to 40 ms. And you have run into the consequence of your own hardening: the ptrace_scope = 1 that prevents tracing without privileges, which turns out to be right, because a process's memory contains in the clear exactly the secrets you took such trouble to encrypt. That the correct solution to that friction is to use eBPF, and not to lower the protection, is the kind of decision that distinguishes an administrator from somebody following recipes.

Notice one detail of this diagnosis that points to what comes next: the cause was a configuration value, and you found it by comparing two sides of a single relationship. That is the territory of the next lesson, with one important difference: here the value was in the application, and now you are going to touch the kernel's. In lesson 07-03: Linux Kernel Tuning you will learn to adjust sysctl for performance — not for security, which you already did in 06-06: virtual memory with swappiness and the dirty page thresholds, the network with somaxconn and the BBR congestion control, the file and process limits, the I/O scheduler and why an NVMe wants none, and the transparent huge pages that every database asks you to turn off. You will also see kernel modules, dkms, and an honest answer to whether compiling the kernel makes sense on a production server. And you will do it with the rule that governs the whole business and that this lesson has already taught you to apply: you do not tune what you have not measured, one change at a time, and measure before and after. Remember that the default somaxconn is still sitting there, and that that errors.log with active_connections=200 had more than one cause.