In Module 1 you read that in Unix "plain text is the universal interface". It sounded like a statement of principle. In the previous lesson you saw the mechanism that makes it possible — pipes and redirection — and here come the tools that give it meaning: a set of small programs, written forty years ago, that know how to cut, sort, count, substitute and aggregate text, and that combined solve in one line what in another environment would require a program.

That is this lesson's concrete promise: bookings.csv and access.log are text files, and Marta needs reports. By the end you will be able to produce them without opening a spreadsheet and without writing code, and — more importantly — you will know when these tools stop being the right answer.

1. cut: cutting out columns
2. tr: translate, delete and squeeze
3. sort: sorting properly
4. uniq: counting occurrences
5. paste, join and comm: comparing listings
6. sed: the stream editor
7. awk: the language of fields
8. A decision table: which tool and when
9. Case A: a billing report for Marta
10. Cases B and C: error paths and anomalous IPs

It extracts parts of each line, either by fields or by characters.

cut is extremely fast and has one serious limitation: it does not understand repeated delimiters. With a file aligned by spaces, every extra space counts as an empty field:

Empty output: field 2 is the space between GET and the next thing. For output separated by variable numbers of spaces — which is what almost any command produces — the right tool is awk, which treats any block of whitespace as a single separator. cut for formats with a fixed delimiter (CSV, /etc/passwd); awk for everything else.

tr operates character by character and only reads from stdin.

You have already seen it in action: tr -s ' ' solves the cut problem from the previous section. Other common uses: tr -d '\r' to clean up files coming from Windows — a classic source of incomprehensible failures — and tr -cd '[:print:]\n' to remove control characters from a dump.

The -k syntax is the most widely misread. -kN means "from field N to the end of the line", not "by field N". To sort by one field only you have to write -kN,N.

-t';' sets the separator, -k6,6nr sorts by field 6 numerically and in reverse. The n and r modifiers are attached to the -k so that they apply only to that field; on their own they would affect the whole line.

-V is specialised and very useful in our case: ls /opt/tramontana/releases/ | sort -V orders 3.1.0, 3.2.0, 3.2.1, 3.3.0 correctly, whereas a normal sort would put 3.10.0 before 3.2.0.

Remember from 03-01 that sort's result depends on the locale. When the output is going to be compared or version-controlled, LC_ALL=C sort.

uniq only compares adjacent lines. That is not a defect: it is what lets it process enormous files without memory. The consequence is that it requires sorted input.

Hence the workhorse of log analysis, which you have already used without explanation:

It groups equal things together, counts each group and sorts by descending frequency. It is the answer to any question of the form "what comes up most often?".

paste sticks files together side by side (paste -d';' a.txt b.txt). join performs a relational join on a common field and requires both files to be sorted by it. comm compares two sorted files and produces three columns: only in the first, only in the second, in both.

No output: all five houses in the catalogue have at least one booking. -23 suppresses columns 2 and 3 and leaves only "in the catalogue but with no bookings", which is exactly the question. <(...) is a process substitution: Bash delivers that command's output as if it were a file, which avoids creating temporaries.

sed reads the input line by line, and for each one it runs the commands you have given it and, unless you say otherwise, prints it. That cycle explains everything: sed does not load the file into memory, which is why it processes files of gigabytes.

s/pattern/replacement/flags, with these flags:

The delimiter does not have to be /: any character will do, and using another one avoids the "picket fence of slashes" when the pattern contains paths.

Capture groups work as in regexes: \1, \2… reuse them in the replacement and & stands for everything matched. With -E there is no need to escape parentheses or braces:

With -n, sed prints nothing of its own accord and only what p selects comes out: sed -n '/ERROR 500/p' errors.log returns the 41 error lines, that is, sed imitating grep. Several commands are chained with a repeated -e or by separating them with ;.

sed -i modifies the file directly. In production, never without a suffix.

With -i.bak-$(date +%F), sed saves the original with that suffix before writing. Without it, if the pattern was wrong — and a badly written pattern can empty half a file without warning — there is no way back: sed does not ask, does not warn and does not undo. The course's prior-copy convention and this option are the same thing, built into a single command. Note as well the ^ and $ anchors: without them, the pattern would also match inside a comment.

The correct procedure is always in two steps: run sed without -i to see the result on screen, and only then add -i.bak-....

awk is a complete programming language disguised as a command. Its model is:

For each line, if the pattern holds, it runs the action. With no pattern, it applies to all of them; with no action, it prints the line.

By default FS is "any block of spaces or tabs", which is precisely what cut cannot do:

For CSV it is changed with -F';'. The BEGIN and END blocks run before the first line and after the last one:

NR>1 skips the header — a pattern with no braces is a condition — the variables create themselves with the value 0, and printf formats: %d integer, %.2f decimal with two figures.

awk -F';' '$5 >= 5 {print $1, $3, $5}' bookings.csv returns the three stays of five nights or more. And now the feature that makes awk something different: associative arrays, which allow you to group and total in a single pass.

total[$3]+=$6 uses the house name as the index: there is nothing to declare and no need to know in advance how many houses there are. The for (h in total) loop walks the keys — in an unpredictable order, hence the final sort — and %-14s left-aligns in 14 characters.

Why awk replaces half a pipeline: that command does in one pass what would require cut | sort | uniq plus a sum that none of the three knows how to do. When you need to group and calculate at the same time, awk is not one option among several: it is the only one in the set that can.

The last row is professional advice, not a surrender. A six-link pipeline with two nested seds is harder to maintain than fifteen lines of readable Python. A reasonable cut-off point: if it does not fit on one screen or if you have to draw it to understand it, change tool. And for JSON, do not even try it with sed: jq exists.

The brief: total billed, average nights and amount per house for the period loaded into bookings.csv. It is built up in steps, verifying as you go.

A prior check with awk -F';' 'NR>1 {n++} END {print n, NF}': 25 records and 6 fields. If any line did not have 6 fields, any subsequent sum would be wrong; the validation from 03-02 already told us the format is correct.

Three associative arrays indexed by the same key, three global accumulators and an END that formats. The awk program goes in single quotes and can span several lines: Bash touches nothing inside it. Note that the variable is called tr_ and not tr: inside awk there is no conflict with the command, but avoiding names that coincide with commands saves confusion when you reread it.

An indispensable final check: 7+6+5+4+3 = 25 bookings and the partial sums add up to the total. A report whose breakdown does not square with the total is not delivered.

B) Paths with the most errors. We extend the pipeline from 03-04 to all the 4xx and 5xx codes:

Five combinations, not ten: the listing is short because the log is. They add up to 23, and the 500s add up to 14, which is exactly what we counted in 03-04. $5 ~ /regex/ is awk's pattern comparison, applied only to the status code field, which avoids the problem we solved in 03-02 by surrounding the pattern with spaces. And there is an incidental finding: the four 404s for /houses/el-molí carry an accent, whereas in the catalogue the house is el-moli. Somebody is generating links with the accented name. It is not a server failure: it is an application failure, and Luis has to be told.

C) IPs with the most requests.

-F'ip=' splits each line on that string, and split($2, a, " ") cuts the rest at the first space to keep the IP. It is a way of extracting a named field without depending on its position.

Reading the figure is the important part. 10.0.2.77 accounts for 138 of the 412 requests — 33% — against the 94 of the second. Is it an attack? We do not know yet, and saying yes would be jumping the gun. We look at what it is doing:

A uniform pattern, almost all successful GET /bookings at a constant rate: it looks more like an automated system — a monitor, a crawler, an integration — than a person. The report for Marta must say that and no more: there is one IP accounting for a third of the traffic with an automated pattern; the available data does not allow a legitimate service to be distinguished from abuse; whose address it is must be identified before any measure is taken. Distinguishing what the data proves from what it suggests is part of the job.

• uniq -c without a sort in front. It only groups adjacent lines; the count will come out fragmented and will look correct.
• sort -k3 expecting it to sort by field 3. It sorts from 3 to the end. Write -k3,3.
• cut -d' ' over aligned output. Repeated spaces create empty fields. Use tr -s ' ' or awk.
• sed -i with no backup suffix. There is no undo. Always -i.bak-$(date +%F), and test beforehand without -i.
• Forgetting NR>1 in a CSV with a header. In awk a string is worth 0, so it gives no error: it silently falsifies the count.
• Quoting an awk program with double quotes. Bash will expand $1. Single quotes always.
• Comparing numbers as text. sort without -n puts 100 before 20. And sort first, format afterwards: printf's padding throws sort's fields out of alignment.
• Tip: leave a wc -l at the end while you are building a pipeline; if the number changes where you did not expect it to, that is where the error is. And remember that awk accepts several files: FILENAME tells you which one it is processing and FNR is the line number within that file, as against NR, which is global.

Exercise 1. Generate a listing of the houses sorted by average amount per night (total amount divided by total nights), from highest to lowest, with two decimal places. Explain why dividing the average amount by the average number of nights is not enough.

Exercise 2. The file /etc/tramontana/app.conf has max_connections=200 and query_timeout=30. Raise them to 400 and 60 respectively in a single command, with a prior copy and verification, and explain every element of the pattern you use.

Exercise 3. Marta asks what percentage of the requests in access.log ended in an error (4xx or 5xx) and which time slot they are concentrated in. Answer with a single pipeline for each question and write the conclusion in two sentences.

Solution 1.

A first attempt with printf "%-14s %7.2f\n", h, amt[h]/nig[h] and sort -k2 -nr comes out unsorted: -k2 means "from field 2 to the end", and since %-14s pads with spaces, field 2 does not start where sort thinks it does. The correction:

Putting the number first and with no padding, sort -rn works unambiguously. It is a technique worth adopting: sort first, format afterwards.

On the question itself: dividing averages is not enough because the average of quotients is not the quotient of averages. A booking of 7 nights weighs far more in the total number of nights than one of 2, and dividing averages treats both the same. The correct calculation is the one we have done, adding up amounts and nights separately and dividing the totals.

Solution 2.

Element by element: -E so as not to escape the parentheses; ^ and $ so that the pattern matches the whole line and not a fragment inside a comment; the group (key) and its \1 so as not to repeat the name in the replacement, which avoids typos; the old value in the pattern, which acts as a check — if the file already had a different value, the substitution would not be applied instead of overwriting it blindly; ; to chain two substitutions in a single command; and -i.bak-$(date +%F) for the copy. The first command, without -i, is the dry run: only when its output is what you expect do you run the second.

Solution 3.

substr($2,1,2) takes the first two characters of the time field. %% prints a literal percent sign in printf.

5.6% of the 412 recorded requests ended in an error, a proportion that on its own is not alarming. However, 15 of the 23 errors — almost two thirds — are concentrated in the 03:00 slot, which points to a one-off problem during the night-time window rather than a general degradation of the service.

That concentration fits with what we had been seeing: the db_timeout entries in errors.log, the exhausted query_timeout=30 and the active_connections=200 sitting at the configured limit. Everything points to the night-time task competing for database connections with the users' requests.

You have turned two text files into defensible reports, and along the way you have closed the circle opened by the Unix philosophy of Module 1: plain text as the universal interface between programs that know nothing about each other.

• cut cuts by a fixed delimiter, with its limitation in the face of repeated delimiters; tr translates, deletes and squeezes characters.
• sort sorts by field with the real syntax -kN,N, with -n, -h, -V, -u and -o; uniq -c counts and requires sorted input, and the pattern sort | uniq -c | sort -rn answers any frequency question.
• You know paste, join and comm for comparing listings, and the process substitution <(...) for avoiding temporary files.
• sed processes line by line; you handle addresses, the s command with its flags and alternative delimiters, d, p with -n, the capture groups, and you know that -i.bak-$(date +%F) is not optional and that the test without -i always comes first.
• awk gives you pattern { action }, the fields $1…$NF, NR, NF, FS, BEGIN/END, printf and the associative arrays that group and total in one pass, replacing half a pipeline.
• You have a decision table and a criterion for knowing when it is time to move to Python.
• And you have produced three real reports: the billing per house with its breakdown squaring, the paths with the most errors — discovering along the way an application bug with the accent in el-molí — and the IP accounting for a third of the traffic, with the honesty to say what the data proves and what it merely suggests.

Up to here you have worked with files. The next lesson changes its object: Process Management deals with programs while they are running. You will see what a PID is and how the process tree is organised from systemd, how to read ps aux and top column by column — including what the three load average numbers really mean — what signals exist and why a professional tries SIGTERM and waits before resorting to kill -9, and how to find out with lsof which process is holding a port or a file. The practical case will be waiting for you already set out: the Tramontana application hangs in the early hours, right in that 03:00 slot you have just identified, and it has to be diagnosed and restarted without losing the requests in flight.