The previous lesson ended with a clear boundary: awk reads and summarizes, but it is not the tool for rewriting text while preserving its shape. When the IPs in access.log have to be anonymized before sending it to a provider, when the separators of a badly exported CSV have to be fixed or when a key in veloz-ops.conf has to change without opening an editor, what you need is a stream editor: a program that takes text in on one side, applies a list of editing commands to each line and drops the result out the other. That is sed, and it works with the same regular expressions you have commanded since 05-04, only applied to transforming instead of filtering.

1. The sed cycle and the pattern space
2. sed, awk or ${var//a/b}: which to use
3. Syntax and options
4. -i: editing files in place, with a safety net
5. Addresses: which lines a command applies to
6. The s/// command in depth
7. Other commands: p, d, q, a, i, c, y
8. Filtering and extracting ranges
9. Chaining commands and blocks
10. Hold space and N, in broad strokes
11. Real recipes from the toolkit
12. Application: veloz_anonymize_log in lib/common.sh

sed does not see the whole file: it works line by line over a buffer called the pattern space. For each line of the input it always repeats the same cycle.

Three consequences follow from this diagram, and they explain almost all of sed's behavior:

• The commands are applied in order over the same buffer. If the first one replaces A with B, the second already sees B. Chaining substitutions is not the same as applying them separately.
• Printing is automatic at the end of the cycle, the command does not do it. That is why sed 's/a/b/' prints every line, not just the ones it changed.
• -n turns that automatic printing off, and from then on only what you explicitly ask for with p comes out. Hence the idiom sed -n '...p'.

All three replace text, and choosing badly is the most common cause of convoluted code:

The practical rule: if the text is already in a variable, ${var//a/b} and do not spawn a process; if a file or a stream has to be rewritten, sed; if you have to sum, count or look at columns, awk. And if you find yourself counting fields with sed, or substituting with awk in several passes, you have almost certainly picked the wrong tool.

The commands go inside single quotes for the same reason as in awk: they contain $, & and backslashes that Bash would expand. The options that actually get used:

-E deserves an explicit recommendation. By default sed speaks BRE, the dialect from 05-04 in which +, ?, (, ) and | are literals and have to be escaped: s/[0-9]\+/N/. With -E you write s/[0-9]+/N/, which is what you have in your head. Use -E always, unless you are copying someone else's recipe written in BRE.

-i is sed's most useful and most dangerous option: it rewrites the original file without asking and with no way to undo. A badly written pattern can empty a configuration file in production.

Two habits that avoid disaster:

1. Always test without -i first. Run the command as is, look at the output, and only then add -i. It is free and it solves 95% of the scares.
2. Use a backup suffix, -i.bak, not a bare -i. It costs five keystrokes and leaves the original recoverable.

There is also a classic incompatibility here: in GNU sed (Linux) the suffix goes attached, -i.bak, and a bare -i is valid; on BSD/macOS -i demands an argument, so with no backup you write -i ''. Consequence: sed -i.bak works on both, sed -i only on Linux, and a script using sed -i '' will fail on Linux by interpreting '' as a file name. If the script has to run in both places, the clean way out is to write to a temporary file (mktemp, 05-01) and move it over with mv.

And a serious warning: never run sed -i over files in /etc without a prior backup. sed -i 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config is a security change made blind, with no validation and no way back. The implications are covered in 08-03.

A command with no address applies to every line. Prefixing it with an address restricts it:

So sed -n '/2026-08-03 10:/,/2026-08-03 11:/p' /var/log/veloz/app.log extracts one hour from the log, and sed '1!d' shipments.csv leaves only the header. The ! reads as "except": 1!d is "delete the lines that are not line 1". It is the idiomatic way to keep only something, and it shows up constantly in real recipes.

It is the star command and the one that justifies 90% of sed's uses. Its form is s/regex/replacement/flags:

The delimiter does not have to be /. Any character will do, and choosing well avoids "leaning toothpick syndrome" when working with paths:

The flags that go after the last delimiter:

In the replacement there are two metacharacters you have to know. & means "everything that matched", and \1, \2… are the capture groups from 05-04, with the groups written \\(...\\) in BRE or (...) with -E:

Practical corollary: if you want a literal & in the replacement, escape it (\&), just as you have to escape the delimiter if it shows up inside the pattern. And remember that s/// without g changes only the first occurrence on each line: it is oversight number one.

d has an important quirk: it aborts the cycle, so the later commands do not run on that line. And q is a silent optimizer: sed '100q' huge_file stops reading at line 100, whereas head -100 with a pipe would keep reading. The a/i/c syntax with the text on the same line is a convenient GNU extension but not portable; the POSIX form uses a backslash and a newline.

With -n and p you filter just like with grep: sed -n '/ERROR/p' app.log does exactly the same as grep ERROR app.log. If you are only going to filter, use grep. It is more readable, it is usually faster, and it has -c, -v, -i, -l and -o for the common cases. sed wins as soon as the operation stops being "yes or no" and becomes "transform", or when what you want is a range by line number, which grep cannot do:

That ;20q is the detail that makes the difference in a two-gigabyte log: without it, sed keeps reading to the end even though it no longer prints anything.

Several commands are separated by ; or by several -e. And if you want to apply a group of commands to the same address, you group them with braces:

In the second example, { s/^/ /; p } applies only to the lines in the range: first it indents, then it prints. Order matters —remember section 1: the commands see the buffer just as the previous one left it—. If you swapped p and s, you would print the line without the indentation.

Besides the pattern space, sed has a second buffer, the hold space, which survives between lines. It is handled with h (copy pattern→hold), H (append), x (swap) and G (append hold→pattern). It is what gives sed a minimal memory: sed G report.txt adds a blank line after each line, and the cryptic sed '1!G;h;$!d' file reverses the order of the lines —sed's tac—.

The N command appends the next line to the pattern space, allowing a regex to span two lines —useful for joining an app.log line with its stack trace—. That said, here is the tool's honest limit: as soon as you need this level of acrobatics, awk or a scripting language will be more readable and easier to maintain. Know that it exists, do not make it your style.

Anonymizing the IPs in access.log while keeping the first octet so you still know the source network, and normalizing the separators of a CSV exported with semicolons and a decimal comma:

Changing a configuration key idempotently, that is, so it can be run a thousand times with the same result and works whether the key existed or not:

The if is not decoration: sed cannot add a line that does not exist, only transform the ones that are there. Without the prior check, the script would seem to work and would change nothing. Note the ^ anchor, which avoids touching a key that merely contains that text, and the .* that eats the old value whatever it is.

Cleaning up a configuration file to see it without noise, and inserting a header into a CSV that lost it:

The IP recipe becomes a reusable function, following the library rules from 05-06: definitions only, namespace prefix, no top-level code:

Three decisions deserve a comment. The function does not use -i: it writes to standard output and lets the caller decide (veloz_anonymize_log "$LOG" > /tmp/share.log), which is the right thing in a library —a function that modifies other people's files by surprise is a trap—. The two separate -e are more readable than one long ; and let you comment one out while debugging. And the -- marks the end of options, so a file called -weird.log is treated as a file and not as an option; it is the same defensive habit as in 03-05. Code 66 is EX_NOINPUT, from the table in 05-03.

Verifying before trusting is mandatory with sed: a quick check like head -3 /var/log/veloz/access.log | veloz_anonymize_log /dev/stdin is enough to see that the IPs come out masked and the rest of the line intact.

• Forgetting the g flag. s/,/;/ changes only the first comma on each line. For all of them, s/,/;/g.
• Using -i without testing first. Always run the command without -i, look at the output and only then edit. And use -i.bak, which works on GNU and on BSD.
• Expecting ERE by default. sed 's/[0-9]+/N/' does nothing, because in BRE + is a literal +. Add -E.
• Escaping slashes by hand. If the pattern carries paths, change the delimiter: s|/a/b|/c/d|.
• Confusing the . inside an IP. [0-9]{1,3}.[0-9] also matches 192x168; a literal dot is written \..
• Believing sed can add missing keys. It only transforms existing lines; to add, check with grep -q and use >>.
• Using sed to filter. If it is a "yes or no", grep; if the text has to change, sed; if there are columns or arithmetic, awk.
• Tip: with large files, sed -n '10,20p;20q' and sed '100q' avoid reading more than needed. LC_ALL=C sed ... also speeds things up.
• Tip: a long sed script is pulled out into a .sed file and invoked with -f, just as in awk: it takes # comments and gets versioned.

Exercise 1. Write a command that extracts from /var/log/veloz/app.log only the lines in the interval between 10:00:00 and 11:00:00 and strips the date from them (leaving only time, level and message), without modifying the original file.

Exercise 2. Write a function veloz_set_conf for lib/common.sh that sets a key to a value in ~/veloz-ops/etc/veloz-ops.conf idempotently, with a backup, and that preserves the file's 600 permissions.

Exercise 3. The provider sends shipments-raw.csv with ; as the separator, amounts with a decimal comma, stray spaces around the fields and an initial comment line starting with #. Write a single sed that leaves it in shipments.csv format.

Solution 1.

The /start/,/end/ range selects the stretch, and the braces apply both commands only to it: first the date is stripped with an s/// anchored at ^, then it is printed with p. -n is essential —without it the whole file would come out, with the range duplicated on top by the p—. And the order inside the braces matters: if p came first, it would print the line with the date still on it.

Solution 2.

Here double quotes are used, because $key and $value have to be interpolated, and that is why the prior validation with =~ (05-04) is not optional: without it, a key containing | or & would break the substitution or inject behavior. The | delimiter avoids trouble if the value is a path, and chmod 600 at the end restores the permissions, because sed -i creates a new file and renames it, so the permissions can change. Code 64 is EX_USAGE.

Solution 3.

The order is deliberate and it is the whole exercise. First the comment is deleted (d aborts the cycle, so that line is not even processed). Then the ; are normalized absorbing the spaces around them, which fixes the interior padding in one go. Next the ends of the line are trimmed. And only at the end is the decimal fixed: if that substitution came earlier, the , just created by the ; step could be mistaken for the decimal one. It is worth verifying with head -3 shipments-raw.csv | sed -E '...' before overwriting anything.

sed is a stream editor: for each line it copies it into the pattern space, applies the commands in order and prints it at the end of the cycle unless -n prevents it —hence the idiom sed -n '...p'—. Its commands accept addresses (line number, $, regex, ranges 2,5 and /start/,/end/, and the negation !) that decide which lines they apply to, and the king of them all is s/regex/replacement/flags, with a free delimiter so you do not fight with paths, flags g, i, p and w, and a replacement where & is what matched and \1 the groups from 05-04. Around it are d, p, q, a, i, c and y, the { } blocks for grouping by address, and the hold space (h, x, G, N) as a minimal memory worth knowing and not abusing. -E to write ERE comfortably, -i.bak always with a backup and testing first without -i, and a clear boundary with its neighbors: ${var//a/b} for a variable, grep to filter, awk for columns and arithmetic, sed to rewrite.

With awk and sed, the toolkit already knows how to read, summarize and transform any text you throw at it. But all that text still comes from files somebody left on disk. An operations script needs something more: to ask the system itself what state it is in —how much disk is left, what load it is carrying, who is connected, what operating system version it runs, whether the veloz-api service is alive— so it can decide based on the answer. That is the next lesson (06-03): the commands that interrogate the system and how to turn their answers into thresholds and decisions inside a script.