You already know how Bash processes a line before launching it: it expands variables, substitutes commands, splits on spaces. This lesson adds the most powerful piece of that processing — globbing — and then something that looks very much like it and has nothing to do with it: regular expressions.

Confusing the two is the most widespread mistake among people who have been using the terminal for years. They look alike because both use * and ?, and those mean different things in each. The confusion is not academic: it produces commands that appear to work, return something plausible and are wrong. By the end of this lesson you will have the distinction fixed in your mind, you will be able to describe any set of files with a pattern and to build regular expressions that stand up to real cases over access.log and bookings.csv.

1. Who expands what: the shell first, the program afterwards
2. Globbing: the shell's wildcards
3. POSIX classes and brace expansion
4. Globbing options: globstar, dotglob, nullglob, failglob
5. Regular expressions: BRE, ERE and PCRE
6. The metacharacters, one by one
7. Grouping, alternation and back-references
8. Greediness: when the regex takes too much
9. A method for building a regex step by step
10. Practical regexes from the running example

The rule, in one sentence: globbing is done by Bash over file names before the command runs; regular expressions are interpreted by the program over the text it receives.

The command never gets to see the wildcard. Check it with a command that does nothing but print its arguments:

In the first case echo received two already-resolved arguments. In the second, the quotes prevented globbing and it received the literal. This is why ls *.log and grep '.*\.log' file bear no resemblance to each other: the first asks the shell for files, the second asks the program to look for a pattern inside the text.

Here the two things live side by side: *.log was expanded by Bash (which is why grep knows there are two files and prefixes the name), while GET is the pattern grep interprets. When a pattern is for the program, it goes in single quotes. Always. If you write grep *.log file without quotes and there is an access.log in the directory, Bash substitutes it and grep ends up looking for the string "access.log" inside errors.log. A plausible and wrong result.

The last two rows are subtle and worth their weight in gold. A glob has to match the complete name, whereas grep 'GET' matches any line containing that string in any position.

Three properties of globbing to keep in mind:

• * does not cross slashes. /opt/*/app finds nothing nested any deeper; that is what ** is for, and you will see it in section 4.
• Hidden files do not match *. echo * in your $HOME does not list .bashrc. It is deliberate: it stops an rm * from taking your configuration with it.
• The result comes out sorted according to the locale, with the consequences you already know from 03-01.

The course convention — looking with ls before an rm — is exactly a globbing check: you run the pattern with a harmless command and see what the dangerous one would have received.

Inside square brackets you can use named classes, which are more readable and more correct with accented characters than a hand-written range:

The double brackets are confusing: the outer ones are the set's, the inner ones are part of the class name. They combine with other characters inside the same set: [[:digit:]-] matches a digit or a hyphen.

{a,b} and {1..10} look like wildcards and are something else: they generate text, whether the files exist or not. It happens before globbing.

(The real output comes out on a single line; here it is split so that it can be read.) That directory structure you already have was created in exactly this way, with mkdir -p and a brace expansion. The ranges accept a step: {1..10}, {a..e}, {0..30..5}.

The most profitable everyday use is avoiding the repetition of a long path:

The brace {,.bak-...} expands into two arguments: the original and the original with a suffix. It is the shorthand form of the course's backup convention.

They are enabled with shopt -s and disabled with shopt -u.

This behaviour is surprising and it is worth understanding before you suffer from it. By default, if a glob finds nothing, Bash leaves it as it is and passes it to the command as literal text:

Notice that the message contains the asterisk: ls received the unexpanded pattern and tried to open a file literally called *.gz. With ls it is harmless; with a destructive command, less so. Imagine rm /srv/tramontana/backups/temp/*.tmp when there are no .tmp files left: at best it gives an error, but a command that interpreted the argument differently could do something unexpected.

• nullglob is what you want when the pattern can legitimately match nothing.
• failglob is the safest for interactive work: the command aborts before it runs.

Now it is Bash that refuses, and ls never ran.

A regular expression describes a set of strings. The historical problem is that there are three dialects and the Unix tools do not agree on which one they use.

In other words: in BRE the parentheses are literal and \\( is the metacharacter; in ERE it is the other way round. Hence the same expression gives different results with and without -E:

The third one looks for the literal string GET|POST, which never appears. A firm recommendation: always use grep -E. You write fewer backslashes, it reads better and it is the syntax shared by awk, egrep and almost every modern language. Reserve -P for what only PCRE offers (\d, lazy quantifiers, lookahead), knowing that it is not available on every machine.

We work on /var/log/tramontana/access.log, which has this format:

The difference between * and + is the one that generates the most false positives: * accepts total absence, so grep 'ms=[0-9]*' matches even a line with an empty ms=.

The escapes only exist in PCRE:

\b deserves attention because it solves a constant problem. Searching for 200 in the log also matches ms=200 or /bookings/1200:

The spaces are more restrictive than \b (which would also accept =200). In this case the first is the correct one, because the status code is always surrounded by spaces. grep's -w option does the same as wrapping the whole pattern in \b.

Parentheses do two things: they delimit the scope of an operator and they capture what they match so that it can be reused.

Without parentheses, the alternation would extend to the end of the pattern and 1[4-9]|2[0-9]: would mean "09:1[4-9]" or "2[0-9]:", which is something else.

A back-reference \1 matches exactly the same text that the first group captured. It is used to detect repetitions, which is something no pattern without capture can express:

No output: there are no duplicated words. A real use is finding repeated octets or, in bookings.csv, detecting a doubled separator:

Also no results, which is exactly what we want: no empty field caused by a double semicolon. A search with no output is a result, not a failure; it is worth saying out loud because it takes some getting used to.

The quantifiers *, + and {n,} are greedy: they match as much as possible. That is the cause of the classic error. Let us try to extract just the path from a log line with sed (which you will see in depth in 03-05; here it is only a demonstration):

We expected /bookings/1012 and we have got three fields. The reason: the .* inside the group ate everything it could while the pattern still matched globally. The solution in ERE is to forbid the separating character, instead of accepting any character:

[^ ]* cannot cross a space, so the group stops where it should. [^X]* instead of .* is the technique that solves 90% of greediness problems and it works in every dialect.

PCRE also offers lazy quantifiers, which match the minimum: .*?. With grep -oP '\/.*?\s' you would get the same result. It is convenient, but it depends on -P, so in portable scripts prefer the negated class.

Nobody writes a complex regular expression in one go. The method is incremental and always the same:

1. Look at the real data. head -3 of the file, not from memory.
2. Start with the most distinctive part and check that it matches something: grep -E 'ip=' access.log | head -3.
3. Add one piece at a time, verifying the count with -c after each addition. If the number changes unexpectedly, the problem is in the last thing you added.
4. Use -o to see exactly what is matching, not the whole line. It is the most useful debugging tool grep has.
5. Check the false negatives too: grep -vE 'pattern' file | head shows you what is escaping you. It is usually more revealing than what does match.
6. Only when the pattern is validated should you use it on something that modifies data.

-o prints only the part that matches, one per line. With that you can see immediately whether your pattern is going too far or falling short.

Extracting IP addresses. A "properly done" IP would require validating that each octet is 0-255, which produces an unreadable pattern. For your own logs, with a known format, this is enough and it reads well:

([0-9]{1,3}\.){3} is "a group of one to three digits followed by a dot, repeated three times", plus a final group with no dot. The dots are escaped: without the backslash they would match any character.

Error status codes. The 4xx and 5xx ones, which are the interesting ones:

The spaces around the pattern avoid matching the first three digits of a booking identifier.

ISO dates. [0-9]{4}-[0-9]{2}-[0-9]{2} is enough for a controlled format. If you want to restrict it to valid months: [0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01]). That level of rigour makes sense when validating input, not when searching in your own logs.

Validating the format of bookings.csv. The file has the header id;date;house;guest;nights;amount and 25 records. A valid line is: a four-digit id, an ISO date, a house name in lower case with hyphens, a guest name, a number of nights and a decimal amount.

Only one line fails validation: the header, which indeed does not comply with the format of a record. That confirms that the 25 records are correct. Note the three design decisions in the pattern: ^ and $ force the whole line to be validated (without anchors, grep would match any fragment and the validation would be worthless); [^;]+ in the guest name accepts spaces and accents but not an extra semicolon; and [0-9]+\.[0-9]{2} requires exactly two decimal places in the amount.

To see which one fails, grep -nvE '...' with -n gives you the line number. This pattern, saved, is an integrity check you can run before every import.

• Not quoting the grep pattern. If it contains *, ? or [, Bash expands it first and you search for something else. Single quotes always.
• Believing that * in a regex means "anything". In a regex, * applies to the previous element. "Anything" is .*.
• Forgetting to escape the dot in an IP or in an extension. 10.0.2.15 also matches 10x0y2z15. Write 10\.0\.2\.15.
• Using .* where a negated class belongs. If the result takes too much, change .* for [^delimiter]*.
• Mixing dialects. If you write \d without -P, grep looks for the literal letter d and gives no warning. It is a silent failure.
• Validating without anchors. A validation pattern without ^ and $ validates nothing.
• Tip: when a pattern does not work, do not rewrite it from scratch. Take pieces off it until it matches something and then add them back one at a time.
• Tip: grep --color=always -oE 'pattern' is your debugger. And echo 'test line' | grep -E 'pattern' lets you test against a made-up case without touching the real file.

Exercise 1. Without using find or grep, list with a single glob all the release directories of the 3.2 series in /opt/tramontana/releases/, and explain why 3.2* and 3.2.* are not equivalent. Then create with a single command the structure /home/operator/work/2026/{10,11,12}/{reports,data}.

Exercise 2. Extract from access.log all the POST requests that returned an error code (4xx or 5xx), showing only the method, the path and the code. Build the pattern step by step and show the verification of each step.

Exercise 3. Marta asks you to make sure that bookings.csv has no badly formatted amounts before the monthly import. Write a check that detects amounts without exactly two decimal places, verify that the current file is clean, and prove that your pattern works by testing it against a made-up line that is genuinely wrong.

Solution 1.

3.2* would also match a hypothetical 3.20 or 3.25, because * includes the empty string and does not require the dot. 3.2.* forces there to be a dot after the 2, which in a semantic versioning scheme is exactly the difference between "the 3.2 series" and "anything starting with 3.2". With four releases the mistake is invisible; with forty, it is not.

The brace expansion generates the six paths and mkdir -p creates the intermediate levels. Note that this is not globbing: it works precisely because the directories do not exist yet.

Solution 2. Step by step, verifying with -c after each addition:

The first step confirms that there are POSTs. The second does not change the count, which shows that all the POST lines have a path in the expected format: if it had dropped, we would have malformed lines. The third filters by error code: 18 of the log's 23 error responses are POST requests. [^ ]+ instead of .* stops the path from eating the rest of the line, and [45][0-9]{2} matches any 4xx or 5xx without listing them. -o trims the output down to the essentials and the final pipeline groups it; that sort | uniq -c | sort -rn is the pattern you will study in depth in 03-05.

Solution 3.

Only the header. The 25 records have a well-formed amount. To exclude the header from the report and keep only real errors:

No output and exit code 1: grep found no line that breaks the rule, which is the desired result. tail -n +2 starts at line 2, skipping the header.

The important part of the exercise is the last one: a validator you have never seen fail is not validated. You test it against a known bad case:

It detects the amount with a single decimal place. Now you know that the pattern really does discriminate and is not returning "all fine" because of a syntax error. Test a good case too (340.50) and confirm that it does not flag it: a validator must fail when it should and only when it should.

This lesson has given you two pattern languages and, above all, the boundary between them.

• Globbing is done by Bash over file names, before anything runs. The command receives the already-resolved list and never sees the wildcard. You check it with echo.
• Regular expressions are interpreted by the program over the text it receives, and that is why they go in single quotes.
• You have mastered the wildcards *, ?, [abc], [a-z], [!abc] and the POSIX classes, and you know that * neither crosses slashes nor matches hidden files.
• You distinguish brace expansion — which generates text whether the files exist or not — from globbing, and you use {,.bak-$(date +%F)} for backups.
• You control globstar, dotglob, nullglob and failglob, and you know what Bash does by default when a pattern matches nothing: pass it literally.
• You know the difference between BRE, ERE and PCRE, and why the recommendation is grep -E.
• You handle the metacharacters one by one, grouping, alternation and the back-references \1.
• You understand greediness and you know that the portable solution is to replace .* with a negated class [^X]*.
• And you have a method for building patterns: real data, one piece at a time, -c to count, -o to see what matches, -v to see what is escaping, and testing the validator against a bad case before trusting it.

In the next lesson, Searching Files and Content: find, locate and grep, these patterns stop being an exercise and become the interface to three tools. find walks the directory tree applying criteria that include globs; grep applies regular expressions to millions of lines in seconds; and locate answers instantly by consulting a database. You will know which one to use in each case, how to combine them with xargs without a name containing spaces ruining everything, and how to locate on srv-tramontana the releases that have to be purged, the .bak-* files scattered around the system and the configuration file where somebody left a credential written down.