Everything you have done in this module happened inside srv-tramontana. But a server only exists for whoever can reach it, and when Marta writes "the bookings website will not load", none of the earlier tools helps you work out where the break is. It could be in the network, in name resolution, in the port, in the application or in Marta's own browser, and those are five completely different problems with five different solutions.

This lesson gives you the diagnostic tools and, above all, the order in which they are used. That order is what separates somebody who resolves an incident in five minutes from somebody who spends two hours poking at things at random. We configure nothing here: persistent network configuration is 06-01, the firewall is 06-03 and SSH in depth is 06-02. Here you look, you measure and you conclude.

1. The minimum indispensable recap
2. ip: reading the real configuration
3. Connectivity and path: ping, traceroute, mtr
4. Name resolution
5. Ports and sockets with ss
6. Testing services: curl, wget, nc
7. What each tool tells you when "the website is down"
8. Transfer: scp and rsync over SSH
9. Bandwidth with iperf3
10. A layered diagnostic methodology
11. The Tramontana case: "the bookings website will not load"

Just enough to understand the commands' output, not a networking course.

• Layers. Data travels encapsulated: the application (HTTP) goes inside transport (TCP/UDP), which goes inside network (IP), which goes inside link (Ethernet). Each tool looks at a different layer, and that is why diagnosis is done from the bottom up.
• IP, mask and CIDR. 10.0.2.15/24 means that the first three octets identify the network and the last one the host. Anything in 10.0.2.x is directly reachable; for the rest an intermediary is needed.
• Gateway. That intermediary: the address everything non-local is sent to.
• DNS. Translates names into addresses. It is a separate layer and it fails on its own account, which explains the classic "the browser does not work but pinging the IP does".
• Ports. A 16-bit number identifying the service within a machine: 22 SSH, 80 HTTP, 443 HTTPS, 5432 PostgreSQL, 8080 our application's. Below 1024 they require privileges to open.
• TCP against UDP. TCP establishes a connection, guarantees ordering and resends what is lost; it is what the web uses. UDP fires and forgets; DNS and streaming use it. The difference matters when diagnosing: in TCP there are observable states, in UDP there is nothing to look at.

ip replaces ifconfig, route and arp, which have been obsolete for more than a decade and may not even be installed.

-brief gives the compact view you want 90% of the time: interface, state and addresses. lo is the local loopback; enp0s3 is the VM's card, UP and with the 10.0.2.15/24 you have known since Module 1.

Two routes. The second says that the 10.0.2.0/24 network is directly reachable. The first, default, is the gateway: anything that is not local goes out through 10.0.2.2, which on a VM with VirtualBox NAT is the host itself acting as a router. proto dhcp indicates that the configuration was given by a DHCP server, not by a file.

-s link gives the counters: errors and dropped other than zero point to a physical or saturation problem, and here they are at zero. ip neigh is the ARP table — which MAC addresses we have resolved — and REACHABLE confirms that we are talking to the gateway right now, without needing to send a single ping.

Three pieces of information: the TTL (64 suggests a Linux one hop away), the round-trip time and the loss. Response 2 is missing.

Why a lost ping is not always a failure. ICMP is the lowest-priority traffic on the network: equipment discards it as soon as it has more important things to do, and many firewalls block it entirely. A 33% loss over three packets is statistically nothing either. The practical consequences:

• A ping failing does not prove that the host is down. It may be filtered.
• A ping working does not prove that the service works. The machine responds; the application may be dead.
• Occasional loss in ICMP does not imply loss in TCP.

ping answers a single question: "is there an IP path to there?". Nothing more. With -c you limit the packets, with -i 0.2 you shorten the interval and with -M do -s 1472 you can diagnose MTU problems.

Each line is a hop. -n avoids reverse resolution and speeds the output up a great deal. The asterisks at hop 3 mean that that piece of equipment does not respond, not that the path is cut off there: if the later hops do answer, that router simply ignores diagnostic traffic. tracepath does the same without privileges and discovers the path's MTU.

mtr combines ping and traceroute in real time and is the best tool for intermittent losses: mtr -rwc 100 destination sends a hundred packets and gives a report with the loss percentage and the latency of each hop, which is how you tell a problem of your own from one belonging to the provider.

The order is set by /etc/nsswitch.conf:

files means /etc/hosts first. That is why a forgotten entry there beats any DNS in the world and produces baffling incidents: the machine resolves a name to an IP that no longer exists and no change in the DNS fixes it. When a name resolves wrongly, look at /etc/hosts before anything else.

An important detail: dig does not consult /etc/hosts, it goes straight to the DNS. It is exactly what you want in order to distinguish "the DNS is wrong" from "somebody touched /etc/hosts". The sections of the reply are QUESTION (what was asked), ANSWER (the answer, with its TTL in seconds), AUTHORITY and ADDITIONAL.

dig @1.1.1.1 against a plain dig is the decisive diagnostic: if an external server resolves and yours does not, the problem is your resolver, not the domain.

ss replaces netstat and is the central tool of this lesson.

Option by option:

The Local Address column is the one that gives the most information and the one fewest people read:

Look at the last line of the output above. executable is listening on 127.0.0.1:8080. Keep that fact in mind.

TIME-WAIT deserves an explanation because it frightens people for no reason: after a connection closes, the end that closed first keeps the socket for about 60 seconds in case a delayed packet arrives. Thousands of TIME-WAIT entries on a server with traffic are normal. Many CLOSE-WAIT entries, on the other hand, are a symptom: they mean the application is not closing its sockets, and that ends up exhausting descriptors.

netstat -tulpn does the same thing and you will still see it in old documentation; it is in the net-tools package, which is no longer installed by default. Write ss.

That one-liner is the one you will use most: -s silences the progress bar, -o /dev/null discards the body and -w prints exactly what interests you. A code and a time.

--resolve is the option that resolves more incidents than it appears to: it lets you test the server before touching the DNS, or check whether the problem is one of resolution without changing anything.

"Connection refused" is a highly informative message: it means the packet arrived and the machine actively replied that there is nobody listening there. It is not a firewall — that would give a timeout — nor a routing problem.

nc checks a port with no further ceremony, and wget downloads files (-c resumes, -r recursive, -O - dumps to stdout):

Two almost identical tests with opposite results. That is a finding, and it confirms what we already saw in ss.

telnet host port still works as a last resort for talking to a text service by hand, but nc and curl do it better and are more widely available.

The important reading of the table: each tool rules out a layer. They are not for "trying things out"; they are for eliminating hypotheses in order.

scp copies and that is that. rsync is superior in almost everything: it transfers only the differences, compresses with -z, preserves permissions with -a, shows the progress with --progress and — the essential point according to the course convention — accepts --dry-run to see what it would do before doing it. You were already using it locally back in 02-04; over SSH it works the same way, the only change being that the destination carries user@host:.

ping measures latency, not capacity. To measure real throughput you have to generate traffic: iperf3 -s at one end and iperf3 -c <server> -t 10 at the other. The report gives the Mbit/s achieved and, with -u, the loss and the jitter in UDP. It is the way to answer "the network is slow" with data, which almost always turns out to be something else.

This is the section to remember. You work from the bottom up and you skip no step.

Three rules that go with the diagram:

1. Write down the result of each step. A diagnosis is a chain of eliminations, and without a record you end up repeating tests.
2. Change nothing while you are diagnosing. If you touch the configuration halfway through, you no longer know what caused what.
3. Test from both sides. Something working on the server and not from outside narrows the problem down better than any other test.

13:05. A message from Marta. We apply the procedure.

Step 1: IP and route? ip -brief a gives enp0s3 UP 10.0.2.15/24 and ip r shows the default route via 10.0.2.2. Correct.

Step 2: the gateway?

No loss and latency in tenths of a millisecond. Correct.

Step 3: does the name resolve?

getent hosts is better than dig for this question: it queries by the same route the applications use, respecting /etc/nsswitch.conf, so it includes /etc/hosts. It resolves to 10.0.2.15, which is the correct IP. Correct.

Step 4: is the port listening?

There it is. The process is listening, but on 127.0.0.1:8080: it only accepts connections from the machine itself. From Marta's laptop it is unreachable, and no firewall and no DNS has anything to do with it.

Step 5: confirm it from both sides.

Diagnosis closed: the application is working perfectly and is listening on the wrong interface. It replies 200 over the local loopback and refuses the connection over its own IP.

The cause. The process restarted last night, and app.conf does not set the listening address, so the application used its default value. Before the restart it had been working for months because somebody had started it by hand with a parameter that was never recorded anywhere. It is the exact pattern 03-06 warned you about: a service launched by hand does not survive a restart with the same configuration.

The solution is to add the listening address to /etc/tramontana/app.conf with the usual convention — a .bak-$(date +%F) copy, sed -i, diff -u — and restart the process with SIGTERM and a wait. The definitive solution, having the configuration fixed in a service unit that starts on its own and always the same way, is systemd and that is 05-05.

The report for Marta. The service never stopped working: it was still serving requests, but only from the server itself, because after last night's restart it came up listening only on its internal address. It is already corrected in the configuration file, so the next restart will keep the setting. This protects against the incident recurring for the same reason. It does not protect against two things: nobody warned us of the outage — we detected it because you said so, not because of a monitoring system — and the service is still started in a way that depends on somebody doing it properly. Both have known solutions — monitoring and service management — and I propose we tackle them in the next module.

• Concluding "it is down" because the ping fails. ICMP is filtered all the time. Test the port with nc or curl.
• Concluding "it works" because the ping responds. The machine responds; the service may be dead.
• Testing only from the server. curl to 127.0.0.1 works even with the service badly bound. Test over the real IP too.
• Ignoring ss's Local Address column. 127.0.0.1:port against 0.0.0.0:port is the difference between accessible and unreachable.
• Forgetting /etc/hosts. It beats the DNS and produces incidents that are impossible to explain by looking only at the DNS.
• Using dig to find out how an application resolves. dig ignores /etc/hosts; use getent hosts.
• Being alarmed by TIME-WAIT entries. They are normal. Accumulated CLOSE-WAIT entries are a symptom.
• Touching the configuration mid-investigation. Diagnose first, change afterwards, and only one thing at a time.
• Tip: curl -w '%{http_code} %{time_namelookup} %{time_connect} %{time_total}\n' splits the total time between DNS, connection and response. With a single command you know which phase is the slow one.
• Tip: save the output of ip a, ip r and ss -tulpn from a server while it is working properly. Comparing with diff -u against the current state is the fastest diagnosis there is.

Exercise 1. Document your VM's complete network configuration: interfaces with their state and address, gateway, DNS servers and error counters. Save it in /home/operator/data/network-reference.txt with the date, and explain what that file will be for.

Exercise 2. Check whether the application's service is accessible from another machine without leaving the server, and explain why curl http://127.0.0.1:8080/ does not answer that question. State which command gives you the definitive answer and why.

Exercise 3. Apply the diagnostic methodology to this case: from laptop-student, ping srv-tramontana responds, but the browser hangs indefinitely when opening http://bookings.tramontana.example:8080/. List the checks in order, with the command for each one, and say what conclusion you would draw from each possible result.

Solution 1.

The braces group several commands so that all the block's output can be redirected with a single >, as you learned in 03-04. Its usefulness is the one from the last tip: when in three months' time something stops working, a diff -u network-reference.txt <(ip -brief a) will tell you in a second what has changed, which is the question that really matters in an incident. A server documented while it works is diagnosed in minutes; one with no reference, by guesswork.

Solution 2.

curl http://127.0.0.1:8080/ does not answer the question because the local loopback is a different path: the traffic does not even leave through the network interface, so it works even if the service is bound only to 127.0.0.1. It is the test that generates the most false "well, it works for me" claims.

The definitive command is ss -tulpn, and specifically its Local Address column: 127.0.0.1:8080 says, unambiguously and without depending on any other machine, that the service cannot be accessible from outside. curl against the real IP confirms it empirically, but ss gives you the cause as well as the symptom.

Solution 3. The key detail in the description is that the browser hangs instead of giving an immediate error. "Connection refused" is fast; a long timeout suggests that the packets are being lost with no reply, which points to filtering rather than to a downed service. With that hypothesis, the order:

1. ping srv-tramontana — we already know it responds: there is an IP and a route, layers 1 to 3 ruled out.
2. getent hosts bookings.tramontana.example from the laptop. If it resolves to an IP other than 10.0.2.15, there is the fault and we are done. If it resolves correctly, we continue.
3. nc -zv 10.0.2.15 8080 from the laptop. Distinguishing the three answers is the important part: succeeded means the port is reachable and the problem is at the HTTP layer; Connection refused means nobody is listening or is listening only locally; and a block with no reply until the time runs out is the signature of a firewall silently discarding packets, which is what fits the symptom described.
4. sudo ss -tulpn | grep 8080 on the server, to find out whether it is listening and on which address. If it is 0.0.0.0:8080, the service is fine and the break is somewhere in between.
5. curl -s -o /dev/null -w '%{http_code} %{time_connect} %{time_total}\n' http://10.0.2.15:8080/ from the server. If it responds 200 and nothing arrives from outside, it is confirmed that the problem is between the two machines.
6. Compare the VM's network view with the reference one from exercise 1.

The most likely conclusion given the symptom: filtering somewhere along the path, whether the server's firewall or the VM's network configuration. And here is the honest limit of this lesson: the diagnosis goes as far as identifying that the break is one of filtering; correcting it is the material of 06-01 and 06-03. Knowing where your diagnosis ends and saying so clearly is worth more than venturing a solution you cannot justify.

You close Module 3 with the ability to look outside the server and say, with evidence, where the problem is.

• You have the minimum recap — layers, CIDR, gateway, DNS, ports, TCP against UDP — enough to read the output of any of these tools.
• You read the real configuration with ip: -brief a for interfaces, ip r for the default route, -s link for the error counters and ip neigh for the ARP table.
• You use ping knowing that it answers a single question, and that neither a failure proves something is down nor a success proves the service works; you walk the path with traceroute and chase intermittent losses with mtr.
• You understand /etc/nsswitch.conf's resolution order, you know that /etc/hosts beats the DNS, you read the sections of a dig reply and you know the crucial difference between dig and getent hosts.
• You break ss -tulpn down option by option, and above all you read the Local Address column, which distinguishes an accessible service from one bound only to the loopback; you interpret the TCP states without being alarmed by TIME-WAIT.
• You test services with curl — -I, -v, -w '%{http_code}', --resolve — with nc and with wget, and you transfer with scp and rsync over SSH.
• And you have a layered methodology — IP? gateway? DNS? port? service? — that you have applied to a real case until you found an application listening on 127.0.0.1, with the corresponding report for Marta and the honesty to point out what is left uncovered.

Take stock of the whole module. You arrived knowing how to run commands and you leave knowing how to compose them: you have shaped your environment with variables, aliases and history; you have learned to describe sets of files with wildcards and text patterns with regular expressions, being clear about who expands what; you have interrogated a server with find, locate and grep until you found an exposed credential; you have understood where every byte travels with pipes and redirection; you have turned access.log and bookings.csv into reports with sort, uniq, sed and awk; you have diagnosed an application blocked by a backup that was saturating the disk; you have rescheduled that backup with a lock, a priority and logging; and you have just located a service listening on the wrong interface. That is exactly what the Unix philosophy of Module 1 promised, now in your hands.

And you have also run into the same wall several times. The crontab line in the last exercise accumulated escapes until it became unreadable. The billing report's pipeline no longer fitted on one screen. Every procedure you have designed — check before deleting, copy before editing, verify the result — depended on you remembering to do it, step by step, without making a mistake, at three in the morning. In Module 4: Shell Scripting that wall disappears. You will learn to save your procedures in named files, with variables, arguments and control structures; to write reusable functions; to debug with set -x and to armour your scripts with set -euo pipefail and serious error handling; and to get as far as production scripts, including the backup script you have been promised for three lessons. Everything you have learned here is the vocabulary; Module 4 is the grammar that lets you write with it. Update your VM snapshot and I will see you there.