So far you have seen containers that work and containers that fail obviously. In real work, the most frequent situation is a third one: a container that does not work and does not say why. It is Up 3 hours but it does not respond. It exits with code 1 and disappears before you can look at it. It gets marked unhealthy with no further explanation. And there you are, with twenty tabs open, trying things at random.

This lesson gives you a method and the five tools that support it: docker logs, docker exec, docker inspect, docker stats and docker events. You are going to learn to read them thoroughly, to get into minimal images that do not have so much as ping, and to extract exactly the piece of data you need from inspect's JSON. And you are going to apply it to three real Aurora Libros breakdowns, among them the one that has been waiting for you for three lessons: why aurora-api cannot find aurora-db even though both are running on the same machine. By the end of this lesson you will know the exact answer.

1. A method before a command
2. docker logs in depth
3. The golden rule: stdout and stderr
4. docker exec: getting into the container
5. Debugging minimal images with no tools
6. docker inspect: the container's complete JSON
7. docker top: the processes inside
8. docker stats: real-time consumption
9. docker events: the daemon's stream
10. Case A: the container exits with code 1
11. Case B: ECONNREFUSED on /books
12. Case C: the healthcheck stuck on unhealthy

The difference between someone who debugs fast and someone who flails around is not the commands, but the order in which the questions get asked:

The four questions, in this order and without skipping any:

The most common mistake is to start at 4 —getting into the container to look around— when the answer was in 2. And the second most common mistake is never doing 3, spending hours convinced you passed an environment variable that in fact had a typo.

The options that actually get used:

Combinations that solve real problems:

Three important details:

• docker logs works with the container stopped. The logs live on the host and survive docker stop. They only disappear with docker rm. That is why --rm is your enemy while debugging.
• --tail without -f is the fastest way to see the end of a 100,000-line log without dumping the whole thing to your terminal.
• With -f, Ctrl+C only interrupts the viewer: it does not touch the container. Unlike docker attach, here Ctrl+C is completely safe.

docker logs shows exactly two things: the standard output and the standard error of PID 1. Not one thing more.

The third line does not appear anywhere, because it was written to a file inside the container. You can separate the two streams with the shell's standard redirection:

And from that comes the rule that governs all container operations:

An application in a container writes its logs to standard output and standard error. Never to a file.

The reasons:

Your server.js has been doing this right since lesson 01-07: it uses console.log and console.error, which in Node write to stdout and stderr respectively.

And a practical check for when docker logs is suspiciously empty:

If those descriptors pointed to a file instead of to the console, there would be your explanation.

A note on scope: where those logs are really stored, how to change the logging driver, how to rotate them and how to ship them to a centralized system is the content of lesson 05-06. Here we stop at reading them.

You already know from lesson 03-01 that exec launches a new process inside the container's namespaces, and that this is why leaving it does not affect the service. Its options:

If you get it wrong, the error is unmistakable:

A trick that almost always works:

Your aurora-api runs with USER node since lesson 02-04. That is excellent for production and a nuisance for debugging:

Three things to learn from this output:

1. -u root gives you privileges inside the container even if the image declares another user. The USER restriction protects you from the code running inside, not from whoever controls the Docker daemon.
2. Installing tools with apk add inside a running container is legitimate for debugging and absolutely forbidden as a way of "fixing" anything: the moment you recreate the container, that curl is gone. What gets fixed, gets fixed in the Dockerfile.
3. And there is the diagnosis handed to you on a plate: errorDb: getaddrinfo ENOTFOUND aurora-db. The /health endpoint you wrote in lesson 01-07 is doing exactly the job you designed it for.

Look at /etc/hosts: there are three entries and none of them is called aurora-db. The command getent hosts aurora-db did not even print anything (it returned code 2, "not found"). Keep that fact: it is half the answer to case B.

A well-optimized image ships no curl, no ping, no netstat, no dig. And distroless images or ones built FROM scratch do not even ship a shell. That is excellent for security and size (lesson 05-04) and maddening when something fails.

The modern solution: an ephemeral container loaded with tools that shares the sick container's namespaces.

What each option does:

And now, from inside that container, you diagnose api-debug's network as if you were in it:

Four conclusions in four commands, without installing anything in the production image:

1. ss -tlnp confirms the API is listening on 3000, on 0.0.0.0 (good: not on 127.0.0.1, which would be unreachable from outside).
2. ps -ef shows the neighboring container's node server.js thanks to --pid container:, and also that it runs as UID 1000.
3. nslookup aurora-db answers NXDOMAIN: Docker's internal DNS server (127.0.0.11) exists and responds, but it does not know that name.
4. curl works even though the API image has no curl, because netshoot provides it.

That NXDOMAIN is the definitive proof. We will come back to it in case B.

Docker Desktop includes a built-in version of this idea:

It opens a shell with a set of tools mounted over the container without modifying it: it works even on distroless images with no shell, and when you leave, no trace is left. It requires a Pro subscription or above; the netshoot trick is free, works on any Docker and is worth knowing anyway.

docker inspect returns everything Docker knows about a container: around 200 lines of JSON.

The six blocks that matter:

Reading 200 lines of JSON is not debugging. What you do is extract the specific piece of data, with --format or with jq:

A complete example on the fleet:

Two observations that will be decisive in five minutes: both are on the bridge network and each has its own IP.

With jq you can run richer queries over the raw JSON:

A security note, and it is a serious one: docker inspect shows passwords in the clear. Anyone with access to the Docker daemon can read every environment variable of every container. That is why environment variables are not a secrets mechanism; secret managers are, and they are covered in lesson 05-03.

Two peculiarities make it special:

• The PIDs are the host's, not the ones inside the container. docker exec aurora-db ps would show that same postgres as PID 1; here it is 24817. It is the double numbering of PID namespaces.
• It does not require the image to have ps. The command is run by the daemon on the host. That is why it works even on distroless images.

It is good for quickly answering: how many processes are there? Are there zombie processes? Is the application spawning children it should not?

Without --no-stream, the view refreshes continuously like a top. Column by column:

That LIMIT of 7.628 GiB on both rows is the alarm bell you will deal with in lesson 03-07: neither container has a memory limit, so either of them can consume all the machine's RAM.

Useful formats:

docker events is the record of everything the daemon does. Without --since, it stays listening live. It is the tool that answers questions no other one can:

The most useful events: create, start, die (with its exitCode), kill, oom, health_status, restart, destroy, and the network ones connect/disconnect. An oom in this list is confirmation that it was the kernel and not you (lesson 03-07).

Symptom: a colleague tells you the API "does not start on his machine". You apply the method.

Question 1: is it running?

Code 1: from lesson 03-02's table, that means the application started and failed on its own. It is not Docker (that would be 125), nor a non-existent executable (127), nor a signal (>128). So we go to the logs.

Question 2: what do the logs say?

Solved in two commands and fifteen seconds. The file is called server.js, not serve.js. Where does that name come from?

Question 3: how is it configured?

There it is: the image's CMD (["server.js"]) was overridden from the command line by the loose argument serve.js, exactly as you learned in lesson 03-01. The image is perfect; the mistake was in the docker run.

A variant of the same symptom worth recognizing, because the diagnosis is completely different:

The docker ps -a command returns nothing: the container was never even created. With a 125 there are no logs to look at and no container to inspect; the error is in your command line. It is the first fork in the decision tree and it saves a lot of wasted time.

Here is the case you have been dragging along since module 2. Let's solve it with the complete method.

Question 1: is it running?

All three are running. It is not a startup problem.

Question 2: what do the logs say?

The message is clear: getaddrinfo is the name resolution function, and ENOTFOUND means the name aurora-db could not be translated into any IP address. It is not a refused connection: it is that the API does not know where to connect.

Question 3: how is it configured?

And here comes the surprise that makes the case interesting: all three are on the same network, the default bridge, with IPs from the same 172.17.0.0/16 range. It is not an isolation problem.

Check the variables, to rule them out:

Correct. The configuration is the one you wanted.

Question 4: what is happening inside?

Docker's internal DNS (127.0.0.11) is configured, but it does not resolve aurora-db. Confirm it with real tools, using the trick from section 5:

This is the definitive diagnosis, and it is two opposite facts in the same output:

In other words: aurora-api can talk to aurora-db; it simply does not know its name. The problem was never a firewall, or ports, or PostgreSQL: it is name resolution.

And the cause is a very specific Docker characteristic: the default bridge network has no internal DNS between containers. Only user-defined networks have it. Nobody had told you until now because it is exactly the subject of the next lesson.

Could you not just put the IP in directly and be done sooner?

You could, and it would be a mistake. Those IPs are assigned by Docker in start-up order: reboot the machine, change the order of the containers and 172.17.0.2 will be a different one. Writing IPs by hand is building a castle on sand. The correct solution —a network of your own where names work— is the first hands-on part of lesson 03-05, and now you know exactly why you need it.

The module's mystery is closed. All that is left is to apply the solution.

Symptom: docker ps shows Up 40 seconds (unhealthy) on aurora-api. What does that mean exactly, and where does that verdict come from?

The complete history is in .State.Health.Log, an array with the last five attempts:

Read it carefully, because it tells the whole story:

That 503 is no accident: you wrote it yourself in lesson 01-07. The /health endpoint returns 200 only if the database and the cache respond, and 503 in any other case:

And now the nuance that separates a real unhealthy from a false positive:

The container is healthy; the service is not. The node process is alive, listening on 3000 and responding in 173 milliseconds. What is broken are its dependencies. The HEALTHCHECK is doing exactly what you asked it to: reporting that this container is not in a fit state to serve traffic.

The four possible health states:

And the detail that surprises everyone: unhealthy does nothing on its own. Docker Engine does not restart the container, does not take it out of rotation and does not notify anyone; it just marks it and emits a health_status event. Who acts on that information is an orchestrator (Swarm in lesson 06-03, Kubernetes in 06-05) or you, looking at docker ps. Not even the --restart policy reacts to an unhealthy, and that nuance is explained in lesson 03-07.

Leave aurora-api as it is: in the next lesson it will turn healthy on its own.

• Starting by getting into the container. 70% of breakdowns show up in docker logs in ten seconds. Follow the order: state → logs → configuration → inside.
• Debugging with --rm. If the container is deleted when it dies, there are no logs, no inspect and no autopsy. During an investigation, no --rm.
• Looking for the logs of an application that writes to a file. An empty docker logs does not mean "nothing is happening": check where the process is really writing.
• Installing tools in the container "to fix it". They are lost when you recreate it. To diagnose, an ephemeral container with netshoot; to fix, the Dockerfile.
• Confusing ECONNREFUSED with ENOTFOUND. The first says "the name resolved but nobody is listening there"; the second, "I do not know who that is". They are two different breakdowns and they lead to different places.
• Reading docker stats with no limits configured. The LIMIT column shows all the host's RAM and the MEM % is deceptively reassuring.
• Believing unhealthy restarts anything. Docker Engine only marks it. Without an orchestrator, nobody acts.
• Dumping the whole docker inspect into the terminal. That is 200 lines of JSON. Use --format or jq and go straight to the fact.
• Tip: keep aliases for the inspect queries you repeat. For example dip() { docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' "$1"; }.
• Tip: docker events in a second terminal while you reproduce the failure. Seeing the die with its exitCode at the exact instant is worth more than ten guesses.

Launch this container, which will fail on purpose, and carry out the complete investigation without running it again:

Answer with specific commands: what state is it in and what code did it exit with? Does that code point at Docker or at the application? What did it write to stdout and what to stderr, separately? What environment variables did it have configured? What was its effective command? How long was it alive (work it out with StartedAt and FinishedAt)?

Start a web-silent container with nginx:alpine without publishing any port. Then, without installing anything inside it and without recreating it:

1. Find out its IP address.
2. Check from an ephemeral netshoot container that Nginx responds on its port 80.
3. Prove that from the host, with curl localhost:80, it does not respond, and explain why that does not contradict the previous point.
4. Find out what processes run inside without using docker exec.

Create an aurora-api:fake-healthy image from auroralibros/aurora-api:1.2.0 that replaces the HEALTHCHECK with one that simply checks the process exists (CMD pgrep node || exit 1). Start it with no database and no cache and compare, with commands, the health state it reports against that of auroralibros/aurora-api:1.2.0 under the same conditions. Then answer: which of the two healthchecks is "better"? In what specific situation would the second one have saved you from an incident, and in which one would it have given you a false alarm?

Solution to exercise 1

Code 78 is in the 1–124 range, so it is the application's, not Docker's. A 125 would have meant an error in docker run itself, a 127 a non-existent executable and a 137 a signal. Here, the program decided to exit with that number, so the explanation is in its code and in its logs.

Separating the streams is what gives the diagnosis: stdout has informational noise and stderr has the real cause. In a 500-line mixed-up log, that filter is the difference between finding it and not finding it.

Confirmed: MODE was defined and API_KEY is nowhere to be seen, which is exactly what stderr was complaining about.

It lived 2.17 seconds, consistent with the sleep 2 in the command. That calculation is more useful than it looks: a container that lives for milliseconds usually fails at startup; one that lives for hours and then dies points to a memory leak or an external event.

Solution to exercise 2

2. Checking that it responds, from netshoot:

Nginx works perfectly. Another way, sharing its network stack directly:

3. From the host:

There is no contradiction at all: the container listens on port 80 of its own network stack, but since no port was published with -p, there is no rule forwarding the host's port 80 to it. The connectivity exists inside Docker's network (where netshoot lives) and does not exist from the host. It is exactly the distinction between "publishing a port" and "two containers talking to each other" that gets formalized in lesson 03-05.

4. Processes without docker exec:

docker top is run by the daemon on the host, so it does not need to get into the container or for the image to have ps. And as a bonus you can see Nginx's good practice: the master runs as root and the workers as user 101.

Solution to exercise 3

Two containers identical in everything except their health check, and opposite verdicts. The one on the left says it is healthy; neither of the two can serve a single book.

The fake healthcheck answers 0 because pgrep node finds the process. And it is entirely right: the process is alive. It is simply measuring what does not matter.

The answers:

• The one in auroralibros/aurora-api:1.2.0 is better in this scenario, because it checks the real ability to provide service (an end-to-end HTTP request, which in turn queries the database and the cache) instead of the mere existence of a process.
• When the second one (pgrep) would have saved you: when the outage is in a dependency and you do not want all your replicas to declare themselves sick at once. If PostgreSQL goes down for 30 seconds, with the strict healthcheck the API's ten replicas all switch to unhealthy simultaneously, the orchestrator takes them out of rotation and you end up with a total outage instead of a degraded service still serving cached content. It is the well-known domino effect of healthchecks that are too deep.
• When it would have given you a false alarm... or worse, none at all: when the Node process is alive but its event loop is blocked, its connection pool exhausted or it answers 500 to everything. pgrep would say "healthy" indefinitely while users see errors. That scenario is literally the one that motivated HEALTHCHECK in lesson 02-04.

The professional solution, which you were already sketching in exercise 3 of lesson 02-04, is to separate two endpoints: one for liveness (/health/live, shallow, for deciding on restarts) and another for readiness (/health, deep, for deciding whether it receives traffic). It is the liveness/readiness distinction developed in lesson 06-05.

You have a method, and that is worth more than the list of commands: is it running? → what do the logs say? → how is it configured? → what is happening inside?, in that order and without skipping steps. You know that a Created points to a non-existent executable, that a 125 means the container was never created and there is nothing to inspect, and that a code between 1 and 124 sends you straight to docker logs.

You handle docker logs with its time windows, its --tail and its -f that you can cut with Ctrl+C without fear, and you have the golden rule engraved: containers log to stdout and stderr, never to a file, and you know how to separate the two streams with a redirection so the error appears on its own. You get in with docker exec choosing the right shell for the base, you know -u root gives you privileges even if the image declares USER node, and —most valuable of all— you know how to debug minimal images with no tools by launching an ephemeral netshoot container that shares the sick one's network and process namespaces, without dirtying the production image.

From docker inspect's JSON you extract the exact fact with Go templates and jq, with a table of queries for IP, networks, ports, mounts, environment, health, exit code and restarts, and you know that same command shows passwords in the clear, which is why environment variables are not a secrets mechanism. With docker top you see the processes from the host even if the image has no ps, with docker stats you read the seven columns —and you have discovered your containers have no memory limit at all— and with docker events you reconstruct what happened and when.

And you have closed the investigation you had been dragging along since module 2. aurora-api, aurora-db and aurora-cache are on the same network and with full connectivity between them: nc -zv 172.17.0.2 5432 answers succeeded. Only one thing fails, and now you can name it precisely: nslookup aurora-db returns NXDOMAIN, because the default bridge network has no internal DNS between containers. It is not a firewall, or a port, or PostgreSQL: it is that the API does not know its database's name.

All that is left is to apply the solution, and it is shorter than the diagnosis. In the next lesson, Docker Networking, you will see why each container has its own network stack and its own localhost, you will compare the bridge, host and none drivers, and you will understand the decisive difference between the default bridge and a user-defined network, where internal DNS resolves container names automatically. You will create aurora-net, recreate the three services inside it, and finally run that curl http://localhost:3000/books that has been waiting six lessons to give you back El jardín de senderos que se bifurcan, Rayuela and the other six titles. Then you will add aurora-web as a reverse proxy, and the Aurora Libros platform will be truly alive.