Debugging network problems

The core idea

"It can't connect" has a small number of causes, and you can rule them out in order: does the name resolve, is anything listening, can packets get there, and is the app using the right address? Work from the outside in and stop guessing.

The toolbox

Most images don't ship dig, curl, nc or ss, and distroless ones don't even have a shell. Bring your own: nicolaka/netshoot is an image packed with network tools. Run it on the same network as the thing you're debugging, or inside its network namespace:

# On a network, as a neighbour
docker run --rm -it --network notes-backend nicolaka/netshoot

# Inside another container's network: same IP, same localhost, same ports
docker run --rm -it --network container:notes nicolaka/netshoot

The checklist

1. Are they on the same network?

$ docker network inspect notes-frontend -f '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
notes 172.21.0.3/16
notes-proxy 172.21.0.2/16

If the caller isn't listed next to the target, nothing else matters. A container started without --network is on the default bridge and can't resolve anyone:

$ docker run --rm alpine wget -qO- -T 3 http://notes:3000/healthz
wget: bad address 'notes:3000'

2. Does the name resolve, from the caller's side?

$ docker run --rm --network notes-frontend nicolaka/netshoot dig +short notes notes-db
172.21.0.3

notes resolves; notes-db returns nothing because it's only on notes-backend. That's by design here, but it's the answer you'd be looking for.

3. Is anything listening, on the right address?

$ docker run --rm --network container:notes nicolaka/netshoot ss -tlnp
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      4096      127.0.0.11:46567      0.0.0.0:*
LISTEN 0      511                *:3000             *:*

*:3000 means the API listens on all interfaces. If you saw 127.0.0.1:3000 instead, other containers could never reach it: the bind-address bug from 03.02 Publishing ports. (127.0.0.11 is Docker's DNS server.)

4. Does the port answer?

$ docker run --rm --network notes-backend nicolaka/netshoot sh -c 'nc -zv -w 2 notes-db 5432; nc -zv -w 2 notes-db 5433'
Connection to notes-db (172.20.0.2) 5432 port [tcp/postgresql] succeeded!
nc: connect to notes-db (172.20.0.2) port 5433 (tcp) failed: Connection refused

"Connection refused" means the host is reachable but nothing listens on that port. A timeout means packets aren't getting there at all: a different network, or a firewall.

5. Is the app actually using that address? Read its config and logs. The most common culprit is in the next section.

The five usual suspects

The localhost one deserves its own example. Inside a container, localhost is that container:

$ docker run --rm --network notes-backend postgres:18 psql postgres://notes:notes@localhost:5432/notes -c 'select 1'
psql: error: connection to server at "localhost" (::1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?

The database is running fine, just in a different container. postgres://notes:notes@notes-db:5432/notes works.

Seeing the request itself

When the connection works but the response is wrong, look at the HTTP exchange from the caller's side:

$ docker exec notes-proxy wget -S -qO- -T 3 http://notes:3000/healthz
HTTP/1.1 200 OK
content-type: application/json
...
{"status":"ok"}

Output trimmed. From netshoot, curl -v gives the same view with more detail. For anything deeper, netshoot also has tcpdump: docker run --rm --network container:notes nicolaka/netshoot tcpdump -i eth0 port 5432 shows every packet the API sends to Postgres.

Recap

Interview questions