Under the hood: namespaces and cgroups · Docker

The core idea

There's no such thing as a "container" inside the Linux kernel. What Docker starts is an ordinary process, with three things set up around it:

Namespaces control what the process can see: its own process list, network, hostname and mount points.
Cgroups (control groups) control what it can use: how much memory and CPU, how many processes.
A root filesystem: the image's files become the whole disk the process sees, starting at /.

Docker also takes away some of root's powers (Linux capabilities) and blocks a few risky requests to the kernel (seccomp), but namespaces and cgroups are the heart of it.

Namespaces: a private view

Each namespace type wraps one part of the system. A process inside sees only what's in its namespace:

Namespace	Isolates	Inside the container you see…
`pid`	process IDs	your app as PID 1, and only its own processes
`net`	network interfaces, ports, routes	its own `eth0` and `localhost`
`mnt`	mount points	the image's filesystem as `/`
`uts`	hostname	a hostname of its own
`ipc`	shared memory, message queues	only its own
`user`	user and group IDs	root inside can map to a normal user outside

Docker gives each container its own pid, net, mnt, uts and ipc namespaces by default. User namespaces are opt-in, and are how Docker's rootless mode runs containers without root on the host.

The important bit: from the host, the container's processes are still right there, with normal PIDs. Namespaces hide the host from the container, not the container from the host.

See it yourself: one process, two views

CheckpointFind a container's process from both sides

Start nginx, then list its processes from inside and from the host.

❯terminal

$ docker run -d --name web nginx:alpine
$ docker exec web ps
PID   USER     TIME  COMMAND
  1 root      0:00 nginx: master process nginx -g daemon off;
 30 nginx     0:00 nginx: worker process
 31 nginx     0:00 nginx: worker process
 ...
 40 root      0:00 ps
$ docker top web
UID      PID      PPID     C    STIME    TTY    TIME       CMD
root     17525    17503    0    20:18    ?      00:00:00   nginx: master process nginx -g daemon off;
statd    17564    17525    0    20:18    ?      00:00:00   nginx: worker process
...

Output trimmed; your PIDs will differ. Inside, nginx is PID 1. On the host, the same process has a normal PID like 17525.

Notice the workers run as nginx inside but show as statd from outside. Both are the same numeric user ID, 101. The container and the host each translate that number using their own /etc/passwd, and on the host 101 happens to be named statd. User names are just labels; the kernel only knows the numbers.

I ran it and my output matches

STUCK? ▸

docker top asks the host, so it shows host PIDs. docker exec ... ps runs inside the container's PID namespace, so it shows container PIDs, starting from 1.

You can even build a namespace by hand. unshare runs a program in brand-new namespaces. Inside a privileged throwaway container (so this works on any OS), ask it for a new PID namespace and run ps there:

❯terminal

$ docker run --rm --privileged ubuntu unshare --pid --fork --mount-proc ps
PID TTY          TIME CMD
  1 ?        00:00:00 ps

ps is PID 1 and can't see a single other process, not even the container it's running in. That's the same kernel feature Docker uses, minus the convenience.

Cgroups: a budget

Namespaces don't limit anything. A process that can only see itself can still eat all the host's memory. Cgroups fix that: the kernel accounts for every cgroup's memory, CPU and process count, and enforces the limits you set.

limits

docker run -d --name capped --memory 64m --cpus 0.5 nginx:alpine
docker stats --no-stream capped
docker exec capped cat /sys/fs/cgroup/memory.max

❯terminal

$ docker stats --no-stream capped
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O     PIDS
9368b6d7002c   capped    0.00%     8.664MiB / 64MiB    13.54%    652B / 126B   0B / 8.19kB   11
$ docker exec capped cat /sys/fs/cgroup/memory.max
67108864

docker stats shows the 64 MiB limit, and 67108864 is the same limit in bytes, read straight from the cgroup files the kernel uses. (That path is for cgroup v2, which current Linux distributions and Docker Desktop use.)

When a container hits its memory limit and can't reclaim any, the kernel's OOM killer ends a process in it. If that's the main process, the container exits with code 137 (128 + signal 9, SIGKILL). Here's a Node one-liner that allocates memory forever, run with a 32 MB limit:

❯terminal

$ docker run --name hog --memory 32m node:22-slim node -e 'const a=[];while(true)a.push(Buffer.alloc(1e6,1))'
$ docker inspect --format '{{.State.Status}} {{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' hog
exited 137 OOMKilled=true

You'll tune limits properly in 03.06 CPU and memory limits.

Predict first

You start a container with --memory 64m. What actually enforces that limit?

Commit to an answer before you read on.

PID 1 is special

Because your app is PID 1 in its namespace, it inherits a job normally done by init: the kernel won't apply default signal behaviour to PID 1. If your app doesn't explicitly handle SIGTERM, docker stop sends it, nothing happens, and after a grace period Docker gives up and sends SIGKILL. On Docker Engine (what servers run) the grace period is 10 seconds by default; check a container's with docker inspect -f '{{.Config.StopTimeout}}' NAME. The Docker Desktop used to test this course set it to 1 second, so the problem is easy to miss on a laptop.

Two fixes: handle SIGTERM in your app (you should anyway, to shut down cleanly), or run with docker run --init, which puts a tiny init process in front of your app to forward signals.

Recap

Interview questions

MidfreqMust-knowConcept

What is a container, at the operating-system level?

MidfreqGood to knowDebugging

docker stop takes ten seconds for one of your containers on the server, every time. Why, and how do you fix it?