sehaj/learn
Docker/07 Production

07.07 · build · 35 min

Project: deploy to a VPS

Take the notes stack to a real server: a hardened production compose file, secrets from files, one deploy script, and rollbacks by image tag.

The brief

The production compose file

project/docker/compose.prod.yaml is the dev file's grown-up sibling. The differences, service by service:

compose.prod.yaml (api)
api:
  image: ${NOTES_API_IMAGE:-ghcr.io/sehajmakkar/notes-api:latest}
  environment:
    DATABASE_URL: postgres://notes@db:5432/notes
    DATABASE_PASSWORD_FILE: /run/secrets/db_password
    REDIS_URL: redis://cache:6379
  secrets: [db_password]
  read_only: true
  cap_drop: [ALL]
  security_opt: ["no-new-privileges:true"]
  restart: unless-stopped
  deploy:
    resources:
      limits:
        memory: 256M
        cpus: "0.5"
  • image: instead of build:, with the tag set by NOTES_API_IMAGE so a deploy picks an exact version.
  • Hardening from 07.01 Run as non-root, the secret file from 07.02 Secrets without leaks, limits from 03.06 CPU and memory limits, restarts from 03.05 Restart policies.
  • The proxy is nginxinc/nginx-unprivileged, read-only with a tmpfs for /tmp, using the same nginx/default.conf.
  • The cache runs Redis as a pure cache: no snapshots, a 64 MB memory cap, least-recently-used eviction.
  • Every service shares one logging block with rotation, defined once as a YAML anchor:
compose.prod.yaml
x-logging: &logging
driver: json-file
options:
  max-size: "10m"
  max-file: "3"

services:
api:
  logging: *logging
  • The backend network is internal: true, so the database and cache can't reach the internet at all:
❯terminal
$ docker compose -f compose.prod.yaml exec cache wget -qO- -T 3 http://example.com
wget: bad address 'example.com'
$ docker compose -f compose.prod.yaml exec api node -e 'fetch("http://example.com").then(r=>console.log(r.status))'
200

The API still can, through frontend.

Every process, non-root

❯terminal
$ for c in notes-prod-proxy-1 notes-prod-api-1 notes-prod-db-1 notes-prod-cache-1; do docker top $c -o pid,user,args | sed -n 2p; done
... statd nginx: master process nginx -g daemon off;
... 1000 node dist/server.js
... 999 postgres
... 999 redis-server *:6379

(statd is just how this host names UID 101, nginx's user: the same effect you saw in 01.03 Under the hood: namespaces and cgroups.)

Prepare the server

On a fresh Ubuntu or Debian VPS, as a user with sudo:

server, once
# Docker Engine and the Compose plugin, from Docker's apt repository (see docs.docker.com/engine/install)
sudo apt-get update && sudo apt-get install -y ca-certificates curl
# ...add Docker's repository as the install docs describe, then:
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Firewall: SSH and HTTP(S) only
sudo ufw allow OpenSSH && sudo ufw allow 80/tcp && sudo ufw allow 443/tcp && sudo ufw enable

# A deploy user that can run Docker (root-equivalent, see 01.05: keep SSH key-only)
sudo adduser --disabled-password deploy && sudo usermod -aG docker deploy

Remember 03.02 Publishing ports: Docker's published ports bypass ufw. That's why only the proxy publishes anything here.

Get the files there

The server needs only four things from the repo, plus a password:

your machine
ssh deploy@YOUR_SERVER 'mkdir -p notes/nginx notes/secrets'
scp project/docker/compose.prod.yaml project/docker/deploy.sh deploy@YOUR_SERVER:notes/
scp project/docker/nginx/default.conf deploy@YOUR_SERVER:notes/nginx/
ssh deploy@YOUR_SERVER 'openssl rand -base64 24 > notes/secrets/db_password.txt && chmod 600 notes/secrets/db_password.txt'

The password is generated on the server and never leaves it. If your GHCR package is private, run docker login ghcr.io on the server once with a token that has read:packages.

Deploy

deploy.sh pulls the images, starts or updates the stack, waits for every healthcheck, and cleans up old images:

deploy.sh
#!/usr/bin/env bash
# Deploy or update the notes stack on the machine Docker points at (see 07.07).
# Usage: ./deploy.sh [image]   e.g. ./deploy.sh ghcr.io/sehajmakkar/notes-api:sha-1a2b3c4
set -euo pipefail
cd "$(dirname "$0")"

export NOTES_API_IMAGE="${1:-ghcr.io/sehajmakkar/notes-api:latest}"

if [ ! -s secrets/db_password.txt ]; then
echo "secrets/db_password.txt is missing. Create it with: openssl rand -base64 24 > secrets/db_password.txt" >&2
exit 1
fi

echo "Deploying $NOTES_API_IMAGE"
docker compose -f compose.prod.yaml pull --ignore-buildable
docker compose -f compose.prod.yaml up -d --wait
docker compose -f compose.prod.yaml ps
docker image prune -f

up --wait is the important part: the command only succeeds once every service with a healthcheck reports healthy, so a broken release fails the deploy instead of silently serving errors. On the server:

❯terminal
$ ssh deploy@YOUR_SERVER
$ cd notes && ./deploy.sh ghcr.io/sehajmakkar/notes-api:sha-f22dfda
Deploying ghcr.io/sehajmakkar/notes-api:sha-f22dfda
...
Container notes-prod-api-1 Healthy
Container notes-prod-cache-1 Healthy
Container notes-prod-db-1 Healthy
Container notes-prod-proxy-1 Healthy
NAME IMAGE SERVICE STATUS
notes-prod-api-1 ghcr.io/sehajmakkar/notes-api:sha-f22dfda api Up 6 seconds (healthy)
...
$ curl -s localhost/notes
[{"id":"74d4627d-...","text":"from production",...}]

Output trimmed. (This run used a local registry in place of GHCR; the commands are identical.) Only the API container was recreated; the database kept running, with its data.

You can also run the deploy from your own machine without logging in to the server, using a Docker context over SSH:

your machine
docker context create notes-vps --docker "host=ssh://deploy@YOUR_SERVER"
docker --context notes-vps ps

Roll back

Every image CI publishes is tagged with its commit (07.06 Project: ship with GitHub Actions). Rolling back is deploying the previous tag:

server
./deploy.sh ghcr.io/sehajmakkar/notes-api:sha-1f76f82

Avoid deploying latest for exactly this reason: you can't roll back to "the latest before this one".

Keep it running

  • Backups: schedule the pg_dump from 04.05 Project: persist Postgres with cron, write it off the server (object storage), and test restores.
  • HTTPS: point a domain at the server and terminate TLS in front of Nginx. Caddy does certificates automatically; certbot works with Nginx. Then publish 443 instead of only 80.
  • Updates: rebuild in CI regularly so images pick up base-image patches (07.03 Scanning images), and run sudo apt-get upgrade on the server.
  • Logs: rotation is on; docker compose -f compose.prod.yaml logs -f api is your first stop.
Predict first

You deploy a release whose API crashes on startup. With deploy.sh, what happens?

Commit to an answer before you read on.
CheckpointProject step 7: deploy to a VPS
  • Prepare a server, copy the four files, generate the password on it.
  • Deploy a sha- tag from GHCR and reach the API through the proxy.
  • Check the hardening: non-root processes, CapEff of zero, no password in docker inspect, no internet from cache.
  • Deploy a second tag, then roll back to the first.
STUCK? ▸

No server yet? Run the whole thing locally: PROXY_PORT=8088 ./deploy.sh with an image from a local registry (02.06 Tags, digests and registries), as this lesson's output was produced. dependency failed to start ... is unhealthy: docker compose -f compose.prod.yaml logs api. Compare files with git diff docker-step-07 -- project/docker.

What you built

Over seven steps the notes API went from a TypeScript file to a scanned, multi-platform image built by CI, running as four hardened containers on a server, deployed and rolled back by tag. That's the whole course: everything you need to containerise, ship and run a real app with Docker.

Interview questions

SeniorfreqMust-knowDesign

Walk me through how you'd deploy a containerised app to a single server.

Report a mistakeEdit this page
☰ Production07.07 · 44/44next →