# Without AI — Part 4: Docker Deep Dive **Track:** Advanced (No AI) **Time for this section:** 1–2 weeks Docker is the technology that runs every service in this homelab. Understanding it deeply — not just copying compose files — is what separates someone who can maintain and troubleshoot a homelab from someone who hopes nothing breaks. --- ## What Docker Actually Is Most explanations say "containers are like lightweight VMs." That is wrong and leads to confusion. Here is what a container actually is: **A container is a Linux process with isolation applied.** Two Linux kernel features provide that isolation: **Namespaces** — the container gets its own view of: - Filesystem (it sees `/` but it is a different tree than the host's `/`) - Network interfaces (its own `eth0`, its own IP on the Docker network) - Process list (it can only see its own processes, not the host's) - User IDs (it can be "root" inside without being root on the host) **cgroups (control groups)** — limits how much of the host's resources the container can use: - CPU cores and usage limits - RAM limits - Disk I/O limits - Network bandwidth limits **Result:** No second kernel, no hardware emulation, no hypervisor. The nginx process in your `homepage` container is a regular Linux process on your machine — it just thinks it is alone. --- ## Images vs Containers ``` Image Container ───────────────────────── ───────────────────────────────────────── A recipe A running instance made from the recipe Read-only, immutable Has a writable layer on top of the image Stored in layers One writable layer per container Shared across containers Separate per container Survives container deletion Deleted with the container (unless volume) ``` **Layers:** Docker images are built in layers. Each line in a `Dockerfile` creates a layer. If you update one layer, only that layer is re-downloaded. This is why pulling an update is fast — most layers are already local. ```bash docker image ls # List local images docker image inspect nginx:alpine # See image metadata and layers docker image history nginx:alpine # See how the image was built, layer by layer docker image pull postgres:16-alpine # Download an image explicitly docker image rm nginx:alpine # Remove a local image ``` --- ## Docker Networks — In Depth Docker provides several networking modes: **bridge (default):** Container gets its own virtual network interface with a private IP (172.x.x.x range). Containers on the same bridge network can reach each other by IP or by name (via Docker's built-in DNS). Containers on different bridge networks are isolated. **host:** Container shares the host's network namespace entirely. `--network host` means no isolation — the container sees all host network interfaces and binds directly to host ports. Used for kitestacks-metrics-api so psutil can see real network stats. **none:** No networking at all. Rarely used. ```bash # Create a named bridge network docker network create kitestacks # See all networks docker network ls # Inspect a network — see which containers are connected and their IPs docker network inspect kitestacks # Connect a running container to a network docker network connect kitestacks my-container # Disconnect docker network disconnect kitestacks my-container ``` **The DNS trick:** When two containers are on the same bridge network, Docker runs a DNS server at `127.0.0.11` inside each container. Container names resolve to their internal IPs. This is why `cloudflared` can connect to `http://grafana:3000` — Docker DNS resolves `grafana` to the grafana container's IP. ```bash # Verify DNS works from inside a container docker exec cloudflared nslookup grafana docker exec cloudflared curl -s http://grafana:3000/api/health ``` --- ## Volumes — Persisting Data Containers are ephemeral. When you delete a container, its writable layer is gone. To keep data, you use volumes. **Bind mount:** You choose the path on the host. ```yaml volumes: - ./data:/forgejo-data # host path : container path - /home/kenpat/books:/books:ro # :ro = read-only ``` Data is at `./data` on the host. You can navigate there with `cd`. You can back it up. **Named volume:** Docker manages the path. ```yaml volumes: - uptime-kuma:/app/data volumes: uptime-kuma: # define the named volume ``` Data is at `/var/lib/docker/volumes/uptime-kuma/_data/` on the host (Docker manages this). ```bash docker volume ls # List named volumes docker volume inspect uptime-kuma # See where it is stored docker volume rm uptime-kuma # Delete a volume (and its data!) ``` **Access a named volume from a one-off container:** ```bash docker run --rm -v uptime-kuma:/data alpine ls /data ``` This is the pattern used throughout this homelab to read or modify volumes without stopping the running service (for reads) or after stopping it (for writes). --- ## Docker Compose — The Full Picture Docker Compose reads a YAML file and manages the lifecycle of multiple containers. ```yaml services: forgejo: image: codeberg.org/forgejo/forgejo:latest container_name: forgejo # Fixed name (not random) restart: unless-stopped # Restart on crash or host reboot env_file: .env # Load environment variables from file environment: FORGEJO__server__DOMAIN: gitforge.kitestacks.com # Override one env var volumes: - ./data:/data # Bind mount: ./data on host → /data in container ports: - "127.0.0.1:2222:22" # Bind host 127.0.0.1:2222 to container port 22 (SSH) networks: - kitestacks depends_on: - authentik-postgres # Start this service before forgejo healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 30s timeout: 10s retries: 3 networks: kitestacks: external: true # Use existing network (don't create a new one) ``` **Key fields explained:** `restart: unless-stopped` - `no` — never restart - `always` — always restart, even on manual stop - `on-failure` — restart only if exit code is non-zero - `unless-stopped` — restart on crash or reboot, but not if you manually stopped it `env_file: .env` Reads `KEY=VALUE` pairs from a file. The `.env` file is in `.gitignore` so secrets never get committed to git. Always use this for passwords, tokens, and secrets. `depends_on` Starts services in dependency order. Does NOT wait for a service to be "ready" — just waits for the container to START. If you need to wait for a database to be ready, add a health check and use `condition: service_healthy`. **Common commands:** ```bash docker compose up -d # Start all services in background docker compose down # Stop and remove containers (not volumes) docker compose down -v # Stop, remove containers AND volumes (data loss!) docker compose restart forgejo # Restart one service docker compose pull # Pull latest images docker compose logs -f forgejo # Follow logs for one service docker compose ps # Show service status docker compose exec forgejo bash # Open shell in running service docker compose config # Validate and show merged config ``` --- ## Port Mappings — When to Use Them ```yaml ports: - "3005:3000" # host_port:container_port - "127.0.0.1:3005:3000" # bind to localhost only (not accessible from outside host) - "0.0.0.0:9100:9100" # bind on all interfaces (accessible from outside) ``` **In this homelab, most services do NOT expose host ports** — they only communicate through the Docker network. Cloudflare Tunnel connects directly to the container via the Docker bridge network, so no host ports are needed for public services. The only services that need host ports: - `node-exporter` on kscloud1 (so Prometheus on monk can scrape it via public IP) - `kitestacks-metrics-api` does NOT use ports — it uses `network_mode: host` - `portainer` uses 9443 (HTTPS) --- ## Inspecting and Debugging ```bash # See everything about a container docker inspect forgejo # See just its IP address on each network docker inspect forgejo --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' # See its environment variables (careful — this shows secrets!) docker inspect forgejo --format '{{range .Config.Env}}{{println .}}{{end}}' # See its mounts docker inspect forgejo --format '{{json .Mounts}}' | python3 -m json.tool # See resource usage docker stats # Live, all containers docker stats forgejo --no-stream # One snapshot for one container # See what the container's filesystem looks like docker exec forgejo ls / docker exec forgejo cat /etc/forgejo/app.ini docker exec forgejo find /data -name "*.db" 2>/dev/null ``` --- ## Common Gotchas **Containers share the host's kernel:** If you run an Alpine-based image but your host kernel is too old, some syscalls may not work. Rare but real. **Named volumes are invisible by default:** New developers spend hours wondering where data went after deleting a container. Named volumes survive `docker compose down`. They do NOT survive `docker compose down -v`. **Order vs readiness:** `depends_on` does not mean "wait until ready." A Postgres container starts in milliseconds, but PostgreSQL inside it takes 3–5 seconds to accept connections. Use healthchecks for real readiness checking. **Port conflicts:** Two containers cannot bind the same host port. If you get `Bind for 0.0.0.0:3000 failed: port is already allocated`, something else is already using that host port. **network_mode: host and named networks cannot coexist:** ```yaml network_mode: host # This means the container has NO network isolation # You cannot also add networks: [...] — they conflict ``` --- ## Practice Exercises 1. Pull the `nginx:alpine` image and run it: `docker run -d -p 8080:80 nginx:alpine` Visit `http://localhost:8080`. Then exec into it and find the nginx config. 2. Run two containers (`alpine`) on the same custom network and verify they can ping each other by container name 3. Create a named volume and mount it in two different containers. Write a file from one container and read it from the other 4. Write a `docker-compose.yml` with three services: one nginx, one redis, one alpine that waits for redis to be healthy before starting 5. Use `docker inspect` to find the IP address of your `forgejo` container on the `kitestacks` network. Confirm it matches what Docker DNS resolves. --- **Next:** [Part 5 — Networking](05-networking.md)