kitestacks-homelab/homelab-mastery/build-guide/without-ai/05-networking.md

Without AI — Part 5: Networking

Track: Advanced (No AI)
Time for this section: 1–2 weeks

Networking is the hardest part to learn and the most important. Every problem in this homelab ultimately involves a packet trying to get somewhere. If you understand how packets travel, you can debug anything.

---

IP Addresses

Every device on a network has an IP address — a number that identifies it.

IPv4: Four octets (0–255) separated by dots: 192.168.1.205

Classes of addresses:

Range	Who Owns It	Used For
10.0.0.0/8	Private	Corporate networks, VPNs
172.16.0.0/12	Private	Docker bridge networks
192.168.0.0/16	Private	Home networks (your router)
100.64.0.0/10	Shared	Tailscale uses this range
Everything else	Public	Routable on the internet

Private addresses are not routable on the internet. Your home router uses NAT (Network Address Translation) to let private-addressed devices reach the internet.

---

Subnetting and CIDR Notation

CIDR (Classless Inter-Domain Routing) notation describes a range of IP addresses:

192.168.1.0/24
              │
              └── prefix length: how many bits are fixed

An IPv4 address is 32 bits. A /24 means the first 24 bits are fixed (the network), leaving 8 bits for hosts. 2^8 = 256 addresses, minus network (.0) and broadcast (.255) = 254 usable host addresses.

CIDR	Addresses	Usable	Example
/32	1	1	A single IP
/31	2	2	Point-to-point link
/30	4	2	Small link
/29	8	6	Small subnet
/28	16	14
/27	32	30
/26	64	62
/25	128	126
/24	256	254	Typical home/office LAN
/16	65,536	65,534	Large network
/12	1,048,576	—	Docker range: 172.16.0.0/12
/8	16,777,216	—	10.x.x.x range

Subnetting practice: Calculating the host range of 172.17.0.0/16:

• Fixed: 172.17 (first 16 bits)
• Variable: last 16 bits
• Host range: 172.17.0.1 to 172.17.255.254
• This covers all of 172.17.x.x

Why /12 covers all Docker networks: 172.16.0.0/12 covers 172.16.0.0 through 172.31.255.255. Docker creates bridge networks in the 172.17.x.x, 172.18.x.x, etc. ranges. All of those are inside 172.16.0.0/12 — so one ufw rule covers all Docker bridges.

---

Ports

A port is a 16-bit number (0–65535) that identifies which service on a host should handle a connection.

IP address = the building
Port       = the apartment number

Well-known ports (0–1023):

Port	Protocol	Service
22	TCP	SSH
25	TCP	SMTP (email sending)
53	UDP/TCP	DNS
80	TCP	HTTP
443	TCP	HTTPS
5432	TCP	PostgreSQL
6379	TCP	Redis

Ephemeral ports (49152–65535): OS assigns these randomly for outbound connections.

In Docker:

ports:
  - "9100:9100"   # host:container — both the same number

Container port 9100 is mapped to host port 9100. External systems connect to the host IP on port 9100. Internally, containers on the Docker network use the container port directly.

---

DNS (Domain Name System)

DNS is a distributed database that maps names to IP addresses.

The hierarchy:

. (root)
└── com
    └── kitestacks
        ├── www      →  Cloudflare anycast IP
        ├── auth     →  Cloudflare anycast IP
        └── grafana  →  Cloudflare anycast IP

Resolution process for grafana.kitestacks.com:

1. Browser checks local cache — not found
2. Browser asks OS resolver (usually 127.0.0.53)
3. OS asks the configured DNS server (your home router, or 8.8.8.8)
4. Resolver asks root nameservers: "who handles .com?"
5. Root says: "Ask Verisign's servers"
6. Resolver asks Verisign: "who handles kitestacks.com?"
7. Verisign says: "Ask Cloudflare's nameservers (vera.ns.cloudflare.com)"
8. Resolver asks Cloudflare: "what is grafana.kitestacks.com?"
9. Cloudflare returns: "Cloudflare's anycast IP: 104.x.x.x"
10. Browser connects to 104.x.x.x on port 443

Internal Docker DNS: Inside the kitestacks Docker network, Docker runs a DNS server at 127.0.0.11. When cloudflared resolves grafana, Docker DNS returns the container's bridge IP.

# Check what an external name resolves to
dig grafana.kitestacks.com

# Check DNS from inside a container
docker exec cloudflared nslookup grafana
docker exec cloudflared cat /etc/resolv.conf   # Shows the DNS server: 127.0.0.11

---

HTTP and HTTPS

HTTP: Plain text request/response protocol. Anyone who can see the traffic can read it.

GET /api/health HTTP/1.1
Host: grafana.kitestacks.com
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json

{"ok": true}

HTTPS: HTTP inside a TLS-encrypted tunnel. The connection is encrypted from client to Cloudflare's edge. Between Cloudflare and your containers (inside Docker network), it is plain HTTP — this is fine because that traffic never leaves the host.

TLS handshake (simplified):

1. Client says "hello, I support these cipher suites"
2. Server sends its certificate (proves it is kitestacks.com)
3. Client verifies certificate against trusted Certificate Authorities
4. Both sides agree on encryption keys (Diffie-Hellman key exchange)
5. Encrypted connection established
6. HTTP requests flow inside this encrypted tunnel

In this homelab, Cloudflare handles TLS entirely. Your containers never see TLS.

---

Cloudflare Tunnel — Technical Details

What cloudflared actually does:

# Watch cloudflared connect
docker logs cloudflared -f
# You see: "Connection established" connIndex=0 location=ORD
# ORD = Chicago data center (or nearest Cloudflare POP to you)

cloudflared establishes persistent multiplexed HTTP/2 connections to Cloudflare's edge network. When a request comes in:

Internet user → Cloudflare edge → tunnel (HTTP/2 multiplexed) → cloudflared
                                                                       ↓
cloudflared reads Ingress rules from Cloudflare API:
  grafana.kitestacks.com → http://grafana:3000

cloudflared → Docker DNS → grafana container IP → sends request

The tunnel connection uses QUIC (UDP-based) when possible, falls back to HTTPS/TCP.

Active-Active with two connectors: Each connector registers separately. Cloudflare maintains a list of active connectors. Incoming requests are distributed across connectors by Cloudflare — no configuration needed on your end. If one connector drops, the others take all traffic within seconds.

---

Tailscale — WireGuard Under the Hood

Tailscale is a managed WireGuard VPN. Understanding WireGuard explains Tailscale.

WireGuard:

• Modern VPN protocol, designed in 2016
• Uses UDP (faster than TCP-based VPNs like OpenVPN)
• Cryptography: Curve25519 key exchange, ChaCha20-Poly1305 encryption
• Each peer has a public/private key pair (like SSH keys)
• Configured via static peer lists with IP allowances

The NAT problem: Home machines are behind NAT. Their public IP is the router's IP, not their own. Two NAT-ed machines cannot easily make direct connections.

Tailscale's solution — UDP hole punching:

1. Both machines connect to Tailscale's coordination server (DERP)
2. Tailscale orchestrates a "hole punch": both machines send packets to each other simultaneously, which opens NAT mappings on both routers
3. Direct WireGuard connection established peer-to-peer
4. Tailscale coordination servers are no longer involved in the data path

# Check Tailscale status
tailscale status

# See your device's Tailscale IP
tailscale ip -4

# Check connectivity to kscloud1
tailscale ping 100.123.x.x

# See if connection is direct or via relay
tailscale status --json | python3 -m json.tool | grep -A5 "kscloud1"

Why Tailscale IPs are stable: Each device's 100.x.x.x IP is tied to its machine identity in Tailscale's database. It does not change when you move networks or reconnect.

---

Firewalls (ufw)

ufw (Uncomplicated Firewall) is a frontend for iptables/nftables.

kscloud1's firewall configuration:

# View current rules
sudo ufw status verbose

# Default policies
sudo ufw default deny incoming    # Block all inbound by default
sudo ufw default allow outgoing   # Allow all outbound

# Allow specific services
sudo ufw allow ssh                 # Allow SSH (port 22)
sudo ufw allow from 172.16.0.0/12 to any port 8000 proto tcp  # Docker → metrics API

# Why 172.16.0.0/12 and not just the specific Docker subnet?
# Docker creates a new bridge network with a random 172.x subnet for each network.
# /12 covers ALL possible Docker subnets so this rule always works.

The ufw/Docker conflict: Docker modifies iptables rules directly, bypassing ufw. This means Docker's port mappings (-p 9100:9100) are accessible regardless of ufw rules. Only services running in network_mode: host are controlled by ufw.

kscloud1's metrics API uses network_mode: host, so it needs an explicit ufw allow rule for Docker containers to reach it.

---

Reverse Proxies

A reverse proxy receives requests on behalf of backend services:

Client → Reverse Proxy → Backend A
                      → Backend B
                      → Backend C

In this homelab:

• Cloudflare + cloudflared — the primary reverse proxy routing by hostname
• nginx (homepage container) — secondary proxy forwarding /api/* to metrics API

nginx config that proxies API calls:

location /api/ {
    proxy_pass http://host.docker.internal:8000/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

host.docker.internal resolves to the host machine's IP from inside a Docker container. This lets the nginx container reach the metrics API running in network_mode: host.

---

Diagnosing Network Problems

"I can't reach the service from outside"

# Is cloudflared running and connected?
docker logs cloudflared | tail -20

# Is the target container running and on the right network?
docker inspect homepage --format '{{range .NetworkSettings.Networks}}{{println .}}{{end}}'

# Can cloudflared reach the container?
docker exec cloudflared curl -s http://homepage:3000

"Two containers can't talk to each other"

# Are they on the same network?
docker network inspect kitestacks | grep -A5 "Containers"

# DNS resolution working?
docker exec service-a nslookup service-b

# Is the target port open inside the container?
docker exec service-b ss -tlnp

"The database won't accept connections"

# Is Postgres listening?
docker exec authentik-postgres ss -tlnp | grep 5432

# From another container, can we reach it?
docker exec authentik nc -zv authentik-postgres 5432

# Is it bound to the right interface on kscloud1?
docker exec authentik-postgres ss -tlnp | grep 5432
# Should show: *:5432 or 100.123.x.x:5432, not 127.0.0.1:5432

---

Next: Part 6 — Full Build