kitestacks-homelab/homelab-mastery/architecture/decisions.md

# Architecture Decisions — The Why Behind Every Choice

For every technology choice, there was a reason. Understanding the "why" is what separates someone who copied commands from someone who designed a system.

---

## Why Docker Instead of Running Services Directly?

**Problem:** Running 15+ services directly on a Linux host creates dependency hell — different Python versions, conflicting library versions, services affecting each other.

**Options considered:**
- Bare metal: install each app directly on the OS
- Virtual machines: one VM per service
- Docker containers: isolated processes with their own dependencies

**Decision:** Docker

**Why:**
- Each container has its own filesystem, dependencies, and runtime — they can't conflict
- Starting/stopping/updating one service doesn't affect others
- The `docker-compose.yml` file IS the documentation — it shows exactly what the service needs to run
- Portability: move the same compose file to a new machine and it works identically
- Isolation: if Karakeep gets compromised, it can't easily touch Forgejo's data

**What you'd say to a hiring manager:** *"I containerized every service using Docker and Docker Compose so each has isolated dependencies and the entire deployment is reproducible from a single YAML file."*

---

## Why Cloudflare Tunnel Instead of Port Forwarding?

**Problem:** How do you make home services accessible from the internet?

**Traditional approach:** Open port 80 and 443 on the home router, configure NAT, point DNS to home IP.

**Problems with that:**
- Exposes your home IP address publicly (DDoS risk, can be found, ISP tracks it)
- Dynamic home IP means DNS breaks every time IP changes
- Some ISPs block residential port 80/443
- Router configuration is error-prone and varies by hardware

**Decision:** Cloudflare Tunnel (cloudflared)

**Why:**
- cloudflared makes an OUTBOUND connection to Cloudflare — no inbound ports needed
- Home IP never exposed
- Works regardless of ISP restrictions
- Cloudflare handles TLS/HTTPS — you don't manage SSL certificates
- Free tier covers everything needed
- Bonus: built-in DDoS protection

**The trade-off:** You depend on Cloudflare. If Cloudflare has an outage, your site goes down even if your hardware is fine. This is acceptable — Cloudflare's uptime is better than most home internet connections.

---

## Why Authentik for SSO Instead of Separate Logins Per App?

**Problem:** 9 services means 9 different usernames and passwords to manage. Adding a user requires going into 9 admin panels. Removing access means 9 places to deactivate.

**Options:**
- Separate logins per service (no SSO)
- Authelia (simpler, forward-auth proxy only)
- Authentik (full OIDC provider, more complex)
- Keycloak (enterprise-grade, very heavy)

**Decision:** Authentik

**Why:**
- One account controls access to everything
- Apps that support native OIDC (Grafana, Kavita, Open WebUI, Karakeep) get real SSO — the user is authenticated inside the app
- Can restrict which groups can access which applications (Portainer restricted to homelab-admin group)
- Self-hosted — user data stays on your infrastructure
- Authentik supports both native OIDC (for apps that support it) and proxy provider (for apps that don't)

**The trade-off:** Authentik is complex to set up and has a significant memory footprint. Authelia would be simpler. But Authelia only does forward-auth proxy — it can't give an app a real JWT. Authentik does both.

---

## Why a Shared Postgres Instead of Separate Authentik Databases?

**Problem:** After setting up active-active failover, users kept getting `invalid_grant` errors when signing in through SSO.

**Root cause:** OAuth2 authorization codes are rows in a database. The flow is:
1. `/authorize` → code stored in Database A (monk's Authentik)
2. `/token` → looks for code in Database B (kscloud1's Authentik)
3. Code not found → `invalid_grant`

Cloudflare Tunnel load-balances between monk and kscloud1 for every HTTP request. Steps 1 and 2 of the OAuth flow can hit different hosts.

**Options:**
- Sync databases continuously (complex, slow, conflict-prone)
- Use sticky sessions (Cloudflare paid feature)
- Share one database (simple, reliable)

**Decision:** Shared Postgres on kscloud1, accessible only over Tailscale

**Why:**
- Both monk and kscloud1 Authentik read/write the same database — authorization codes always found
- Tailscale binding means the database is never exposed to the public internet (security)
- Simple: one line change in each `docker-compose.yml` to point to a different host
- Cost: free (already paying for kscloud1)

**The trade-off:** If kscloud1 goes down and Tailscale connectivity breaks, monk's Authentik can't start. Rollback procedure: restore monk's compose to use a local Postgres.

---

## Why Tailscale Instead of WireGuard or OpenVPN?

**Problem:** Need private networking between monk (home) and kscloud1 (Hetzner cloud) without exposing the Authentik database to the public internet.

**Options:**
- WireGuard: manual key exchange, manual routing, technical to configure
- OpenVPN: even more complex, slower
- Tailscale: managed WireGuard, automatic key exchange, works behind NAT

**Decision:** Tailscale

**Why:**
- Works instantly — install, authenticate, done
- Handles NAT traversal automatically (monk is behind home router NAT)
- Devices get stable 100.x.x.x IPs regardless of actual network location
- Free for up to 100 devices
- Uses WireGuard under the hood — same encryption, much easier configuration

**The trade-off:** Tailscale is a managed service — you trust Tailscale's coordination servers. The actual data is encrypted peer-to-peer (Tailscale can't see it), but they control device authentication. Self-hosted alternative: Headscale.

---

## Why Active-Active Instead of Active-Passive Failover?

**The context:** The user travels. When away from home, monk might be inaccessible (home network down, ISP outage, power). kscloud1 should keep the site running.

**Active-Passive:** kscloud1 only starts serving if monk is detected as down. Cloudflare would need health checks and failover rules.

**Active-Active:** Both monk and kscloud1 are always in the Cloudflare Tunnel rotation. Every request might hit either host.

**Decision:** Active-Active

**Why:**
- Simpler: no health checks to configure, no failover logic
- Instant: if monk goes down, kscloud1 is already handling 50% of traffic
- Free: Cloudflare Tunnel active-active is free; health-check-based failover requires paid plans

**The trade-off:** Stateful apps (Forgejo, OpenProject, Kavita) have separate databases on each host. A user might see different data depending on which host answers. This was explicitly accepted: the point is uptime, not data consistency across hosts.

---

## Why nginx for the Portal Instead of a Pre-Built Dashboard?

**Options:**
- gethomepage (what was used before) — nice but limited customization
- Heimdall — similar limitations
- Custom static site + nginx — full control

**Decision:** Custom static HTML/CSS/JS + nginx

**Why:**
- Complete visual control — the cyberpunk theme, the layout, every pixel
- Static files served by nginx are extremely fast and reliable
- Can proxy the metrics API for real-time stats without CORS issues
- No framework dependencies — no Node.js, no build step, just files

**The trade-off:** More work to build and maintain than a pre-built dashboard. But you now understand every line of it.

---

## Why Python + FastAPI for the Metrics API?

**Problem:** The portal needs real-time system stats (CPU, RAM, network), weather, and Forgejo activity. These can't come from static HTML files.

**Options:**
- Shell scripts + cron → write stats to a JSON file the frontend reads
- Node.js + Express
- Python + FastAPI

**Decision:** Python FastAPI

**Why:**
- Python's `psutil` library reads system metrics with one line of code
- FastAPI is modern, fast, and automatically documents the API
- `async/await` means the API doesn't block while waiting for weather API responses
- Python is readable — you can understand and modify the code

**The special requirement:** The container needs `network_mode: host` and `pid: host`. Without these:
- `network_mode: host`: the container can see the host's network interfaces and report real network throughput (not container-level)
- `pid: host`: psutil can read the host's `/proc` filesystem, showing actual system stats instead of container stats

---

## Why the Forgejo Repo for Documentation?

You could keep documentation in Notion, Google Docs, or a wiki.

**Why Forgejo:**
- It's self-hosted — you own the data
- Git tracks every change with a timestamp and message
- The documentation lives alongside the configs it describes
- Hiring managers can see the commit history and read your documentation directly

**What this shows to a hiring manager:** You treat documentation like code — version-controlled, structured, maintained.