kitestacks-homelab/homelab-mastery/architecture/overview.md

# KiteStacks Architecture — Full System Overview

## The Big Picture

```
                        INTERNET
                           │
                    ┌──────▼──────┐
                    │  Cloudflare  │  DNS + TLS termination
                    │   (edge)     │  Zero Trust Tunnel
                    └──────┬──────┘
                           │  HTTPS (443) only
          ┌────────────────┼────────────────┐
          │ connector 1    │ connector 2    │ connector 3
          │                │               │
   ┌──────▼──────┐         │        ┌──────▼──────┐
   │    MONK     │         │        │   KSCLOUD1  │
   │ (home PC)   │         │        │ (Hetzner VPS│
   │             │  Active │        │  5.78.x.x)  │
   │ All 9       │  Active │        │             │
   │ services    │         │        │ All 9       │
   │             │         │        │ services    │
   └──────┬──────┘         │        └──────┬──────┘
          │                │               │
          └────────────────┼───────────────┘
                     TAILSCALE VPN
                    (100.x.x.x range)
                           │
                  ┌────────▼────────┐
                  │  SHARED DB LAYER │
                  │  on kscloud1    │
                  │  Postgres :5432  │
                  │  Redis    :6379  │
                  │  (Tailscale     │
                  │   only, private)│
                  └─────────────────┘
```

---

## Every Service and What It Does

### The Nine Public Services

| Service | Container Name | What It Does | Why It's Here |
|---------|---------------|--------------|---------------|
| **Portal** | `homepage` | The public website (kitestacks.com) — custom nginx serving static HTML/CSS/JS with a cyberpunk theme | Front door to everything. Shows system stats, recent activity, links to all services |
| **Authentik** | `authentik` | Identity provider — handles all logins via OIDC/OAuth2 SSO | Single place to manage all user accounts and access control |
| **Forgejo** | `forgejo` | Self-hosted Git platform (like GitHub but yours) | Store all homelab code, config, and documentation |
| **OpenProject** | `openproject` | Project management (like Jira) | Task tracking, project planning |
| **Open WebUI** | `kite-openwebui` | ChatGPT-like AI chat interface | Access multiple AI models through one interface |
| **Karakeep** | `karakeep` | Bookmark and read-it-later manager | Save links, articles, and content |
| **Kavita** | `kavita` | eBook and manga reader | Personal digital library |
| **Grafana** | `grafana` | Monitoring dashboards | Visualize CPU, RAM, network, uptime across both hosts |
| **Uptime Kuma** | `uptime-kuma` | Status page and uptime monitoring | Monitor that all 9 services are up and alert if they go down |

### The Infrastructure Services (Not Public-Facing)

| Service | What It Does |
|---------|-------------|
| `cloudflared` | Cloudflare Tunnel connector — creates encrypted outbound tunnel to Cloudflare edge |
| `prometheus` | Metrics collection — scrapes system stats from both monk and kscloud1 every 15 seconds |
| `node-exporter` | Exposes host system metrics (CPU, RAM, disk, network) for Prometheus to scrape |
| `kite-litellm` | LLM proxy gateway — routes AI requests to OpenRouter (multiple free models) |
| `portainer` | Docker management UI — visual interface to manage all containers |
| `kitestacks-metrics-api` | Python FastAPI service — serves real-time system stats, weather, and Forgejo activity to the portal |

---

## How Traffic Flows

### When Someone Visits www.kitestacks.com

```
1. Browser sends HTTPS request to www.kitestacks.com
2. DNS resolves to Cloudflare's anycast IP (not your home IP)
3. Cloudflare terminates TLS — your home router never sees HTTPS
4. Cloudflare routes the request through the tunnel to whichever
   cloudflared connector responds first (monk or kscloud1)
5. cloudflared resolves "homepage" via Docker DNS
6. Request hits the nginx container serving the static portal
7. Portal's JavaScript fetches /api/metrics and /api/activity
   from the kitestacks-metrics-api container via nginx proxy
8. Page renders with live system stats and recent git activity
```

### When Someone Clicks "Sign In with Authentik"

```
1. App (e.g., Grafana) redirects browser to auth.kitestacks.com/application/o/authorize/
2. Authentik presents login page
3. User enters credentials — Authentik validates against its database
   (stored on kscloud1's Postgres, shared over Tailscale)
4. Authentik generates an authorization code and redirects back to Grafana
5. Grafana's backend calls auth.kitestacks.com/application/o/token/
   to exchange the code for an access token
6. Authentik validates the code (found in shared DB) and returns a JWT
7. Grafana reads the user's email/name from the JWT and logs them in
```

**The critical detail:** Steps 1 and 5 can hit different tunnel connectors (monk vs kscloud1). The authorization code from step 4 must exist in whichever database step 5 hits. That's why both connectors point to the SAME Postgres on kscloud1 — otherwise step 5 returns `invalid_grant` because the code isn't found.

---

## The Two Hosts in Detail

### Monk (Primary Home Machine)

- **Role:** Primary production host
- **Network:** Home LAN, no open ports on router (Cloudflare Tunnel handles all inbound)
- **Services:** All 9 public services + all infrastructure services
- **Data:** Each service has its own database/storage
- **Authentik DB:** Points to kscloud1's Postgres over Tailscale (100.x.x.x)

### kscloud1 (Hetzner VPS)

- **Role:** Permanent cloud replica — always on, even when monk is off (travel, power outage, etc.)
- **Network:** Public IP, Cloudflare Tunnel connector 3
- **Services:** Full replica of all 9 public services (separate databases except Authentik)
- **Hosts:** The shared Authentik Postgres + Redis (bound to Tailscale interface only)
- **Resources:** 3 vCPU, 3.7 GB RAM — tight but functional

### What's the Same Across Both

- Same Cloudflare Tunnel token (different connector IDs assigned automatically)
- Same Authentik database (shared via Tailscale)
- Same Authentik secret key (required for JWT validation)
- Same kavita.db (one-time sync — users and OIDC config)

### What's Different Across Both

- Forgejo data (separate repos — accepted inconsistency)
- OpenProject data (separate projects)
- Karakeep bookmarks (separate)
- Kavita book files (monk has them, kscloud1 doesn't — covers synced, books not)

---

## The Docker Network

Every container joins the `kitestacks` external Docker bridge network:

```bash
docker network create kitestacks
```

This is what makes Cloudflare Tunnel work. The cloudflared container is also on this network, so when Cloudflare tells cloudflared to route `http://grafana:3000`, Docker's internal DNS resolves `grafana` to the grafana container's IP on that network.

Without this shared network, cloudflared can't reach the service containers by name.

---

## Why No Open Ports on the Router

Traditional homelab: open port 80/443 on home router → NAT to home server → expose home IP.

Problems with that:
- Your home IP is public (DDoS risk, targeted attacks)
- Router configuration is fragile
- ISP can change your IP (dynamic IP)
- Some ISPs block port 80/443

Cloudflare Tunnel approach:
- cloudflared container makes an OUTBOUND connection to Cloudflare
- Cloudflare holds that connection open
- Inbound requests come through Cloudflare, over that existing outbound tunnel
- Your home IP is never exposed
- Works on any network, any ISP, any firewall

This is why you can run a public website from a home PC with zero router configuration.

---

## Tailscale — The Private Backbone

Tailscale creates a private overlay network (VPN mesh) across all your devices:

```
monk (100.x.x.x) ←—— encrypted ——→ kscloud1 (100.x.x.x)
monk (100.x.x.x) ←—— encrypted ——→ pixel-6 (100.x.x.x)
```

Used in this project for:
1. **Shared Authentik DB:** kscloud1's Postgres binds to its Tailscale IP, not its public IP. Only devices on the tailnet can connect. Monk points to that address.
2. **Forgejo activity feed:** On kscloud1, the metrics API fetches recent commits from monk's Forgejo via monk's Tailscale IP — so both portal instances show the same activity feed.
3. **SSH/Admin access:** You can SSH into any device on the tailnet from anywhere.

---

## The Monitoring Stack

```
node-exporter (monk)  →  prometheus (monk)  →  grafana (monk)
node-exporter (kscloud1) ↗       (scrapes 5.78.x.x:9100)
```

Prometheus scrapes metrics every 15 seconds from:
- `node-exporter:9100` — monk's own node-exporter (via Docker DNS)
- `5.78.x.x:9100` — kscloud1's node-exporter (via public IP, port exposed 0.0.0.0)

Grafana visualizes both, letting you switch between hosts in the instance picker.

---

## The Portal Architecture

The portal is NOT gethomepage or any pre-built dashboard. It's a custom-built static site:

```
nginx (container: "homepage")
  ├── /         → serves static HTML/CSS/JS from ./public/
  └── /api/*    → proxy_pass to kitestacks-metrics-api:8000 (host)

kitestacks-metrics-api (network_mode: host, pid: host)
  ├── GET /api/metrics   → psutil reads HOST's CPU/RAM/disk/network
  ├── GET /api/weather   → wttr.in API → current weather by IP geolocation
  ├── GET /api/activity  → Forgejo API → recent commits
  └── GET /api/health    → {"ok": true}
```

The metrics API runs with `network_mode: host` and `pid: host` so it reads the HOST machine's process table and `/proc` filesystem — not the container's. Without this, it would report container stats, not laptop stats.