From 4dc535905aca4d96978e342b28ed583eadccc59e Mon Sep 17 00:00:00 2001 From: kenpat Date: Thu, 18 Jun 2026 21:08:03 +0000 Subject: [PATCH] Update RUNBOOK with BookStack OIDC fix and kscloud1 SSH recovery --- RUNBOOK.md | 1575 ++++++++-------------------------------------------- 1 file changed, 219 insertions(+), 1356 deletions(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 1fd0196..8c6857c 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -1,8 +1,8 @@ # KiteStacks Homelab — Complete Setup Runbook -**Last Updated:** 2026-06-12 +**Last Updated:** 2026-06-18 **Status:** Production (monk primary, kscloud1 Hetzner cloud replica) -**Maintainer:** kenpat (kenpat7177@gmail.com) +**Maintainer:** kenpat --- @@ -19,1412 +19,275 @@ Internet Tailscale overlay network (VPN mesh): monk kscloud1 ← hosts shared Authentik Postgres + Redis - T14s (off) - pixel-6 - samurai ``` -**Nine public subdomains** route through the same Cloudflare Tunnel token. Both monk and kscloud1 are connectors so the site stays up when either goes offline. +**Public subdomains** route through the same Cloudflare Tunnel token. +Both monk and kscloud1 are connectors so the site stays up if either goes offline. -| Subdomain | Container | Port | -|-----------|-----------|------| -| www.kitestacks.com | homepage (nginx portal) | | -| auth.kitestacks.com | authentik | | -| gitforge.kitestacks.com | forgejo | | -| tasks.kitestacks.com | osticket (nginx proxy → osticket-app) | 8080 | -| portainer.kitestacks.com | portainer | 9443 (HTTPS) | -| ai.kitestacks.com | kite-openwebui | | -| links.kitestacks.com | karakeep | | -| kavita.kitestacks.com | kavita | | -| grafana.kitestacks.com | grafana | | -| status.kitestacks.com | uptime-kuma | | - -**Important — active-active data model:** monk and kscloud1 each run their own copies of all stateful apps (Forgejo, Kavita, OSticket, etc.) with independent databases. Data is intentionally NOT synced between them (except for Authentik, which shares a single Postgres+Redis on kscloud1 over Tailscale). If kscloud1 serves a request, the user sees kscloud1's database. This is the accepted tradeoff for guaranteed uptime. +| Subdomain | Service | Port | +|-----------|---------|------| +| auth.kitestacks.com | Authentik | 9000 | +| portainer.kitestacks.com | Portainer | 9443 | +| wiki.kitestacks.com | BookStack | 6875 (monk) / 6877 (kscloud1) | +| grafana.kitestacks.com | Grafana | 3000 | +| gitforge.kitestacks.com | Forgejo | 3006 | +| links.kitestacks.com | Karakeep | 3100 | +| status.kitestacks.com | Uptime Kuma | 3001 | +| tasks.kitestacks.com | OSTicket | 8080 | +| flux.kitestacks.com | FluxCD | — | --- -## Phase 0 — Prerequisites +## Service Inventory -**Accounts & services you need before starting:** +### Running on monk +authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy, +bookstack, bookstack-db, cloudflared, flux, forgejo, grafana, +karakeep, karakeep-chrome, karakeep-meilisearch, kavita, +kite-litellm, kite-openwebui, kitestacks-metrics-api, kitestacks-portal, +node-exporter, ntfy, osticket, osticket-app, osticket-db, +portainer, prometheus, uptime-kuma, blackbox-exporter -- Domain name (kitestacks.com) registered and nameservers pointed to Cloudflare -- Cloudflare account (free tier is fine — do NOT enable Zero Trust/Access, which costs money; the Tunnel UI lives under the "Zero Trust" nav but configuring tunnel hostnames is free) -- Hetzner Cloud account (for kscloud1 VPS) -- OpenRouter account + API key (for LiteLLM/AI services) -- Discord server (for community panel + #recent-activities webhook) -- SSH key pair (`~/.ssh/id_ed25519_kscloud1` for kscloud1 access) +### Running on kscloud1 (extras) +bookstack, bookstack-db-ks, kite-monitor, osticket-app-118, +osticket-db-118, www-backup, homepage-backup, cloudflared, +authentik-postgresql, authentik-redis -# Software on T14s (GitOps Node): +### Shared infrastructure on kscloud1 +- PostgreSQL `:5432` — Authentik DB used by both hosts (Tailscale only) +- Redis `:6379` — Authentik session cache (Tailscale only) + +--- + +## Cloudflare Tunnel + +### How it works +Both monk and kscloud1 run `cloudflared` as Docker containers using the **same tunnel token**. Cloudflare load-balances across both connectors (active-active). The tunnel token is stored in: +- monk: `~/kitestacks-live/docker/cloudflared/.env` → `TUNNEL_TOKEN` +- kscloud1: `/opt/kitestacks/docker/cloudflared/.env` → `TUNNEL_TOKEN` + +### Fix: Phantom 3rd Replica +If `cloudflared tunnel info` shows 3 connectors instead of 2, the native cloudflared systemd service on monk is running alongside the Docker container. ```bash -# K3s (Kubernetes) -curl -sfL https://get.k3s.io | sh - -sudo chmod 644 /etc/rancher/k3s/k3s.yaml -mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config - -# Flux CLI -curl -s https://fluxcd.io/install.sh | sudo bash +# Check systemd cloudflared on monk +systemctl status cloudflared +# Disable it — Docker container is the correct one +sudo systemctl disable --now cloudflared ``` -# Docker (official install script or distro package) -curl -fsSL https://get.docker.com | sh -sudo usermod -aG docker $USER # log out and back in +### Adding a new hostname route +In Cloudflare Zero Trust → Networks → Tunnels → your tunnel → Edit → Public Hostname: +- Subdomain: `newservice` +- Domain: `kitestacks.com` +- Service: `http://container-name:port` -# Tailscale -curl -fsSL https://tailscale.com/install.sh | sh +Both monk and kscloud1 must have the container running on the same port. + +--- + +## Authentik SSO + +### Architecture +Authentik uses a **shared database** hosted on kscloud1. monk's Authentik containers connect via Tailscale. + +- monk containers: `authentik`, `authentik-worker`, `authentik-ldap`, `authentik-ldap-proxy` +- DB: PostgreSQL on kscloud1 at `:5432` +- Redis: kscloud1 at `:6379` + +### Adding OIDC SSO for a new app + +1. In Authentik admin (`https://auth.kitestacks.com/if/admin/`): + - **Providers** → Create → OAuth2/OpenID Provider + - Name the provider after the app (e.g. `bookstack`) + - Set `issuer_mode` based on the app's requirements (see Debug doc) + - Note the Client ID and Client Secret + +2. **Application** → Create → link to the provider + +3. **Policy Binding** → bind the `default-authentication-flow` to the application + +4. Configure the app with: + - `OIDC_ISSUER` = discovery base URL + - `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` + - Callback URL = `https://yourapp.kitestacks.com/auth/callback` + +### Checking OIDC discovery URL +```bash +# Per-provider (issuer_mode=per_provider) +curl -s https://auth.kitestacks.com/application/o//.well-known/openid-configuration | python3 -m json.tool + +# Global (issuer_mode=global) +# Note: global issuer URL does NOT serve a JSON discovery doc at /.well-known/ +# Use per-provider mode for apps that auto-discover endpoints (BookStack, etc.) +``` + +### Changing provider issuer_mode via SQL +```bash +docker run --rm --network host \ + -e PGPASSWORD="" \ + postgres:16 psql -h -U authentik authentik -c \ + "UPDATE authentik_providers_oauth2_oauth2provider SET issuer_mode='per_provider' WHERE provider_ptr_id=;" ``` --- -## Phase 1 — Monk (Primary Host) — Base Setup +## Portainer -Monk is the primary home server. All services live under `~/kitestacks-live/docker/`, one directory per app. +### OAuth setup (Authentik) +Portainer CE uses AuthenticationMethod=3 (OAuth). Configured via the BoltDB. -### 1.1 Create the shared Docker network +Key settings: +- `OAuthLoginURI`: `https://auth.kitestacks.com/application/o/authorize/` +- `OAuthTokenURI`: `https://auth.kitestacks.com/application/o/token/` +- `OAuthUserURI`: `https://auth.kitestacks.com/application/o/userinfo/` +- `OAuthClientID`: `portainer` +- `OAuthRedirectURI`: `https://portainer.kitestacks.com` +- `OAuthAutoCreateUsers`: `true` +- `OAuthDefaultTeamID`: `0` -All containers join this network. `cloudflared` resolves container names within it for tunnel routing. +### Pre-creating an admin user before first OAuth login +OAuth auto-created users default to Role:2 (regular user) and can't see environments. +Pre-create them as Role:1 (admin) via the API before they log in: ```bash -docker network create kitestacks -``` - -### 1.2 Directory structure - -``` -~/kitestacks-live/ -└── docker/ - ├── authentik/ - ├── cloudflared/ - ├── forgejo/ - ├── grafana/ - ├── karakeep/ - ├── kavita/ - ├── kite-ai/ - ├── kitestacks-portal/ # production portal (container: homepage) - ├── kitestacks-portal-test/ # dev portal + metrics API - ├── openproject/ - ├── portainer/ - ├── prometheus/ - └── uptime-kuma/ -``` - -### 1.3 Tailscale on monk - -```bash -sudo tailscale up -# Accept the auth link, join tailnet as "monk" -# monk will get Tailscale IP -``` - ---- - -## Phase 2 — kscloud1 (Hetzner VPS) — Provision & Base Setup - -### 2.1 Provision on Hetzner - -- Server: Hetzner CX22 (or equivalent) — 3 vCPU, 3.7 GB RAM, 75 GB disk -- OS: Debian/Ubuntu -- Region: your choice -- Public IP: -- Add your SSH public key at provision time - -### 2.2 SSH access - -```bash -ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@ -``` - -Password for sudo: `` (non-interactive sudo: `echo | sudo -S `) - -### 2.3 Install Docker on kscloud1 - -```bash -curl -fsSL https://get.docker.com | sh -sudo usermod -aG docker kenpat -# Log out and back in -``` - -### 2.4 Create the kitestacks Docker network on kscloud1 - -```bash -docker network create kitestacks -``` - -All service directories live under `/opt/kitestacks/docker/` (same one-dir-per-app pattern as monk). - -### 2.5 Tailscale on kscloud1 - -```bash -curl -fsSL https://tailscale.com/install.sh | sh -sudo tailscale up -# Join the same tailnet; kscloud1 will get Tailscale IP -``` - -### 2.6 ufw on kscloud1 - -kscloud1 has ufw active with default-deny. Fix docker-bridge-to-host traffic: - -```bash -echo | sudo -S ufw allow from /12 to any port proto tcp -# Allows homepage metrics API to be reached from within docker containers -``` - ---- - -## Phase 3 — Cloudflare Tunnel Setup - -### 3.1 Create the tunnel - -In the Cloudflare dashboard: **Zero Trust → Networks → Tunnels → Create a tunnel** - -- Type: Cloudflared -- Name: kitestacks (or any name) -- Copy the **Tunnel Token** — this is the `TUNNEL_TOKEN` used in all cloudflared containers - -### 3.2 Add public hostnames - -Still in the tunnel config, add one public hostname per subdomain. Use the container name as the service target (cloudflared resolves these within the `kitestacks` Docker network): - -| Public Hostname | Service | -|----------------|---------| -| www.kitestacks.com | `http://homepage:` | -| auth.kitestacks.com | `http://authentik:` | -| gitforge.kitestacks.com | `http://forgejo:` | -| tasks.kitestacks.com | `http://osticket:8080` | -| portainer.kitestacks.com | `https://portainer:9443` (enable "No TLS Verify") | -| ai.kitestacks.com | `http://kite-openwebui:` | -| links.kitestacks.com | `http://karakeep:` | -| kavita.kitestacks.com | `http://kavita:` | -| grafana.kitestacks.com | `http://grafana:` | -| status.kitestacks.com | `http://uptime-kuma:` | -| portainer.kitestacks.com | `https://portainer:` (enable "No TLS Verify") | - -> **Note:** Portainer uses HTTPS internally. All others are plain HTTP (TLS is handled by Cloudflare at the edge). - ---- - -## Phase 4 — Shared Authentik Database on kscloud1 - -Authentik on both monk and kscloud1 must share ONE Postgres+Redis to prevent `invalid_grant` errors. OAuth2 authorization codes are per-DB rows — if `/authorize` hits one connector and `/application/o/token/` hits another, the code is not found. The shared DB lives on kscloud1, bound only to the Tailscale interface. - -### 4.1 Deploy shared Postgres + Redis on kscloud1 - -`/opt/kitestacks/docker/authentik/docker-compose.yml`: - -```yaml -services: - authentik-postgres: - image: postgres:16 - container_name: authentik-postgres - restart: unless-stopped - environment: - POSTGRES_USER: authentik - POSTGRES_PASSWORD: ${PG_PASS} - POSTGRES_DB: authentik - ports: - - "::" # Tailscale IP only — not public - volumes: - - ./postgres:/var/lib/postgresql/data - - authentik-redis: - image: redis:alpine - container_name: authentik-redis - restart: unless-stopped - ports: - - "::" # Tailscale IP only — not public - volumes: - - redis-data:/data - - authentik: - image: ghcr.io/goauthentik/server:latest - container_name: authentik - restart: unless-stopped - command: server - environment: - AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY} - AUTHENTIK_REDIS__HOST: - AUTHENTIK_POSTGRESQL__HOST: - AUTHENTIK_POSTGRESQL__USER: authentik - AUTHENTIK_POSTGRESQL__NAME: authentik - AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS} - AUTHENTIK_ERROR_REPORTING__ENABLED: "false" - AUTHENTIK_BOOTSTRAP_EMAIL: - AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD} - volumes: - - ./media:/media - - ./custom-templates:/templates - ports: - - ":" - networks: - - default - - kitestacks - depends_on: - - authentik-postgres - - authentik-redis - - authentik-worker: - image: ghcr.io/goauthentik/server:latest - container_name: authentik-worker - restart: unless-stopped - command: worker - environment: - AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY} - AUTHENTIK_REDIS__HOST: - AUTHENTIK_POSTGRESQL__HOST: - AUTHENTIK_POSTGRESQL__USER: authentik - AUTHENTIK_POSTGRESQL__NAME: authentik - AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS} - AUTHENTIK_ERROR_REPORTING__ENABLED: "false" - volumes: - - ./media:/media - - ./custom-templates:/templates - networks: - - default - - kitestacks - -volumes: - redis-data: - -networks: - kitestacks: - external: true -``` - -`.env` keys: `PG_PASS`, `AUTHENTIK_SECRET_KEY`, `AUTHENTIK_BOOTSTRAP_PASSWORD` - -> **Wait ~2 minutes** after first boot for Postgres migrations and Authentik startup before testing. - ---- - -## Phase 5 — Deploy Services on Monk - -Start with cloudflared first so tunnel connectors register, then deploy services in any order (they all restart-unless-stopped). - -### 5.1 cloudflared - -`~/kitestacks-live/docker/cloudflared/docker-compose.yml`: - -```yaml -services: - cloudflared: - image: cloudflare/cloudflared:latest - container_name: cloudflared - restart: unless-stopped - command: tunnel --no-autoupdate run - environment: - - TUNNEL_TOKEN= - networks: - - default - - kitestacks - -networks: - kitestacks: - external: true -``` - -```bash -cd ~/kitestacks-live/docker/cloudflared && docker compose up -d -``` - -> **Important:** After starting the Docker container, check for a pre-existing native cloudflared systemd service and disable it — both will connect with the same token and register as separate phantom replicas in the CF dashboard: -> ```bash -> systemctl status cloudflared -> sudo systemctl stop cloudflared && sudo systemctl disable cloudflared -> ``` - -### 5.2 Authentik (monk side — points to shared DB on kscloud1) - -`~/kitestacks-live/docker/authentik/docker-compose.yml`: - -```yaml -services: - authentik: - image: ghcr.io/goauthentik/server:latest - container_name: authentik - restart: unless-stopped - command: server - environment: - AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY} - AUTHENTIK_REDIS__HOST: - AUTHENTIK_POSTGRESQL__HOST: - AUTHENTIK_POSTGRESQL__USER: authentik - AUTHENTIK_POSTGRESQL__NAME: authentik - AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS} - AUTHENTIK_ERROR_REPORTING__ENABLED: "false" - volumes: - - ./media:/media - - ./custom-templates:/templates - ports: - - ":" - networks: - - default - - kitestacks - - authentik-worker: - image: ghcr.io/goauthentik/server:latest - container_name: authentik-worker - restart: unless-stopped - command: worker - environment: - AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY} - AUTHENTIK_REDIS__HOST: - AUTHENTIK_POSTGRESQL__HOST: - AUTHENTIK_POSTGRESQL__USER: authentik - AUTHENTIK_POSTGRESQL__NAME: authentik - AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS} - AUTHENTIK_ERROR_REPORTING__ENABLED: "false" - volumes: - - ./media:/media - - ./custom-templates:/templates - networks: - - default - - kitestacks - -networks: - kitestacks: - external: true -``` - -`.env` keys: `AUTHENTIK_SECRET_KEY` (same value as kscloud1), `PG_PASS` (same value as kscloud1) - -> **Critical:** `AUTHENTIK_SECRET_KEY` and `PG_PASS` must be IDENTICAL between monk and kscloud1 or shared-DB authentication will fail. - -> **Rollback note:** If Tailscale connectivity breaks, monk's authentik will fail to connect to . To roll back: restore a local postgres+redis to monk's compose, `docker start authentik-postgres authentik-redis`, then `docker compose up -d`. Do a fresh `pg_dump` from kscloud1 first to avoid losing any logins made since the shared-DB migration. - -### 5.3 Forgejo - -`~/kitestacks-live/docker/forgejo/docker-compose.yml`: - -```yaml -services: - forgejo: - image: codeberg.org/forgejo/forgejo:11 - container_name: forgejo - restart: unless-stopped - environment: - - USER_UID=1000 - - USER_GID=1000 - - FORGEJO__server__DOMAIN=gitforge.kitestacks.com - - FORGEJO__server__ROOT_URL=https://gitforge.kitestacks.com/ - - FORGEJO__server__SSH_DOMAIN=gitforge.kitestacks.com - - FORGEJO__server__SSH_PORT= - ports: - - ":" - - ":" - volumes: - - ./data:/data - networks: - - default - - kitestacks - -networks: - kitestacks: - external: true -``` - -### 5.4 KiteStacks Portal (Production) - -The portal is a custom nginx-served static site (cyberpunk-themed). The container is named `homepage` to match the Cloudflare tunnel route `http://homepage:`. - -`~/kitestacks-live/docker/kitestacks-portal/docker-compose.yml`: - -```yaml -services: - homepage: - image: nginx:alpine - container_name: homepage - restart: unless-stopped - ports: - - ":" - networks: - - default - - kitestacks - volumes: - - ./public:/usr/share/nginx/html:ro - - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro - extra_hosts: - - "host.docker.internal:host-gateway" - -networks: - kitestacks: - external: true -``` - -`nginx.conf`: - -```nginx -server { - listen ; - server_name _; - - root /usr/share/nginx/html; - index index.html; - - location / { - try_files $uri $uri/ /index.html; - } - - location /api/ { - proxy_pass http://host.docker.internal:; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_read_timeout 10s; - } - - location /images/ { - expires 7d; - add_header Cache-Control "public, max-age=604800"; - } -} -``` - -The `/api/` path proxies to the Metrics API (port on the host). - -### 5.5 Metrics API (Portal Backend) - -The Metrics API provides real-time system stats (CPU, RAM, storage, network, uptime), weather, and Forgejo recent-activity data to the portal. It runs in `network_mode: host` with `pid: host` to read actual host metrics via psutil. - -`~/kitestacks-live/docker/kitestacks-portal-test/docker-compose.yml` (this is the compose that owns the shared metrics-api container): - -```yaml -services: - kitestacks-portal-test: - image: nginx:alpine - container_name: kitestacks-portal-test - restart: unless-stopped - ports: - - ":" - extra_hosts: - - "host.docker.internal:host-gateway" - volumes: - - ./public:/usr/share/nginx/html:ro - - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro - depends_on: - - metrics-api - - metrics-api: - build: ./api - container_name: kitestacks-metrics-api - restart: unless-stopped - pid: host - network_mode: host - environment: - - HOST_PROC=/host/proc - - HOST_SYS=/host/sys - - HOST_ETC=/host/etc - - FORGEJO_API_BASE=http://localhost: - volumes: - - /proc:/host/proc:ro - - /sys:/host/sys:ro - - /etc/os-release:/host/etc/os-release:ro - - /etc/hostname:/host/etc/hostname:ro - - /etc/localtime:/host/etc/localtime:ro - - /:/host:ro - - /var/run/docker.sock:/var/run/docker.sock:ro -``` - -`api/main.py` serves four endpoints: -- `GET /api/metrics` — CPU, RAM, storage, network, uptime, system info -- `GET /api/weather` — weather via ipapi.co + open-meteo.com (cached 10 min) -- `GET /api/activity` — recent Forgejo commits (cached 60s) -- `GET /api/health` — `{"ok": true}` - -`FORGEJO_API_BASE` on monk points to `http://localhost:` (monk's own Forgejo). On kscloud1 it points to `http://:` (monk's Forgejo over Tailscale) so both connectors show consistent recent activity from the same source. - -**Python deps:** `fastapi`, `uvicorn`, `psutil`, `httpx` - -```bash -cd ~/kitestacks-live/docker/kitestacks-portal-test && docker compose up -d -cd ~/kitestacks-live/docker/kitestacks-portal && docker compose up -d -``` - -### 5.6 Karakeep (Bookmarks) - -`~/kitestacks-live/docker/karakeep/docker-compose.yml`: - -```yaml -services: - karakeep: - image: ghcr.io/karakeep-app/karakeep:${KARAKEEP_VERSION:-release} - container_name: karakeep - restart: unless-stopped - environment: - - PORT= - - MEILI_ADDR=http://karakeep-meilisearch: - - BROWSER_WEB_URL=http://karakeep-chrome: - - DATA_DIR=/data - volumes: - - ./data:/data - networks: - - default - - kitestacks - - internal - - karakeep-chrome: - image: gcr.io/zenika-hub/alpine-chrome:124 - container_name: karakeep-chrome - restart: unless-stopped - command: chromium-browser --headless --remote-debugging-address= --remote-debugging-port= - networks: - - internal - - karakeep-meilisearch: - image: getmeili/meilisearch:v1.41.0 - container_name: karakeep-meilisearch - restart: unless-stopped - environment: - - MEILI_NO_ANALYTICS=true - volumes: - - ./meilisearch:/meili_data - networks: - - internal - -networks: - kitestacks: - external: true - internal: -``` - -`.env` keys: `KARAKEEP_VERSION`, `NEXTAUTH_SECRET`, `NEXTAUTH_URL`, `MEILI_MASTER_KEY`, `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`, `OAUTH_WELLKNOWN_URL`, `OAUTH_PROVIDER_NAME`, `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING` - -**SSO note for Karakeep:** The `OAUTH_CLIENT_ID` is `karakeep`. The Authentik redirect URI must be `https://links.kitestacks.com/api/auth/callback/custom` (NextAuth.js uses provider id `custom`, NOT `authentik` — the callback path is `/api/auth/callback/custom`). - -### 5.7 Kavita (eBook Reader) - -`~/kitestacks-live/docker/kavita/docker-compose.yml`: - -```yaml -services: - kavita: - image: ghcr.io/kareadita/kavita:latest - container_name: kavita - restart: unless-stopped - environment: - - TZ=UTC - ports: - - ":" - volumes: - - ./config:/kavita/config - - ../../library/books:/books - networks: - - default - - kitestacks - -networks: - kitestacks: - external: true -``` - -**SSO config for Kavita:** Do NOT edit `config/kavita.db` directly — settings written to the `ServerSetting` table are overwritten by Kavita on restart. Always configure OIDC through Kavita's Settings UI: -- Go to `http://localhost:` (or SSH tunnel `ssh -L 5099:localhost: kenpat@` for kscloud1) -- Settings → OIDC: - - Authority: `https://auth.kitestacks.com/application/o/kavita/` (trailing slash required — must exactly match the `issuer` field in Authentik's discovery doc) - - Client ID: `kavita` - - Client Secret: (96-char hex from Authentik) - - Enabled: true, ProviderName: `authentik` - -### 5.8 OSticket (Help Desk) — replaced OpenProject 2026-06-12 - -OSticket is a PHP-based help desk ticketing system. It runs behind an nginx proxy container that Cloudflare Tunnel reaches at `http://osticket:8080`. - -**Architecture:** `osticket` (nginx:alpine, port 8080) → `osticket-app` (campbellsoftwaresolutions/osticket, port 80) + `osticket-db` (mariadb:10.11) - -`~/kitestacks-live/docker/osticket/docker-compose.yml`: - -```yaml -services: - osticket: - image: nginx:alpine - container_name: osticket - restart: unless-stopped - depends_on: - - osticket-app - volumes: - - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro - ports: - - ":8080" - networks: - - default - - kitestacks - - osticket-app: - image: campbellsoftwaresolutions/osticket - container_name: osticket-app - restart: unless-stopped - depends_on: - - osticket-db - environment: - - INSTALL_NAME=KiteStacks Help Desk - - INSTALL_EMAIL=helpdesk@kitestacks.com - - INSTALL_URL=https://tasks.kitestacks.com/ - - ADMIN_FIRSTNAME=Ken - - ADMIN_LASTNAME=Pat - - ADMIN_EMAIL= - - ADMIN_USERNAME= - - ADMIN_PASSWORD=${OSTICKET_ADMIN_PASS} - - INSTALL_SECRET=${OSTICKET_INSTALL_SECRET} - - MYSQL_HOST=osticket-db - - MYSQL_DATABASE=osticket - - MYSQL_USER=osticket - - MYSQL_PASSWORD=${OSTICKET_DB_PASS} - volumes: - - osticket_uploads:/data/upload/include/i18n - networks: - - default - - osticket-db: - image: mariadb:10.11 - container_name: osticket-db - restart: unless-stopped - environment: - - MYSQL_ROOT_PASSWORD=${OSTICKET_DB_ROOT} - - MYSQL_DATABASE=osticket - - MYSQL_USER=osticket - - MYSQL_PASSWORD=${OSTICKET_DB_PASS} - volumes: - - osticket_db:/var/lib/mysql - networks: - - default - -volumes: - osticket_uploads: - osticket_db: - -networks: - kitestacks: - external: true -``` - -`nginx.conf`: -```nginx -server { - listen 8080; - location / { - proxy_pass http://osticket-app:80; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto https; - proxy_set_header X-Forwarded-Host $host; - proxy_read_timeout 120s; - } -} -``` - -`.env` keys: `OSTICKET_DB_PASS`, `OSTICKET_DB_ROOT`, `OSTICKET_ADMIN_PASS`, `OSTICKET_INSTALL_SECRET` - -**First boot:** The install script runs automatically, connects to MariaDB, and seeds the database. Wait for `Database installation successful` in `docker logs osticket-app` before testing. The same `.env` values are used on kscloud1 so both instances share the same admin credentials (separate databases). - -**Cloudflare Tunnel:** Route `tasks.kitestacks.com → http://osticket:8080` in the tunnel config. - -**kscloud1 note:** Same compose, host port `:8080` (8090 is used since port 80 is taken by caddy). - -### 5.9 Grafana + Prometheus + Node Exporter - -`~/kitestacks-live/docker/prometheus/docker-compose.yml`: - -```yaml -services: - prometheus: - image: prom/prometheus - container_name: prometheus - restart: unless-stopped - ports: - - ":" - volumes: - - ./prometheus.yml:/etc/prometheus/prometheus.yml - - prometheus-data:/prometheus - networks: - - default - - kitestacks - - node-exporter: - image: prom/node-exporter - container_name: node-exporter - restart: unless-stopped - ports: - - ":" - networks: - - default - - kitestacks - -volumes: - prometheus-data: - -networks: - kitestacks: - external: true -``` - -`~/kitestacks-live/docker/prometheus/prometheus.yml`: - -```yaml -global: - scrape_interval: 15s - -scrape_configs: - - job_name: "t14-node" - static_configs: - - targets: ["node-exporter:"] # monk (this host) - - - job_name: "kscloud1-node" - static_configs: - - targets: [":"] # kscloud1 (cloud replica) -``` - -`~/kitestacks-live/docker/grafana/docker-compose.yml`: - -```yaml -services: - grafana: - image: grafana/grafana-oss - container_name: grafana - restart: unless-stopped - ports: - - ":" - environment: - - GF_SERVER_ROOT_URL=https://grafana.kitestacks.com - - GF_AUTH_GENERIC_OAUTH_ENABLED=true - - GF_AUTH_GENERIC_OAUTH_NAME=authentik - - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana - - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET} - - GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile - - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.kitestacks.com/application/o/authorize/ - - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://authentik:/application/o/token/ - - GF_AUTH_GENERIC_OAUTH_API_URL=http://authentik:/application/o/userinfo/ - - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true - - GF_AUTH_OAUTH_AUTO_LOGIN=false - volumes: - - ./data:/var/lib/grafana - - ./provisioning:/etc/grafana/provisioning - networks: - - default - - kitestacks - -networks: - kitestacks: - external: true -``` - -`.env` keys: `GRAFANA_OAUTH_CLIENT_SECRET` - -**Grafana provisioning** (`./provisioning/`): - -`datasources/prometheus.yml`: -```yaml -apiVersion: 1 -datasources: - - name: Prometheus - type: prometheus - access: proxy - url: http://prometheus: - uid: "000000001" - isDefault: true - editable: true -``` - -`dashboards/dashboard.yml`: -```yaml -apiVersion: 1 -providers: - - name: default - folder: '' - type: file - options: - path: /etc/grafana/provisioning/dashboards -``` - -Place `node-exporter-full.json` (Grafana dashboard ID 1860) in `./provisioning/dashboards/`. The dashboard's instance picker dropdown will show both `t14-node` (monk) and `kscloud1-node`. - -### 5.10 Uptime Kuma - -`~/kitestacks-live/docker/uptime-kuma/docker-compose.yml`: - -```yaml -services: - uptime-kuma: - image: louislam/uptime-kuma:latest - container_name: uptime-kuma - restart: unless-stopped - ports: - - ":" - volumes: - - uptime-kuma:/app/data - networks: - - default - - kitestacks - -volumes: - uptime-kuma: - -networks: - kitestacks: - external: true -``` - -Configure monitors in the Uptime Kuma UI for all 9 public subdomains plus internal health checks. - -### 5.11 Portainer - -`~/kitestacks-live/docker/portainer/docker-compose.yml`: - -```yaml -services: - portainer: - image: portainer/portainer-ce:latest - container_name: portainer - restart: unless-stopped - ports: - - ":" - volumes: - - portainer_data:/data - - /var/run/docker.sock:/var/run/docker.sock - networks: - - default - - kitestacks - -volumes: - portainer_data: - -networks: - kitestacks: - external: true -``` - -**SSO — CONFIGURED (2026-06-12):** Authentik OAuth2 provider (Client ID: `portainer`) is live. OAuth configured via the Portainer API on both monk and kscloud1. Portal card updated to a live link on all 3 copies. - -- Cloudflare Tunnel: `portainer.kitestacks.com → https://portainer:9443` (No TLS Verify) — **add this in CF dashboard if not yet present** -- Access restricted to `homelab-admin` Authentik group via PolicyBinding - -**To re-configure OAuth via Portainer API** (if settings get reset): -```bash -JWT=$(curl -sk -X POST https://localhost:9443/api/auth \ +# Get auth token +TOKEN=$(curl -sk -X POST https://portainer.kitestacks.com/api/auth \ -H "Content-Type: application/json" \ - -d '{"Username":"admin","Password":""}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])") + -d '{"username":"admin","password":""}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])") -curl -sk -X PUT https://localhost:9443/api/settings \ - -H "Authorization: Bearer $JWT" \ +# Create user as admin (Role:1), no password needed for OAuth users +curl -sk -X POST "https://portainer.kitestacks.com/api/users" \ + -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ - -d '{ - "AuthenticationMethod": 3, - "OAuthSettings": { - "ClientID": "portainer", - "ClientSecret": "", - "AuthorizationURI": "https://auth.kitestacks.com/application/o/authorize/", - "AccessTokenURI": "https://auth.kitestacks.com/application/o/token/", - "ResourceURI": "https://auth.kitestacks.com/application/o/userinfo/", - "RedirectURI": "https://portainer.kitestacks.com", - "UserIdentifier": "email", - "Scopes": "openid email profile", - "OAuthAutoCreateUsers": true, - "SSO": true, - "LogoutURI": "https://auth.kitestacks.com/application/o/portainer/end-session/" - } - }' + -d '{"username":"user@example.com","role":1}' ``` -**Password reset for monk Portainer** (if admin password is lost): use a Go container with bbolt to patch BoltDB directly — see DEBUG-DOCUMENTATION.md for the full procedure. +### Reset admin password (if locked out) +```bash +# Stop Portainer +docker stop portainer -**kscloud1 Portainer:** admin username `kenpat7177`, password in `.env` (same as OSticket admin). OAuth also configured via API. +# Reset password (shows new temp password) +docker run --rm -v portainer_data:/data portainer/helper-reset-password -### 5.12 Kite AI (LiteLLM + Open WebUI) +# Restart +docker start portainer +``` -`~/kitestacks-live/docker/kite-ai/docker-compose.yml`: +--- +## BookStack + +### Setup (both monk and kscloud1) +Location: +- monk: `~/kitestacks-live/docker/bookstack/docker-compose.yml` +- kscloud1: `/opt/kitestacks/docker/bookstack/docker-compose.yml` + +Key environment variables: ```yaml -services: - litellm: - image: ghcr.io/berriai/litellm:main-latest - container_name: kite-litellm - restart: unless-stopped - ports: - - ":" - command: ["--config", "/app/config.yaml", "--port", ""] - volumes: - - ./litellm_config.yaml:/app/config.yaml - env_file: .env - networks: - - default - - kitestacks - - kite-openwebui: - image: ghcr.io/open-webui/open-webui:main - container_name: kite-openwebui - restart: unless-stopped - ports: - - ":" - environment: - - WEBUI_NAME=Kite AI - - WEBUI_URL=https://ai.kitestacks.com - - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY} - - ENABLE_SIGNUP=false - - ENABLE_OAUTH_SIGNUP=true - - OAUTH_PROVIDER_NAME=Authentik - - OPENID_PROVIDER_URL=https://auth.kitestacks.com/application/o/open-webui/.well-known/openid-configuration - - OAUTH_CLIENT_ID=open-webui - - OAUTH_CLIENT_SECRET=${OPENWEBUI_OAUTH_CLIENT_SECRET} - - OAUTH_SCOPES=openid email profile - - OPENID_REDIRECT_URI=https://ai.kitestacks.com/oauth/oidc/callback - - OPENAI_API_BASE_URL=http://litellm:/v1 - - OPENAI_API_KEY=${LITELLM_MASTER_KEY} - - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true - - ENABLE_OAUTH_PERSISTENT_CONFIG=false - volumes: - - open-webui:/app/backend/data - depends_on: - - litellm - networks: - - default - - kitestacks - -volumes: - open-webui: - -networks: - kitestacks: - external: true +- APP_URL=https://wiki.kitestacks.com +- DB_HOST=bookstack-db +- AUTH_METHOD=oidc +- OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/ +- OIDC_ISSUER_DISCOVER=true +- OIDC_CLIENT_ID=bookstack +- OIDC_CLIENT_SECRET= +- OIDC_USER_ATTRIBUTE=email +- APP_KEY= ``` -`litellm_config.yaml`: - -```yaml -model_list: - - model_name: kite-free - litellm_params: - model: openrouter/openrouter/free - api_key: os.environ/OPENROUTER_API_KEY - - model_name: kite-llama - litellm_params: - model: openrouter/meta-llama/llama-3.1-8b-instruct:free - api_key: os.environ/OPENROUTER_API_KEY - - model_name: kite-deepseek - litellm_params: - model: openrouter/deepseek/deepseek-chat-v3-0324:free - api_key: os.environ/OPENROUTER_API_KEY - -general_settings: - master_key: os.environ/LITELLM_MASTER_KEY -``` - -`.env` keys: `OPENROUTER_API_KEY`, `LITELLM_MASTER_KEY`, `WEBUI_SECRET_KEY`, `OPENWEBUI_OAUTH_CLIENT_SECRET` - -### 5.13 BookStack (Internal Wiki) - -`~/kitestacks-live/docker/bookstack/docker-compose.yml` (internal only, no Cloudflare tunnel): - -```yaml -services: - bookstack: - image: lscr.io/linuxserver/bookstack:latest - container_name: bookstack - restart: unless-stopped - environment: - - PUID=1000 - - PGID=1000 - - TZ=America/Chicago - - APP_URL=http://: - - DB_HOST=bookstack-db - - DB_PORT= - - DB_USER=bookstack - - DB_PASS=${BOOKSTACK_DB_PASS} - - DB_DATABASE=bookstackapp - ports: - - ":" - volumes: - - ./bookstack:/config - depends_on: - - bookstack-db - - bookstack-db: - image: mariadb:11 - container_name: bookstack-db - restart: unless-stopped - environment: - - MYSQL_ROOT_PASSWORD=${BOOKSTACK_DB_ROOT_PASS} - - MYSQL_DATABASE=bookstackapp - - MYSQL_USER=bookstack - - MYSQL_PASSWORD=${BOOKSTACK_DB_PASS} - volumes: - - ./db:/var/lib/mysql -``` - -Access at `http://:` (LAN only). No Cloudflare tunnel — internal wiki only. - ---- - -## Phase 6 — Authentik SSO Configuration - -Log in to `https://auth.kitestacks.com` → Admin Interface. Create one OAuth2/OpenID Provider + Application for each service. - -### General Provider Settings (all providers) - -- Client type: Confidential -- Signing Key: default RS256 key -- Token validity: 10 min access, 30 days refresh -- Scopes: `openid`, `email`, `profile` - -### 6.1 Create Providers - -| App | Client ID | Redirect URI | -|-----|-----------|-------------| -| Grafana | `grafana` | `https://grafana.kitestacks.com/login/generic_oauth` | -| Open WebUI | `open-webui` | `https://ai.kitestacks.com/oauth/oidc/callback` | -| Forgejo | `forgejo` | `https://gitforge.kitestacks.com/user/oauth2/authentik/callback` | -| Karakeep | `karakeep` | `https://links.kitestacks.com/api/auth/callback/custom` | -| Kavita | `kavita` | `https://kavita.kitestacks.com/signin-oidc` | -| Portainer | `portainer` | `https://portainer.kitestacks.com` | - -> **Note:** OpenProject was removed 2026-06-12 and replaced by OSticket. The OpenProject Authentik provider and application have been deleted from the shared DB. OSticket does not currently have an Authentik SSO provider (local auth only). - -> **Auth code TTL:** All OAuth2 providers have `access_code_validity = minutes=10` (bumped from 1 min on 2026-06-12) to prevent `invalid_grant` errors during monk reconnect. The default 60-second window was too short for monk's ~5-minute startup after going offline. - -For each provider, create a matching Application (slug = Client ID, launch URL = `https://.kitestacks.com`). - -### 6.2 Portainer — restrict to homelab-admin group - -After creating the Portainer Application, add a Policy Binding: -- Applications → Portainer → Policy/Group Bindings → Add -- Select group: `homelab-admin` - -### 6.3 Configure Forgejo OAuth source - -In Forgejo Admin → Authentication Sources → Add: -- Type: OAuth2 -- Name: `authentik` -- Provider: OpenID Connect -- Client ID: `forgejo` -- Client Secret: (paste from Authentik) -- Auto Discovery URL: `https://auth.kitestacks.com/application/o/forgejo/.well-known/openid-configuration` -- Additional scopes: `email profile` - -### 6.4 Karakeep — important redirect_uri fix - -Karakeep uses NextAuth.js. The OAuth provider ID is `custom`, so the callback path is `/api/auth/callback/custom`, NOT `/api/auth/callback/authentik`. Ensure the Authentik Karakeep provider's redirect URI is exactly `https://links.kitestacks.com/api/auth/callback/custom`. - -If you need to update this after initial setup via direct Postgres edit: - +### Generate APP_KEY ```bash -# Connect to the shared authentik-postgres on kscloud1 -docker exec -it authentik-postgres psql -U authentik -d authentik -# Extract PG_PASS with: grep PG_PASS .env | cut -d= -f2- -BEGIN; -UPDATE authentik_providers_oauth2_oauth2provider -SET _redirect_uris = '["https://links.kitestacks.com/api/auth/callback/custom"]' -WHERE name = 'Karakeep'; -COMMIT; -# Then restart authentik + authentik-worker on BOTH monk and kscloud1 and wait for healthy +docker run --rm --entrypoint /bin/bash lscr.io/linuxserver/bookstack:latest appkey +``` + +### OIDC Configuration +BookStack uses `OIDC_ISSUER_DISCOVER=true` to auto-discover all endpoints from Authentik. +The `OIDC_ISSUER` must match the per-app discovery URL base (not the global Authentik URL). + +The Authentik bookstack provider must have `issuer_mode='per_provider'` so its discovery +document returns the correct per-app issuer URL. See Debug doc for full troubleshooting. + +### Fix cache permissions after artisan runs +Running `php artisan` as root creates root-owned cache dirs that block the app: +```bash +docker exec bookstack chown -R abc:users /config/www/framework/cache/ +``` + +### Clear Laravel config/cache +```bash +docker exec bookstack php /app/www/artisan config:clear +docker exec bookstack php /app/www/artisan cache:clear ``` --- -## Phase 7 — Deploy Full Replica on kscloud1 - -All 9 service directories live under `/opt/kitestacks/docker/` on kscloud1. The same docker-compose patterns apply, with these differences: - -- OpenProject uses port `:` on host (port is taken by the pre-existing caddy) -- `ENABLE_SIGNUP=true` on Open WebUI (can't SSO if Authentik has no providers yet) -- `FORGEJO_API_BASE=http://:` for metrics-api (monk's Forgejo over Tailscale) -- Authentik on kscloud1 uses the same shared DB (it's the host — localhost resolves fine; use `` for consistency) - -### 7.1 Deploy cloudflared on kscloud1 (2nd connector) - -Same `docker-compose.yml` as monk — same `TUNNEL_TOKEN`. Cloudflare assigns a new connector ID automatically. +## kscloud1 Access +### SSH ```bash -ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@ -cd /opt/kitestacks/docker/cloudflared && docker compose up -d +ssh -i ~/.ssh/id_ed25519_kscloud1 root@ ``` -### 7.2 Deploy all other services +### If SSH key is lost / not working +1. Open Hetzner Cloud console: `console.hetzner.cloud` → your server → Console tab +2. Log in as `root` (Linux user password) +3. Serve the key from monk over Tailscale: + ```bash + # On monk — start temporary HTTP server + cat ~/.ssh/id_ed25519_kscloud1.pub > ~/key.txt + python3 -m http.server 7777 --directory ~/ + ``` +4. In Hetzner console, type: + ```bash + curl http://:7777/key.txt > /root/.ssh/authorized_keys + ``` +5. Enable root SSH (if needed): + ```bash + sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config + systemctl restart ssh + ``` -Follow the same patterns as Phase 5 for each service. Start with authentik (depends on shared Postgres+Redis which is already running), then remaining services in any order. +--- -**OpenProject on kscloud1** — first boot takes ~4–5 minutes (Postgres initdb + Rails migrations). Watch logs: +## OSTicket SMTP + +**Config:** smtp.gmail.com:587, STARTTLS +**From:** `kitestacks.helpdesk@gmail.com` (app password stored in DB) + +To test email delivery: Admin Panel → Diagnostics → Send Test Email + +--- + +## Forgejo + +Runs on monk at `localhost:3006` (port 2222 for SSH git). + +### Generate API token for automation ```bash -docker logs -f openproject 2>&1 | grep -E "Listening|ERROR" -``` - -### 7.3 Sync Authentik DB from kscloud1 to kscloud1 - -The shared DB is already on kscloud1, so kscloud1's own Authentik was bootstrapped with it. No sync needed. Monk's Authentik also uses it over Tailscale. - -### 7.4 Sync Kavita DB from monk to kscloud1 - -The kavita.db must be synced so both connectors show the same users and OIDC settings. Direct file copy produces "database disk image is malformed" due to WAL mode — use sqlite3 backup: - -```bash -# On monk: create a consistent backup -docker run --rm \ - -v ~/kitestacks-live/docker/kavita/config:/data \ - -v /tmp:/out \ - python:3-alpine sh -c " - pip install -q sqlite3-api 2>/dev/null; python3 -c \" -import sqlite3, shutil -src = sqlite3.connect('/data/kavita.db') -dst = sqlite3.connect('/out/kavita-backup.db') -src.backup(dst) -dst.close(); src.close() -\"" - -scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-backup.db kenpat@:/tmp/ - -# On kscloud1: replace kavita.db -ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@ << 'EOF' - cd /opt/kitestacks/docker/kavita - docker compose stop kavita - rm -f config/kavita.db config/kavita.db-wal config/kavita.db-shm - cp /tmp/kavita-backup.db config/kavita.db - chown 1000:1000 config/kavita.db # kavita container runs as root, but 1000:1000 is fine - docker compose start kavita -EOF -``` - -After sync, configure Kavita OIDC via SSH port-forward (NOT direct DB edit): - -```bash -ssh -L 5099:localhost: kenpat@ -# Open http://localhost: in browser -# Settings → OIDC: Authority=https://auth.kitestacks.com/application/o/kavita/ (trailing slash!) -# Client ID: kavita, Secret: (96-char hex from Authentik), Enabled: true -``` - -### 7.5 Sync Kavita cover images from monk to kscloud1 - -```bash -# On monk: -tar czf /tmp/kavita-covers.tar.gz -C ~/kitestacks-live/docker/kavita/config/covers . -scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-covers.tar.gz kenpat@:/tmp/ - -# On kscloud1: -ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@ << 'EOF' - mkdir -p /opt/kitestacks/docker/kavita/config/covers - docker run --rm \ - -v /opt/kitestacks/docker/kavita/config/covers:/covers \ - -v /tmp/kavita-covers.tar.gz:/covers.tar.gz \ - alpine sh -c "cd /covers && tar xzf /covers.tar.gz && chown -R 1000:1000 ." -EOF -``` - -Note: book files themselves are not synced. Cover images load correctly; browsing the library when served by kscloud1 shows entries but no actual files — this is the accepted stale-data tradeoff. - -### 7.6 Grafana provisioning on kscloud1 - -Same provisioning structure as monk. The Prometheus data source on kscloud1 points to kscloud1's own Prometheus (`http://prometheus:`), which only scrapes kscloud1's node-exporter (monk is behind home NAT, not reachable from kscloud1 directly). - -### 7.7 ufw — allow metrics API - -```bash -echo | sudo -S ufw allow from /12 to any port proto tcp +docker exec -u git forgejo forgejo admin user generate-access-token \ + --username kenpat --token-name "my-token" --raw \ + --scopes "read:user,write:user,read:repository,write:repository" ``` --- -## Phase 8 — Portal UI - -The portal lives at `~/kitestacks-live/docker/kitestacks-portal/public/index.html` on monk and must be kept in sync across all 3 copies: - -| Host | Path | -|------|------| -| monk (production) | `~/kitestacks-live/docker/kitestacks-portal/public/index.html` | -| monk (test) | `~/kitestacks-live/docker/kitestacks-portal-test/public/index.html` | -| kscloud1 | `/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html` | - -### Portal Panels - -| Panel | Color | Cards | -|-------|-------|-------| -| INFRASTRUCTURE | cyan | Portainer (live), Authentik, Cloudflare | -| MONITORING | magenta | Grafana, Prometheus (coming-soon), Node Exporter (coming-soon), Uptime Kuma | -| AI & AUTOMATION | purple | Kite AI (coming-soon), LiteLLM (coming-soon), OpenRouter (coming-soon), FluxCD (coming-soon) | -| KNOWLEDGE BASE | pink | Kavita, Karakeep | -| DEVELOPMENT | cyan | Forgejo, OSticket | -| SYSTEM STATUS | — | live metrics from /api/metrics | -| RECENT ACTIVITY | — | last 8 commits from Forgejo via /api/activity | -| COMMUNITY | — | Discord invite link | - -### Update workflow - -Edit `public/index.html` on monk test portal first, verify at `http://localhost:`, then copy to production and kscloud1. +## Common Docker Operations ```bash -# Test change at localhost: first, then: -cp ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html \ - ~/kitestacks-live/docker/kitestacks-portal/public/index.html +# View logs for a service +docker logs --tail 50 -f -scp ~/kitestacks-live/docker/kitestacks-portal/public/index.html \ - kenpat@:/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html +# Restart a service +cd ~/kitestacks-live/docker/ && docker compose restart + +# Full stack restart +docker compose down && docker compose up -d + +# Update a container image +docker compose pull && docker compose up -d + +# Check all running containers +docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" ``` - -No container restarts needed — nginx serves the files directly from the bind-mounted volume. - ---- - -## Phase 9 — Monitoring - -### Prometheus scrapes - -Monk's Prometheus scrapes both: -- `node-exporter:` (monk itself, via Docker DNS) -- `:` (kscloud1, direct public IP — kscloud1's node-exporter is :9100) - -kscloud1's Prometheus only scrapes itself (monk is behind home NAT). - -### Grafana "Node Exporter Full" dashboard - -Dashboard ID 1860 (provisioned from file). Use the `instance` dropdown to switch between `monk` and `kscloud1` views. Both instances appear because Prometheus has both scrape jobs. - ---- - -## Phase 10 — Discord Integration - -### Discord card on portal - -The Community panel on the portal has a Discord card linking to `https://discord.gg/QbdveTb6Kw`. No setup needed beyond keeping the invite valid. - -### Discord #recent-activities push webhook - -A webhook posts to the Discord `#recent-activities` channel when Forgejo receives new commits. This is configured in Forgejo's webhook settings per-repository: - -1. In Forgejo, go to a repo → Settings → Webhooks → Add Webhook → Gitea (compatible format) -2. Target URL: your Discord webhook URL (Discord → channel settings → Integrations → Webhooks → Copy Webhook URL) -3. Content type: `application/json` -4. Events: Push events (select relevant events) -5. Active: enabled - -The webhook URL is stored only in Forgejo's DB — do not commit it to this repo. - ---- - -## Phase 11 — Verification Checklist - -Run after any major change or new host deployment. - -### Public subdomains - -```bash -for sub in www auth gitforge tasks ai links kavita grafana status; do - code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com") - echo "$sub: $code" -done -``` - -Expected: all return 200 (or 301/302 for redirect-based logins). - -### SSO flows - -- [ ] `https://auth.kitestacks.com` → Authentik admin login works -- [ ] `https://grafana.kitestacks.com` → "Sign in with authentik" → lands in Grafana -- [ ] `https://ai.kitestacks.com` → "Sign in with Authentik" → lands in Open WebUI -- [ ] `https://gitforge.kitestacks.com` → "Sign In with authentik" → lands in Forgejo -- [ ] `https://links.kitestacks.com` → Karakeep login with Authentik → works -- [ ] `https://kavita.kitestacks.com` → "Sign in with authentik" → works - -### Failover test (stop monk's cloudflared) - -```bash -docker stop cloudflared -sleep 5 -for sub in www auth gitforge tasks ai links kavita grafana status portainer; do - code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com") - echo "$sub: $code" -done -docker start cloudflared -``` - -All subdomains should return 200/302 (served by kscloud1 alone). - -**Verified 2026-06-16:** www=200, auth=302, status=302, portainer=200 — zero downtime during monk cloudflared outage. kscloud1 took over immediately. - -### Authentik shared DB health - -```bash -# On monk — check authentik can reach shared DB -docker exec authentik sh -c 'python3 -c "import psycopg2; psycopg2.connect(host=\"\", dbname=\"authentik\", user=\"authentik\", password=\"$AUTHENTIK_POSTGRESQL__PASSWORD\")"' && echo "DB reachable" - -# Check both authentik containers are healthy -docker inspect --format '{{.Name}}: {{.State.Health.Status}}' authentik authentik-worker -``` - ---- - -## Phase 12 — FluxCD GitOps Bootstrap (T14s) - -Once K3s and Flux are installed, bootstrap the cluster to the homelab repository. - -```bash -flux bootstrap git \ - --url=ssh://git@localhost:2222/kenpat/kitestacks-homelab.git \ - --branch=main \ - --path=clusters/T14s \ - --private-key-file=$HOME/.ssh/id_ed25519_gitforge -``` - -Flux will now automatically sync the manifests in `apps/kavita/` to the cluster. - ---- - -## Key File Locations Reference - -| What | Path on monk | -|------|-------------| -| All Docker services | `~/kitestacks-live/docker/` | -| Portal HTML (production) | `~/kitestacks-live/docker/kitestacks-portal/public/index.html` | -| Portal HTML (test) | `~/kitestacks-live/docker/kitestacks-portal-test/public/index.html` | -| Metrics API source | `~/kitestacks-live/docker/kitestacks-portal-test/api/main.py` | -| Prometheus config | `~/kitestacks-live/docker/prometheus/prometheus.yml` | -| Grafana provisioning | `~/kitestacks-live/docker/grafana/provisioning/` | -| Book library | `~/kitestacks-live/library/books/` | -| Kavita config | `~/kitestacks-live/docker/kavita/config/` | -| Kavita cover images | `~/kitestacks-live/docker/kavita/config/covers/` | -| Authentik media | `~/kitestacks-live/docker/authentik/media/` | -| Homelab git repo | `~/kitestacks-from-assassin-t14/opt/kitestacks-autosync/kitestacks-homelab/` (or cloned from gitforge.kitestacks.com) | - -| What | Path on kscloud1 | -|------|-----------------| -| All Docker services | `/opt/kitestacks/docker/` | -| Shared Authentik Postgres data | `/opt/kitestacks/docker/authentik/postgres/` | -| Portal HTML | `/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html` | - ---- - -## Upcoming: Oracle Cloud Migration - -kscloud1 (Hetzner) is planned to be replaced by an Oracle Cloud VPS. When the new VPS is ready: - -1. Provision Oracle VPS, install Docker + Tailscale -2. Deploy all services using the same Phase 7 playbook -3. Note: shared Authentik Postgres+Redis must move too — update both monk's and all connectors' `AUTHENTIK_REDIS__HOST` and `AUTHENTIK_POSTGRESQL__HOST` to the new Tailscale IP -4. Add Oracle VPS cloudflared as a new tunnel connector (same TUNNEL_TOKEN) -5. Verify all 9 subdomains work from Oracle VPS -6. Remove kscloud1 cloudflared from the tunnel connectors -7. Decommission kscloud1 Hetzner server - ---- - -## Secrets Inventory (keys only — values in .env files) - -| .env location | Keys | -|---------------|------| -| `authentik/.env` | `AUTHENTIK_SECRET_KEY`, `PG_PASS` | -| `grafana/.env` | `GRAFANA_OAUTH_CLIENT_SECRET` | -| `osticket/.env` | `OSTICKET_DB_PASS`, `OSTICKET_DB_ROOT`, `OSTICKET_ADMIN_PASS`, `OSTICKET_INSTALL_SECRET` | -| `kite-ai/.env` | `OPENROUTER_API_KEY`, `LITELLM_MASTER_KEY`, `WEBUI_SECRET_KEY`, `OPENWEBUI_OAUTH_CLIENT_SECRET` | -| `karakeep/.env` | `KARAKEEP_VERSION`, `NEXTAUTH_SECRET`, `NEXTAUTH_URL`, `MEILI_MASTER_KEY`, `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`, `OAUTH_WELLKNOWN_URL`, `OAUTH_PROVIDER_NAME`, `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING` | -| `cloudflared/docker-compose.yml` | `TUNNEL_TOKEN` (inline) | - -Do not commit `.env` files, webhook URLs, or database files to this repo. - ---- - -## Security Posture -The entire KiteStacks homelab is secured by a Zero Trust architecture: -1. **No Open Inbound Ports:** All public subdomains are routed through Cloudflare Tunnels (edge-to-container). The home router has 0 forwarded ports. -2. **Network Isolation:** Internal communication between nodes (`monk` and `kscloud1`) happens strictly over a WireGuard-based Tailscale mesh network (`100.x.x.x`). -3. **SSO Protection:** Authentik acts as the Identity Provider (OIDC/LDAP) and reverse proxy (Embedded Outpost) protecting all sensitive endpoints (`/scp/` for osTicket, Portainer, Grafana, Kite AI, etc.). -4. **Standalone Bypasses:** The only apps fully public without SSO are the main Portal (`www`) and the read-only FluxCD GitOps Dashboard (`flux`), which was isolated into a standalone Nginx container specifically to decouple it from Authentik. - -## Troubleshooting -For detailed diagnostics, password resets, and specific issue fixes (such as missing osTicket emails, Uptime Kuma connection loops, or Authentik token errors), please see: -[**docs/DEBUGGING.md**](docs/DEBUGGING.md) - -**Important Architecture Note:** The Cloudflare Tunnel is load-balanced between `monk` and `kscloud1`. Any new container (e.g., `ntfy`) MUST be deployed to **both** nodes to prevent random 502 Bad Gateway errors.