kenpat/kitestacks-homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
kitestacks-homelab/RUNBOOK.md
KiteStacks AutoSync 4b8925ca7e security: complete IP, port, and password redaction across all docs 
Redact all remaining IPv4 addresses, port numbers, and credential values
from RUNBOOK.md, AUTHENTIK.md, and authentik-sso-setup.md. Replace with
descriptive placeholders (<IP_REDACTED>, <port>, <REDACTED>, etc.).
Docker image version tags (postgres:16, forgejo:11, etc.) preserved.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 16:16:23 -05:00

42 KiB
Raw Blame History

KiteStacks Homelab — Complete Setup Runbook

Last Updated: 2026-06-11
Status: Production (monk primary, kscloud1 Hetzner cloud replica)
Maintainer: kenpat (kenpat7177@gmail.com)

Architecture Overview

Internet
   │
   └── Cloudflare (DNS + Tunnel)
          │  Active-Active across 3 connectors
          ├── cloudflared on monk (primary home machine)
          ├── cloudflared on kscloud1 (Hetzner VPS, <KSCLOUD1_PUBLIC_IP>)
          └── cloudflared on assassin (T14, currently OFF)

Tailscale overlay network (VPN mesh):
  monk      <MONK_TAILSCALE_IP>
  kscloud1  <KSCLOUD1_TAILSCALE_IP>  ← hosts shared Authentik Postgres + Redis
  assassin  <ASSASSIN_TAILSCALE_IP>    (off)
  pixel-6   <PIXEL6_TAILSCALE_IP>
  samurai   <SAMURAI_TAILSCALE_IP>

Nine public subdomains route through the same Cloudflare Tunnel token. Both monk and kscloud1 are connectors so the site stays up when either goes offline.

Subdomain Container Port
www.kitestacks.com homepage (nginx portal)
auth.kitestacks.com authentik
gitforge.kitestacks.com forgejo
tasks.kitestacks.com openproject
ai.kitestacks.com kite-openwebui
links.kitestacks.com karakeep
kavita.kitestacks.com kavita
grafana.kitestacks.com grafana
status.kitestacks.com uptime-kuma

Important — active-active data model: monk and kscloud1 each run their own copies of all stateful apps (Forgejo, Kavita, OpenProject, etc.) with independent databases. Data is intentionally NOT synced between them (except for Authentik, which shares a single Postgres+Redis on kscloud1 over Tailscale). If kscloud1 serves a request, the user sees kscloud1's database. This is the accepted tradeoff for guaranteed uptime.

Phase 0 — Prerequisites

Accounts & services you need before starting:

  • Domain name (kitestacks.com) registered and nameservers pointed to Cloudflare
  • Cloudflare account (free tier is fine — do NOT enable Zero Trust/Access, which costs money; the Tunnel UI lives under the "Zero Trust" nav but configuring tunnel hostnames is free)
  • Hetzner Cloud account (for kscloud1 VPS)
  • OpenRouter account + API key (for LiteLLM/AI services)
  • Discord server (for community panel + #recent-activities webhook)
  • SSH key pair (~/.ssh/id_ed25519_kscloud1 for kscloud1 access)

Software on monk (local machine):

# Docker (official install script or distro package)
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER   # log out and back in

# Tailscale
curl -fsSL https://tailscale.com/install.sh | sh

Phase 1 — Monk (Primary Host) — Base Setup

Monk is the primary home server. All services live under ~/kitestacks-live/docker/, one directory per app.

1.1 Create the shared Docker network

All containers join this network. cloudflared resolves container names within it for tunnel routing.

docker network create kitestacks

1.2 Directory structure

~/kitestacks-live/
└── docker/
    ├── authentik/
    ├── cloudflared/
    ├── forgejo/
    ├── grafana/
    ├── karakeep/
    ├── kavita/
    ├── kite-ai/
    ├── kitestacks-portal/        # production portal (container: homepage)
    ├── kitestacks-portal-test/   # dev portal + metrics API
    ├── openproject/
    ├── portainer/
    ├── prometheus/
    └── uptime-kuma/

1.3 Tailscale on monk

sudo tailscale up
# Accept the auth link, join tailnet as "monk"
# monk will get Tailscale IP <MONK_TAILSCALE_IP>

Phase 2 — kscloud1 (Hetzner VPS) — Provision & Base Setup

2.1 Provision on Hetzner

  • Server: Hetzner CX22 (or equivalent) — 3 vCPU, 3.7 GB RAM, 75 GB disk
  • OS: Debian/Ubuntu
  • Region: your choice
  • Public IP: <KSCLOUD1_PUBLIC_IP>
  • Add your SSH public key at provision time

2.2 SSH access

ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>

Password for sudo: <KSCLOUD1_SUDO_PASSWORD> (non-interactive sudo: echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S <cmd>)

2.3 Install Docker on kscloud1

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker kenpat
# Log out and back in

2.4 Create the kitestacks Docker network on kscloud1

docker network create kitestacks

All service directories live under /opt/kitestacks/docker/ (same one-dir-per-app pattern as monk).

2.5 Tailscale on kscloud1

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# Join the same tailnet; kscloud1 will get Tailscale IP <KSCLOUD1_TAILSCALE_IP>

2.6 ufw on kscloud1

kscloud1 has ufw active with default-deny. Fix docker-bridge-to-host traffic:

echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from <IP_REDACTED>/12 to any port <port> proto tcp
# Allows homepage metrics API to be reached from within docker containers

Phase 3 — Cloudflare Tunnel Setup

3.1 Create the tunnel

In the Cloudflare dashboard: Zero Trust → Networks → Tunnels → Create a tunnel

  • Type: Cloudflared
  • Name: kitestacks (or any name)
  • Copy the Tunnel Token — this is the TUNNEL_TOKEN used in all cloudflared containers

3.2 Add public hostnames

Still in the tunnel config, add one public hostname per subdomain. Use the container name as the service target (cloudflared resolves these within the kitestacks Docker network):

Public Hostname Service
www.kitestacks.com http://homepage:<port>
auth.kitestacks.com http://authentik:<port>
gitforge.kitestacks.com http://forgejo:<port>
tasks.kitestacks.com http://openproject:<port>
ai.kitestacks.com http://kite-openwebui:<port>
links.kitestacks.com http://karakeep:<port>
kavita.kitestacks.com http://kavita:<port>
grafana.kitestacks.com http://grafana:<port>
status.kitestacks.com http://uptime-kuma:<port>
portainer.kitestacks.com https://portainer:<port> (enable "No TLS Verify")

Note: Portainer uses HTTPS internally. All others are plain HTTP (TLS is handled by Cloudflare at the edge).

Phase 4 — Shared Authentik Database on kscloud1

Authentik on both monk and kscloud1 must share ONE Postgres+Redis to prevent invalid_grant errors. OAuth2 authorization codes are per-DB rows — if /authorize hits one connector and /application/o/token/ hits another, the code is not found. The shared DB lives on kscloud1, bound only to the Tailscale interface.

4.1 Deploy shared Postgres + Redis on kscloud1

/opt/kitestacks/docker/authentik/docker-compose.yml:

services:
  authentik-postgres:
    image: postgres:16
    container_name: authentik-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: authentik
      POSTGRES_PASSWORD: ${PG_PASS}
      POSTGRES_DB: authentik
    ports:
      - "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>"   # Tailscale IP only — not public
    volumes:
      - ./postgres:/var/lib/postgresql/data

  authentik-redis:
    image: redis:alpine
    container_name: authentik-redis
    restart: unless-stopped
    ports:
      - "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>"   # Tailscale IP only — not public
    volumes:
      - redis-data:/data

  authentik:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
      AUTHENTIK_BOOTSTRAP_EMAIL: <BOOTSTRAP_ADMIN_EMAIL>
      AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD}
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks
    depends_on:
      - authentik-postgres
      - authentik-redis

  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    networks:
      - default
      - kitestacks

volumes:
  redis-data:

networks:
  kitestacks:
    external: true

.env keys: PG_PASS, AUTHENTIK_SECRET_KEY, AUTHENTIK_BOOTSTRAP_PASSWORD

Wait ~2 minutes after first boot for Postgres migrations and Authentik startup before testing.

Phase 5 — Deploy Services on Monk

Start with cloudflared first so tunnel connectors register, then deploy services in any order (they all restart-unless-stopped).

5.1 cloudflared

~/kitestacks-live/docker/cloudflared/docker-compose.yml:

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    command: tunnel --no-autoupdate run
    environment:
      - TUNNEL_TOKEN=<your_tunnel_token>
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

cd ~/kitestacks-live/docker/cloudflared && docker compose up -d

5.2 Authentik (monk side — points to shared DB on kscloud1)

~/kitestacks-live/docker/authentik/docker-compose.yml:

services:
  authentik:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks

  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

.env keys: AUTHENTIK_SECRET_KEY (same value as kscloud1), PG_PASS (same value as kscloud1)

Critical: AUTHENTIK_SECRET_KEY and PG_PASS must be IDENTICAL between monk and kscloud1 or shared-DB authentication will fail.

Rollback note: If Tailscale connectivity breaks, monk's authentik will fail to connect to <KSCLOUD1_TAILSCALE_IP>. To roll back: restore a local postgres+redis to monk's compose, docker start authentik-postgres authentik-redis, then docker compose up -d. Do a fresh pg_dump from kscloud1 first to avoid losing any logins made since the shared-DB migration.

5.3 Forgejo

~/kitestacks-live/docker/forgejo/docker-compose.yml:

services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:11
    container_name: forgejo
    restart: unless-stopped
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__DOMAIN=gitforge.kitestacks.com
      - FORGEJO__server__ROOT_URL=https://gitforge.kitestacks.com/
      - FORGEJO__server__SSH_DOMAIN=gitforge.kitestacks.com
      - FORGEJO__server__SSH_PORT=<port>
    ports:
      - "<port>:<port>"
      - "<port>:<port>"
    volumes:
      - ./data:/data
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

5.4 KiteStacks Portal (Production)

The portal is a custom nginx-served static site (cyberpunk-themed). The container is named homepage to match the Cloudflare tunnel route http://homepage:<port>.

~/kitestacks-live/docker/kitestacks-portal/docker-compose.yml:

services:
  homepage:
    image: nginx:alpine
    container_name: homepage
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks
    volumes:
      - ./public:/usr/share/nginx/html:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  kitestacks:
    external: true

nginx.conf:

server {
    listen <port>;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://host.docker.internal:<port>;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 10s;
    }

    location /images/ {
        expires 7d;
        add_header Cache-Control "public, max-age=604800";
    }
}

The /api/ path proxies to the Metrics API (port on the host).

5.5 Metrics API (Portal Backend)

The Metrics API provides real-time system stats (CPU, RAM, storage, network, uptime), weather, and Forgejo recent-activity data to the portal. It runs in network_mode: host with pid: host to read actual host metrics via psutil.

~/kitestacks-live/docker/kitestacks-portal-test/docker-compose.yml (this is the compose that owns the shared metrics-api container):

services:
  kitestacks-portal-test:
    image: nginx:alpine
    container_name: kitestacks-portal-test
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./public:/usr/share/nginx/html:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - metrics-api

  metrics-api:
    build: ./api
    container_name: kitestacks-metrics-api
    restart: unless-stopped
    pid: host
    network_mode: host
    environment:
      - HOST_PROC=/host/proc
      - HOST_SYS=/host/sys
      - HOST_ETC=/host/etc
      - FORGEJO_API_BASE=http://localhost:<port>
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /etc/hostname:/host/etc/hostname:ro
      - /etc/localtime:/host/etc/localtime:ro
      - /:/host:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

api/main.py serves four endpoints:

  • GET /api/metrics — CPU, RAM, storage, network, uptime, system info
  • GET /api/weather — weather via ipapi.co + open-meteo.com (cached 10 min)
  • GET /api/activity — recent Forgejo commits (cached 60s)
  • GET /api/health — {"ok": true}

FORGEJO_API_BASE on monk points to http://localhost:<port> (monk's own Forgejo). On kscloud1 it points to http://<MONK_TAILSCALE_IP>:<port> (monk's Forgejo over Tailscale) so both connectors show consistent recent activity from the same source.

Python deps: fastapi, uvicorn, psutil, httpx

cd ~/kitestacks-live/docker/kitestacks-portal-test && docker compose up -d
cd ~/kitestacks-live/docker/kitestacks-portal && docker compose up -d

5.6 Karakeep (Bookmarks)

~/kitestacks-live/docker/karakeep/docker-compose.yml:

services:
  karakeep:
    image: ghcr.io/karakeep-app/karakeep:${KARAKEEP_VERSION:-release}
    container_name: karakeep
    restart: unless-stopped
    environment:
      - PORT=<port>
      - MEILI_ADDR=http://karakeep-meilisearch:<port>
      - BROWSER_WEB_URL=http://karakeep-chrome:<port>
      - DATA_DIR=/data
    volumes:
      - ./data:/data
    networks:
      - default
      - kitestacks
      - internal

  karakeep-chrome:
    image: gcr.io/zenika-hub/alpine-chrome:124
    container_name: karakeep-chrome
    restart: unless-stopped
    command: chromium-browser --headless --remote-debugging-address=<IP_REDACTED> --remote-debugging-port=<port>
    networks:
      - internal

  karakeep-meilisearch:
    image: getmeili/meilisearch:v1.41.0
    container_name: karakeep-meilisearch
    restart: unless-stopped
    environment:
      - MEILI_NO_ANALYTICS=true
    volumes:
      - ./meilisearch:/meili_data
    networks:
      - internal

networks:
  kitestacks:
    external: true
  internal:

.env keys: KARAKEEP_VERSION, NEXTAUTH_SECRET, NEXTAUTH_URL, MEILI_MASTER_KEY, OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_WELLKNOWN_URL, OAUTH_PROVIDER_NAME, OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING

SSO note for Karakeep: The OAUTH_CLIENT_ID is karakeep. The Authentik redirect URI must be https://links.kitestacks.com/api/auth/callback/custom (NextAuth.js uses provider id custom, NOT authentik — the callback path is /api/auth/callback/custom).

5.7 Kavita (eBook Reader)

~/kitestacks-live/docker/kavita/docker-compose.yml:

services:
  kavita:
    image: ghcr.io/kareadita/kavita:latest
    container_name: kavita
    restart: unless-stopped
    environment:
      - TZ=UTC
    ports:
      - "<port>:<port>"
    volumes:
      - ./config:/kavita/config
      - ../../library/books:/books
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

SSO config for Kavita: Do NOT edit config/kavita.db directly — settings written to the ServerSetting table are overwritten by Kavita on restart. Always configure OIDC through Kavita's Settings UI:

  • Go to http://localhost:<port> (or SSH tunnel ssh -L 5099:localhost:<port> kenpat@<host> for kscloud1)
  • Settings → OIDC:
    • Authority: https://auth.kitestacks.com/application/o/kavita/ (trailing slash required — must exactly match the issuer field in Authentik's discovery doc)
    • Client ID: kavita
    • Client Secret: (96-char hex from Authentik)
    • Enabled: true, ProviderName: authentik

5.8 OpenProject (Task Management)

~/kitestacks-live/docker/openproject/docker-compose.yml:

services:
  openproject:
    image: openproject/openproject:15
    container_name: openproject
    restart: unless-stopped
    environment:
      - OPENPROJECT_SECRET_KEY_BASE=${OPENPROJECT_SECRET_KEY_BASE}
      - OPENPROJECT_HOST__NAME=tasks.kitestacks.com
      - OPENPROJECT_HTTPS=false
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_IDENTIFIER=openproject
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_SECRET=${OPENPROJECT_OIDC_SECRET}
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_DISPLAY__NAME=Authentik
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_SCOPE=["openid","email","profile"]
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_ISSUER=https://auth.kitestacks.com/application/o/openproject/
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_AUTHORIZATION__ENDPOINT=https://auth.kitestacks.com/application/o/authorize/
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_TOKEN__ENDPOINT=https://auth.kitestacks.com/application/o/token/
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_USERINFO__ENDPOINT=https://auth.kitestacks.com/application/o/userinfo/
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_END__SESSION__ENDPOINT=https://auth.kitestacks.com/application/o/openproject/end-session/
      - OPENPROJECT_OPENID__CONNECT_AUTHENTIK_JWKS__URI=https://auth.kitestacks.com/application/o/openproject/jwks/
    ports:
      - "<port>:<port>"
    volumes:
      - ./pgdata:/var/openproject/pgdata
      - openproject_assets:/var/openproject/assets
    networks:
      - default
      - kitestacks

volumes:
  openproject_assets:

networks:
  kitestacks:
    external: true

.env keys: OPENPROJECT_SECRET_KEY_BASE, OPENPROJECT_OIDC_SECRET

Known blocker: OpenProject CE (Community Edition) 15 gates all SSO/OmniAuth strategies behind an Enterprise Edition license. The OIDC config above is correct and the provider record will be seeded in the DB, but the SSO button does NOT appear on /login in CE. Options: purchase EE license, or put a forward-auth proxy (oauth2-proxy or Authentik outpost) in front of OpenProject. Currently deferred.

5.9 Grafana + Prometheus + Node Exporter

~/kitestacks-live/docker/prometheus/docker-compose.yml:

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - default
      - kitestacks

  node-exporter:
    image: prom/node-exporter
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks

volumes:
  prometheus-data:

networks:
  kitestacks:
    external: true

~/kitestacks-live/docker/prometheus/prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "t14-node"
    static_configs:
      - targets: ["node-exporter:<port>"]   # monk (this host)

  - job_name: "kscloud1-node"
    static_configs:
      - targets: ["<KSCLOUD1_PUBLIC_IP>:<port>"]     # kscloud1 (cloud replica)

~/kitestacks-live/docker/grafana/docker-compose.yml:

services:
  grafana:
    image: grafana/grafana-oss
    container_name: grafana
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.kitestacks.com
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_NAME=authentik
      - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
      - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.kitestacks.com/application/o/authorize/
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://authentik:<port>/application/o/token/
      - GF_AUTH_GENERIC_OAUTH_API_URL=http://authentik:<port>/application/o/userinfo/
      - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
      - GF_AUTH_OAUTH_AUTO_LOGIN=false
    volumes:
      - ./data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

.env keys: GRAFANA_OAUTH_CLIENT_SECRET

Grafana provisioning (./provisioning/):

datasources/prometheus.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:<port>
    uid: "000000001"
    isDefault: true
    editable: true

dashboards/dashboard.yml:

apiVersion: 1
providers:
  - name: default
    folder: ''
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

Place node-exporter-full.json (Grafana dashboard ID 1860) in ./provisioning/dashboards/. The dashboard's instance picker dropdown will show both t14-node (monk) and kscloud1-node.

5.10 Uptime Kuma

~/kitestacks-live/docker/uptime-kuma/docker-compose.yml:

services:
  uptime-kuma:
    image: louislam/uptime-kuma:latest
    container_name: uptime-kuma
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - uptime-kuma:/app/data
    networks:
      - default
      - kitestacks

volumes:
  uptime-kuma:

networks:
  kitestacks:
    external: true

Configure monitors in the Uptime Kuma UI for all 9 public subdomains plus internal health checks.

5.11 Portainer

~/kitestacks-live/docker/portainer/docker-compose.yml:

services:
  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - default
      - kitestacks

volumes:
  portainer_data:

networks:
  kitestacks:
    external: true

SSO for Portainer (pending): Authentik OAuth2 provider has been created (Client ID: portainer). Two manual steps remain:

  1. Add portainer.kitestacks.com as a Cloudflare Tunnel public hostname → https://portainer:<port> (No TLS Verify enabled)
  2. In Portainer UI → Settings → Authentication → OAuth → Custom:
    • Client ID: portainer
    • Client Secret: <PORTAINER_OAUTH_CLIENT_SECRET>
    • Authorization URL: https://auth.kitestacks.com/application/o/authorize/
    • Access Token URL: https://auth.kitestacks.com/application/o/token/
    • Resource URL: https://auth.kitestacks.com/application/o/userinfo/
    • Redirect URL: https://portainer.kitestacks.com
    • Logout URL: https://auth.kitestacks.com/application/o/portainer/end-session/
    • Scopes: openid email profile, User identifier: email
  3. After SSO works, update the portal's Portainer card from data-coming-soon="1" to a real href on all 3 portal copies.

Access is restricted to the homelab-admin Authentik group via a PolicyBinding.

5.12 Kite AI (LiteLLM + Open WebUI)

~/kitestacks-live/docker/kite-ai/docker-compose.yml:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: kite-litellm
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    command: ["--config", "/app/config.yaml", "--port", "<port>"]
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    env_file: .env
    networks:
      - default
      - kitestacks

  kite-openwebui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: kite-openwebui
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    environment:
      - WEBUI_NAME=Kite AI
      - WEBUI_URL=https://ai.kitestacks.com
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - ENABLE_SIGNUP=false
      - ENABLE_OAUTH_SIGNUP=true
      - OAUTH_PROVIDER_NAME=Authentik
      - OPENID_PROVIDER_URL=https://auth.kitestacks.com/application/o/open-webui/.well-known/openid-configuration
      - OAUTH_CLIENT_ID=open-webui
      - OAUTH_CLIENT_SECRET=${OPENWEBUI_OAUTH_CLIENT_SECRET}
      - OAUTH_SCOPES=openid email profile
      - OPENID_REDIRECT_URI=https://ai.kitestacks.com/oauth/oidc/callback
      - OPENAI_API_BASE_URL=http://litellm:<port>/v1
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
      - ENABLE_OAUTH_PERSISTENT_CONFIG=false
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - litellm
    networks:
      - default
      - kitestacks

volumes:
  open-webui:

networks:
  kitestacks:
    external: true

litellm_config.yaml:

model_list:
  - model_name: kite-free
    litellm_params:
      model: openrouter/openrouter/free
      api_key: os.environ/OPENROUTER_API_KEY
  - model_name: kite-llama
    litellm_params:
      model: openrouter/meta-llama/llama-3.1-8b-instruct:free
      api_key: os.environ/OPENROUTER_API_KEY
  - model_name: kite-deepseek
    litellm_params:
      model: openrouter/deepseek/deepseek-chat-v3-0324:free
      api_key: os.environ/OPENROUTER_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

.env keys: OPENROUTER_API_KEY, LITELLM_MASTER_KEY, WEBUI_SECRET_KEY, OPENWEBUI_OAUTH_CLIENT_SECRET

5.13 BookStack (Internal Wiki)

~/kitestacks-live/docker/bookstack/docker-compose.yml (internal only, no Cloudflare tunnel):

services:
  bookstack:
    image: lscr.io/linuxserver/bookstack:latest
    container_name: bookstack
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
      - APP_URL=http://<MONK_LAN_IP>:<port>
      - DB_HOST=bookstack-db
      - DB_PORT=<port>
      - DB_USER=bookstack
      - DB_PASS=${BOOKSTACK_DB_PASS}
      - DB_DATABASE=bookstackapp
    ports:
      - "<port>:<port>"
    volumes:
      - ./bookstack:/config
    depends_on:
      - bookstack-db

  bookstack-db:
    image: mariadb:11
    container_name: bookstack-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${BOOKSTACK_DB_ROOT_PASS}
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstack
      - MYSQL_PASSWORD=${BOOKSTACK_DB_PASS}
    volumes:
      - ./db:/var/lib/mysql

Access at http://<MONK_LAN_IP>:<port> (LAN only). No Cloudflare tunnel — internal wiki only.

Phase 6 — Authentik SSO Configuration

Log in to https://auth.kitestacks.com → Admin Interface. Create one OAuth2/OpenID Provider + Application for each service.

General Provider Settings (all providers)

  • Client type: Confidential
  • Signing Key: default RS256 key
  • Token validity: 10 min access, 30 days refresh
  • Scopes: openid, email, profile

6.1 Create Providers

App Client ID Redirect URI
Grafana grafana https://grafana.kitestacks.com/login/generic_oauth
Open WebUI open-webui https://ai.kitestacks.com/oauth/oidc/callback
Forgejo forgejo https://gitforge.kitestacks.com/user/oauth2/authentik/callback
Karakeep karakeep https://links.kitestacks.com/api/auth/callback/custom
Kavita kavita https://kavita.kitestacks.com/api/Account/OIDCCallback
OpenProject openproject https://tasks.kitestacks.com/auth/oidc/callback
Portainer portainer https://portainer.kitestacks.com

For each provider, create a matching Application (slug = Client ID, launch URL = https://<subdomain>.kitestacks.com).

6.2 Portainer — restrict to homelab-admin group

After creating the Portainer Application, add a Policy Binding:

  • Applications → Portainer → Policy/Group Bindings → Add
  • Select group: homelab-admin

6.3 Configure Forgejo OAuth source

In Forgejo Admin → Authentication Sources → Add:

  • Type: OAuth2
  • Name: authentik
  • Provider: OpenID Connect
  • Client ID: forgejo
  • Client Secret: (paste from Authentik)
  • Auto Discovery URL: https://auth.kitestacks.com/application/o/forgejo/.well-known/openid-configuration
  • Additional scopes: email profile

6.4 Karakeep — important redirect_uri fix

Karakeep uses NextAuth.js. The OAuth provider ID is custom, so the callback path is /api/auth/callback/custom, NOT /api/auth/callback/authentik. Ensure the Authentik Karakeep provider's redirect URI is exactly https://links.kitestacks.com/api/auth/callback/custom.

If you need to update this after initial setup via direct Postgres edit:

# Connect to the shared authentik-postgres on kscloud1
docker exec -it authentik-postgres psql -U authentik -d authentik
# Extract PG_PASS with: grep PG_PASS .env | cut -d= -f2-
BEGIN;
UPDATE authentik_providers_oauth2_oauth2provider
SET _redirect_uris = '["https://links.kitestacks.com/api/auth/callback/custom"]'
WHERE name = 'Karakeep';
COMMIT;
# Then restart authentik + authentik-worker on BOTH monk and kscloud1 and wait for healthy

Phase 7 — Deploy Full Replica on kscloud1

All 9 service directories live under /opt/kitestacks/docker/ on kscloud1. The same docker-compose patterns apply, with these differences:

  • OpenProject uses port <port>:<port> on host (port is taken by the pre-existing caddy)
  • ENABLE_SIGNUP=true on Open WebUI (can't SSO if Authentik has no providers yet)
  • FORGEJO_API_BASE=http://<MONK_TAILSCALE_IP>:<port> for metrics-api (monk's Forgejo over Tailscale)
  • Authentik on kscloud1 uses the same shared DB (it's the host — localhost resolves fine; use <KSCLOUD1_TAILSCALE_IP> for consistency)

7.1 Deploy cloudflared on kscloud1 (3rd connector)

Same docker-compose.yml as monk — same TUNNEL_TOKEN. Cloudflare assigns a new connector ID automatically.

ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>
cd /opt/kitestacks/docker/cloudflared && docker compose up -d

7.2 Deploy all other services

Follow the same patterns as Phase 5 for each service. Start with authentik (depends on shared Postgres+Redis which is already running), then remaining services in any order.

OpenProject on kscloud1 — first boot takes ~4–5 minutes (Postgres initdb + Rails migrations). Watch logs:

docker logs -f openproject 2>&1 | grep -E "Listening|ERROR"

7.3 Sync Authentik DB from kscloud1 to kscloud1

The shared DB is already on kscloud1, so kscloud1's own Authentik was bootstrapped with it. No sync needed. Monk's Authentik also uses it over Tailscale.

7.4 Sync Kavita DB from monk to kscloud1

The kavita.db must be synced so both connectors show the same users and OIDC settings. Direct file copy produces "database disk image is malformed" due to WAL mode — use sqlite3 backup:

# On monk: create a consistent backup
docker run --rm \
  -v ~/kitestacks-live/docker/kavita/config:/data \
  -v /tmp:/out \
  python:3-alpine sh -c "
    pip install -q sqlite3-api 2>/dev/null; python3 -c \"
import sqlite3, shutil
src = sqlite3.connect('/data/kavita.db')
dst = sqlite3.connect('/out/kavita-backup.db')
src.backup(dst)
dst.close(); src.close()
\""

scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-backup.db kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/

# On kscloud1: replace kavita.db
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
  cd /opt/kitestacks/docker/kavita
  docker compose stop kavita
  rm -f config/kavita.db config/kavita.db-wal config/kavita.db-shm
  cp /tmp/kavita-backup.db config/kavita.db
  chown 1000:1000 config/kavita.db   # kavita container runs as root, but 1000:1000 is fine
  docker compose start kavita
EOF

After sync, configure Kavita OIDC via SSH port-forward (NOT direct DB edit):

ssh -L 5099:localhost:<port> kenpat@<KSCLOUD1_PUBLIC_IP>
# Open http://localhost:<port> in browser
# Settings → OIDC: Authority=https://auth.kitestacks.com/application/o/kavita/ (trailing slash!)
# Client ID: kavita, Secret: (96-char hex from Authentik), Enabled: true

7.5 Sync Kavita cover images from monk to kscloud1

# On monk:
tar czf /tmp/kavita-covers.tar.gz -C ~/kitestacks-live/docker/kavita/config/covers .
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-covers.tar.gz kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/

# On kscloud1:
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
  mkdir -p /opt/kitestacks/docker/kavita/config/covers
  docker run --rm \
    -v /opt/kitestacks/docker/kavita/config/covers:/covers \
    -v /tmp/kavita-covers.tar.gz:/covers.tar.gz \
    alpine sh -c "cd /covers && tar xzf /covers.tar.gz && chown -R 1000:1000 ."
EOF

Note: book files themselves are not synced. Cover images load correctly; browsing the library when served by kscloud1 shows entries but no actual files — this is the accepted stale-data tradeoff.

7.6 Grafana provisioning on kscloud1

Same provisioning structure as monk. The Prometheus data source on kscloud1 points to kscloud1's own Prometheus (http://prometheus:<port>), which only scrapes kscloud1's node-exporter (monk is behind home NAT, not reachable from kscloud1 directly).

7.7 ufw — allow metrics API

echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from <IP_REDACTED>/12 to any port <port> proto tcp

Phase 8 — Portal UI

The portal lives at ~/kitestacks-live/docker/kitestacks-portal/public/index.html on monk and must be kept in sync across all 3 copies:

Host Path
monk (production) ~/kitestacks-live/docker/kitestacks-portal/public/index.html
monk (test) ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html
kscloud1 /opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html

Portal Panels

Panel Color Cards
INFRASTRUCTURE cyan Portainer (coming-soon), Authentik, Cloudflare
MONITORING magenta Grafana, Prometheus (coming-soon), Node Exporter (coming-soon), Uptime Kuma
AI & AUTOMATION purple Kite AI (coming-soon), LiteLLM (coming-soon), OpenRouter (coming-soon), FluxCD (coming-soon)
KNOWLEDGE BASE pink Kavita, Karakeep
DEVELOPMENT cyan Forgejo, OpenProject
SYSTEM STATUS — live metrics from /api/metrics
RECENT ACTIVITY — last 8 commits from Forgejo via /api/activity
COMMUNITY — Discord invite link

Update workflow

Edit public/index.html on monk test portal first, verify at http://localhost:<port>, then copy to production and kscloud1.

# Test change at localhost:<port> first, then:
cp ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html \
   ~/kitestacks-live/docker/kitestacks-portal/public/index.html

scp ~/kitestacks-live/docker/kitestacks-portal/public/index.html \
    kenpat@<KSCLOUD1_PUBLIC_IP>:/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html

No container restarts needed — nginx serves the files directly from the bind-mounted volume.

Phase 9 — Monitoring

Prometheus scrapes

Monk's Prometheus scrapes both:

  • node-exporter:<port> (monk itself, via Docker DNS)
  • <KSCLOUD1_PUBLIC_IP>:<port> (kscloud1, direct public IP — kscloud1's node-exporter is <IP_REDACTED>:9100)

kscloud1's Prometheus only scrapes itself (monk is behind home NAT).

Grafana "Node Exporter Full" dashboard

Dashboard ID 1860 (provisioned from file). Use the instance dropdown to switch between monk and kscloud1 views. Both instances appear because Prometheus has both scrape jobs.

Phase 10 — Discord Integration

Discord card on portal

The Community panel on the portal has a Discord card linking to https://discord.gg/QbdveTb6Kw. No setup needed beyond keeping the invite valid.

Discord #recent-activities push webhook

A webhook posts to the Discord #recent-activities channel when Forgejo receives new commits. This is configured in Forgejo's webhook settings per-repository:

  1. In Forgejo, go to a repo → Settings → Webhooks → Add Webhook → Gitea (compatible format)
  2. Target URL: your Discord webhook URL (Discord → channel settings → Integrations → Webhooks → Copy Webhook URL)
  3. Content type: application/json
  4. Events: Push events (select relevant events)
  5. Active: enabled

The webhook URL is stored only in Forgejo's DB — do not commit it to this repo.

Phase 11 — Verification Checklist

Run after any major change or new host deployment.

Public subdomains

for sub in www auth gitforge tasks ai links kavita grafana status; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com")
  echo "$sub: $code"
done

Expected: all return 200 (or 301/302 for redirect-based logins).

SSO flows

  • https://auth.kitestacks.com → Authentik admin login works
  • https://grafana.kitestacks.com → "Sign in with authentik" → lands in Grafana
  • https://ai.kitestacks.com → "Sign in with Authentik" → lands in Open WebUI
  • https://gitforge.kitestacks.com → "Sign In with authentik" → lands in Forgejo
  • https://links.kitestacks.com → Karakeep login with Authentik → works
  • https://kavita.kitestacks.com → "Sign in with authentik" → works

Failover test (disconnect monk's internet)

With monk's home network off (phone hotspot or at a different location):

for sub in www auth gitforge tasks ai links kavita grafana status; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com")
  echo "$sub: $code"
done

All 9 should still return 200 (served by kscloud1).

Authentik shared DB health

# On monk — check authentik can reach shared DB
docker exec authentik sh -c 'python3 -c "import psycopg2; psycopg2.connect(host=\"<KSCLOUD1_TAILSCALE_IP>\", dbname=\"authentik\", user=\"authentik\", password=\"$AUTHENTIK_POSTGRESQL__PASSWORD\")"' && echo "DB reachable"

# Check both authentik containers are healthy
docker inspect --format '{{.Name}}: {{.State.Health.Status}}' authentik authentik-worker

Key File Locations Reference

What Path on monk
All Docker services ~/kitestacks-live/docker/
Portal HTML (production) ~/kitestacks-live/docker/kitestacks-portal/public/index.html
Portal HTML (test) ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html
Metrics API source ~/kitestacks-live/docker/kitestacks-portal-test/api/main.py
Prometheus config ~/kitestacks-live/docker/prometheus/prometheus.yml
Grafana provisioning ~/kitestacks-live/docker/grafana/provisioning/
Book library ~/kitestacks-live/library/books/
Kavita config ~/kitestacks-live/docker/kavita/config/
Kavita cover images ~/kitestacks-live/docker/kavita/config/covers/
Authentik media ~/kitestacks-live/docker/authentik/media/
Homelab git repo ~/kitestacks-from-assassin-t14/opt/kitestacks-autosync/kitestacks-homelab/ (or cloned from gitforge.kitestacks.com)
What Path on kscloud1
All Docker services /opt/kitestacks/docker/
Shared Authentik Postgres data /opt/kitestacks/docker/authentik/postgres/
Portal HTML /opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html

Upcoming: Oracle Cloud Migration

kscloud1 (Hetzner) is planned to be replaced by an Oracle Cloud VPS. When the new VPS is ready:

  1. Provision Oracle VPS, install Docker + Tailscale
  2. Deploy all services using the same Phase 7 playbook
  3. Note: shared Authentik Postgres+Redis must move too — update both monk's and all connectors' AUTHENTIK_REDIS__HOST and AUTHENTIK_POSTGRESQL__HOST to the new Tailscale IP
  4. Add Oracle VPS cloudflared as a new tunnel connector (same TUNNEL_TOKEN)
  5. Verify all 9 subdomains work from Oracle VPS
  6. Remove kscloud1 cloudflared from the tunnel connectors
  7. Decommission kscloud1 Hetzner server

Secrets Inventory (keys only — values in .env files)

.env location Keys
authentik/.env AUTHENTIK_SECRET_KEY, PG_PASS
grafana/.env GRAFANA_OAUTH_CLIENT_SECRET
openproject/.env OPENPROJECT_OIDC_SECRET, OPENPROJECT_SECRET_KEY_BASE
kite-ai/.env OPENROUTER_API_KEY, LITELLM_MASTER_KEY, WEBUI_SECRET_KEY, OPENWEBUI_OAUTH_CLIENT_SECRET
karakeep/.env KARAKEEP_VERSION, NEXTAUTH_SECRET, NEXTAUTH_URL, MEILI_MASTER_KEY, OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_WELLKNOWN_URL, OAUTH_PROVIDER_NAME, OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING
cloudflared/docker-compose.yml TUNNEL_TOKEN (inline)

Do not commit .env files, webhook URLs, or database files to this repo.