kenpat/kitestacks-homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
kitestacks-homelab/RUNBOOK.md

48 KiB
Raw Blame History

KiteStacks Homelab — Complete Setup Runbook

Last Updated: 2026-06-12
Status: Production (monk primary, kscloud1 Hetzner cloud replica)
Maintainer: kenpat (kenpat7177@gmail.com)

Architecture Overview

Internet
   │
   └── Cloudflare (DNS + Tunnel)
          │  Active-Active across 3 connectors
          ├── cloudflared on monk (primary home machine)
          ├── cloudflared on kscloud1 (Hetzner VPS, <KSCLOUD1_PUBLIC_IP>)
          └── cloudflared on T14s (currently OFF)

Tailscale overlay network (VPN mesh):
  monk      <MONK_TAILSCALE_IP>
  kscloud1  <KSCLOUD1_TAILSCALE_IP>  ← hosts shared Authentik Postgres + Redis
  T14s      <T14S_TAILSCALE_IP>    (off)
  pixel-6   <PIXEL6_TAILSCALE_IP>
  samurai   <SAMURAI_TAILSCALE_IP>

Nine public subdomains route through the same Cloudflare Tunnel token. Both monk and kscloud1 are connectors so the site stays up when either goes offline.

Subdomain Container Port
www.kitestacks.com homepage (nginx portal)
auth.kitestacks.com authentik
gitforge.kitestacks.com forgejo
tasks.kitestacks.com osticket (nginx proxy → osticket-app) 8080
portainer.kitestacks.com portainer 9443 (HTTPS)
ai.kitestacks.com kite-openwebui
links.kitestacks.com karakeep
kavita.kitestacks.com kavita
grafana.kitestacks.com grafana
status.kitestacks.com uptime-kuma

Important — active-active data model: monk and kscloud1 each run their own copies of all stateful apps (Forgejo, Kavita, OSticket, etc.) with independent databases. Data is intentionally NOT synced between them (except for Authentik, which shares a single Postgres+Redis on kscloud1 over Tailscale). If kscloud1 serves a request, the user sees kscloud1's database. This is the accepted tradeoff for guaranteed uptime.

Phase 0 — Prerequisites

Accounts & services you need before starting:

  • Domain name (kitestacks.com) registered and nameservers pointed to Cloudflare
  • Cloudflare account (free tier is fine — do NOT enable Zero Trust/Access, which costs money; the Tunnel UI lives under the "Zero Trust" nav but configuring tunnel hostnames is free)
  • Hetzner Cloud account (for kscloud1 VPS)
  • OpenRouter account + API key (for LiteLLM/AI services)
  • Discord server (for community panel + #recent-activities webhook)
  • SSH key pair (~/.ssh/id_ed25519_kscloud1 for kscloud1 access)

Software on T14s (GitOps Node):

# K3s (Kubernetes)
curl -sfL https://get.k3s.io | sh -
sudo chmod 644 /etc/rancher/k3s/k3s.yaml
mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

# Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash

Docker (official install script or distro package)

curl -fsSL https://get.docker.com | sh sudo usermod -aG docker $USER # log out and back in

Tailscale

curl -fsSL https://tailscale.com/install.sh | sh


---

## Phase 1 — Monk (Primary Host) — Base Setup

Monk is the primary home server. All services live under `~/kitestacks-live/docker/`, one directory per app.

### 1.1 Create the shared Docker network

All containers join this network. `cloudflared` resolves container names within it for tunnel routing.

```bash
docker network create kitestacks

1.2 Directory structure

~/kitestacks-live/
└── docker/
    ├── authentik/
    ├── cloudflared/
    ├── forgejo/
    ├── grafana/
    ├── karakeep/
    ├── kavita/
    ├── kite-ai/
    ├── kitestacks-portal/        # production portal (container: homepage)
    ├── kitestacks-portal-test/   # dev portal + metrics API
    ├── openproject/
    ├── portainer/
    ├── prometheus/
    └── uptime-kuma/

1.3 Tailscale on monk

sudo tailscale up
# Accept the auth link, join tailnet as "monk"
# monk will get Tailscale IP <MONK_TAILSCALE_IP>

Phase 2 — kscloud1 (Hetzner VPS) — Provision & Base Setup

2.1 Provision on Hetzner

  • Server: Hetzner CX22 (or equivalent) — 3 vCPU, 3.7 GB RAM, 75 GB disk
  • OS: Debian/Ubuntu
  • Region: your choice
  • Public IP: <KSCLOUD1_PUBLIC_IP>
  • Add your SSH public key at provision time

2.2 SSH access

ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>

Password for sudo: <KSCLOUD1_SUDO_PASSWORD> (non-interactive sudo: echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S <cmd>)

2.3 Install Docker on kscloud1

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker kenpat
# Log out and back in

2.4 Create the kitestacks Docker network on kscloud1

docker network create kitestacks

All service directories live under /opt/kitestacks/docker/ (same one-dir-per-app pattern as monk).

2.5 Tailscale on kscloud1

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# Join the same tailnet; kscloud1 will get Tailscale IP <KSCLOUD1_TAILSCALE_IP>

2.6 ufw on kscloud1

kscloud1 has ufw active with default-deny. Fix docker-bridge-to-host traffic:

echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from <IP_REDACTED>/12 to any port <port> proto tcp
# Allows homepage metrics API to be reached from within docker containers

Phase 3 — Cloudflare Tunnel Setup

3.1 Create the tunnel

In the Cloudflare dashboard: Zero Trust → Networks → Tunnels → Create a tunnel

  • Type: Cloudflared
  • Name: kitestacks (or any name)
  • Copy the Tunnel Token — this is the TUNNEL_TOKEN used in all cloudflared containers

3.2 Add public hostnames

Still in the tunnel config, add one public hostname per subdomain. Use the container name as the service target (cloudflared resolves these within the kitestacks Docker network):

Public Hostname Service
www.kitestacks.com http://homepage:<port>
auth.kitestacks.com http://authentik:<port>
gitforge.kitestacks.com http://forgejo:<port>
tasks.kitestacks.com http://osticket:8080
portainer.kitestacks.com https://portainer:9443 (enable "No TLS Verify")
ai.kitestacks.com http://kite-openwebui:<port>
links.kitestacks.com http://karakeep:<port>
kavita.kitestacks.com http://kavita:<port>
grafana.kitestacks.com http://grafana:<port>
status.kitestacks.com http://uptime-kuma:<port>
portainer.kitestacks.com https://portainer:<port> (enable "No TLS Verify")

Note: Portainer uses HTTPS internally. All others are plain HTTP (TLS is handled by Cloudflare at the edge).

Phase 4 — Shared Authentik Database on kscloud1

Authentik on both monk and kscloud1 must share ONE Postgres+Redis to prevent invalid_grant errors. OAuth2 authorization codes are per-DB rows — if /authorize hits one connector and /application/o/token/ hits another, the code is not found. The shared DB lives on kscloud1, bound only to the Tailscale interface.

4.1 Deploy shared Postgres + Redis on kscloud1

/opt/kitestacks/docker/authentik/docker-compose.yml:

services:
  authentik-postgres:
    image: postgres:16
    container_name: authentik-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: authentik
      POSTGRES_PASSWORD: ${PG_PASS}
      POSTGRES_DB: authentik
    ports:
      - "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>"   # Tailscale IP only — not public
    volumes:
      - ./postgres:/var/lib/postgresql/data

  authentik-redis:
    image: redis:alpine
    container_name: authentik-redis
    restart: unless-stopped
    ports:
      - "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>"   # Tailscale IP only — not public
    volumes:
      - redis-data:/data

  authentik:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
      AUTHENTIK_BOOTSTRAP_EMAIL: <BOOTSTRAP_ADMIN_EMAIL>
      AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD}
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks
    depends_on:
      - authentik-postgres
      - authentik-redis

  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    networks:
      - default
      - kitestacks

volumes:
  redis-data:

networks:
  kitestacks:
    external: true

.env keys: PG_PASS, AUTHENTIK_SECRET_KEY, AUTHENTIK_BOOTSTRAP_PASSWORD

Wait ~2 minutes after first boot for Postgres migrations and Authentik startup before testing.

Phase 5 — Deploy Services on Monk

Start with cloudflared first so tunnel connectors register, then deploy services in any order (they all restart-unless-stopped).

5.1 cloudflared

~/kitestacks-live/docker/cloudflared/docker-compose.yml:

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    command: tunnel --no-autoupdate run
    environment:
      - TUNNEL_TOKEN=<your_tunnel_token>
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

cd ~/kitestacks-live/docker/cloudflared && docker compose up -d

5.2 Authentik (monk side — points to shared DB on kscloud1)

~/kitestacks-live/docker/authentik/docker-compose.yml:

services:
  authentik:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks

  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

.env keys: AUTHENTIK_SECRET_KEY (same value as kscloud1), PG_PASS (same value as kscloud1)

Critical: AUTHENTIK_SECRET_KEY and PG_PASS must be IDENTICAL between monk and kscloud1 or shared-DB authentication will fail.

Rollback note: If Tailscale connectivity breaks, monk's authentik will fail to connect to <KSCLOUD1_TAILSCALE_IP>. To roll back: restore a local postgres+redis to monk's compose, docker start authentik-postgres authentik-redis, then docker compose up -d. Do a fresh pg_dump from kscloud1 first to avoid losing any logins made since the shared-DB migration.

5.3 Forgejo

~/kitestacks-live/docker/forgejo/docker-compose.yml:

services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:11
    container_name: forgejo
    restart: unless-stopped
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__DOMAIN=gitforge.kitestacks.com
      - FORGEJO__server__ROOT_URL=https://gitforge.kitestacks.com/
      - FORGEJO__server__SSH_DOMAIN=gitforge.kitestacks.com
      - FORGEJO__server__SSH_PORT=<port>
    ports:
      - "<port>:<port>"
      - "<port>:<port>"
    volumes:
      - ./data:/data
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

5.4 KiteStacks Portal (Production)

The portal is a custom nginx-served static site (cyberpunk-themed). The container is named homepage to match the Cloudflare tunnel route http://homepage:<port>.

~/kitestacks-live/docker/kitestacks-portal/docker-compose.yml:

services:
  homepage:
    image: nginx:alpine
    container_name: homepage
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks
    volumes:
      - ./public:/usr/share/nginx/html:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  kitestacks:
    external: true

nginx.conf:

server {
    listen <port>;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://host.docker.internal:<port>;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 10s;
    }

    location /images/ {
        expires 7d;
        add_header Cache-Control "public, max-age=604800";
    }
}

The /api/ path proxies to the Metrics API (port on the host).

5.5 Metrics API (Portal Backend)

The Metrics API provides real-time system stats (CPU, RAM, storage, network, uptime), weather, and Forgejo recent-activity data to the portal. It runs in network_mode: host with pid: host to read actual host metrics via psutil.

~/kitestacks-live/docker/kitestacks-portal-test/docker-compose.yml (this is the compose that owns the shared metrics-api container):

services:
  kitestacks-portal-test:
    image: nginx:alpine
    container_name: kitestacks-portal-test
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./public:/usr/share/nginx/html:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - metrics-api

  metrics-api:
    build: ./api
    container_name: kitestacks-metrics-api
    restart: unless-stopped
    pid: host
    network_mode: host
    environment:
      - HOST_PROC=/host/proc
      - HOST_SYS=/host/sys
      - HOST_ETC=/host/etc
      - FORGEJO_API_BASE=http://localhost:<port>
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /etc/hostname:/host/etc/hostname:ro
      - /etc/localtime:/host/etc/localtime:ro
      - /:/host:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

api/main.py serves four endpoints:

  • GET /api/metrics — CPU, RAM, storage, network, uptime, system info
  • GET /api/weather — weather via ipapi.co + open-meteo.com (cached 10 min)
  • GET /api/activity — recent Forgejo commits (cached 60s)
  • GET /api/health — {"ok": true}

FORGEJO_API_BASE on monk points to http://localhost:<port> (monk's own Forgejo). On kscloud1 it points to http://<MONK_TAILSCALE_IP>:<port> (monk's Forgejo over Tailscale) so both connectors show consistent recent activity from the same source.

Python deps: fastapi, uvicorn, psutil, httpx

cd ~/kitestacks-live/docker/kitestacks-portal-test && docker compose up -d
cd ~/kitestacks-live/docker/kitestacks-portal && docker compose up -d

5.6 Karakeep (Bookmarks)

~/kitestacks-live/docker/karakeep/docker-compose.yml:

services:
  karakeep:
    image: ghcr.io/karakeep-app/karakeep:${KARAKEEP_VERSION:-release}
    container_name: karakeep
    restart: unless-stopped
    environment:
      - PORT=<port>
      - MEILI_ADDR=http://karakeep-meilisearch:<port>
      - BROWSER_WEB_URL=http://karakeep-chrome:<port>
      - DATA_DIR=/data
    volumes:
      - ./data:/data
    networks:
      - default
      - kitestacks
      - internal

  karakeep-chrome:
    image: gcr.io/zenika-hub/alpine-chrome:124
    container_name: karakeep-chrome
    restart: unless-stopped
    command: chromium-browser --headless --remote-debugging-address=<IP_REDACTED> --remote-debugging-port=<port>
    networks:
      - internal

  karakeep-meilisearch:
    image: getmeili/meilisearch:v1.41.0
    container_name: karakeep-meilisearch
    restart: unless-stopped
    environment:
      - MEILI_NO_ANALYTICS=true
    volumes:
      - ./meilisearch:/meili_data
    networks:
      - internal

networks:
  kitestacks:
    external: true
  internal:

.env keys: KARAKEEP_VERSION, NEXTAUTH_SECRET, NEXTAUTH_URL, MEILI_MASTER_KEY, OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_WELLKNOWN_URL, OAUTH_PROVIDER_NAME, OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING

SSO note for Karakeep: The OAUTH_CLIENT_ID is karakeep. The Authentik redirect URI must be https://links.kitestacks.com/api/auth/callback/custom (NextAuth.js uses provider id custom, NOT authentik — the callback path is /api/auth/callback/custom).

5.7 Kavita (eBook Reader)

~/kitestacks-live/docker/kavita/docker-compose.yml:

services:
  kavita:
    image: ghcr.io/kareadita/kavita:latest
    container_name: kavita
    restart: unless-stopped
    environment:
      - TZ=UTC
    ports:
      - "<port>:<port>"
    volumes:
      - ./config:/kavita/config
      - ../../library/books:/books
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

SSO config for Kavita: Do NOT edit config/kavita.db directly — settings written to the ServerSetting table are overwritten by Kavita on restart. Always configure OIDC through Kavita's Settings UI:

  • Go to http://localhost:<port> (or SSH tunnel ssh -L 5099:localhost:<port> kenpat@<host> for kscloud1)
  • Settings → OIDC:
    • Authority: https://auth.kitestacks.com/application/o/kavita/ (trailing slash required — must exactly match the issuer field in Authentik's discovery doc)
    • Client ID: kavita
    • Client Secret: (96-char hex from Authentik)
    • Enabled: true, ProviderName: authentik

5.8 OSticket (Help Desk) — replaced OpenProject 2026-06-12

OSticket is a PHP-based help desk ticketing system. It runs behind an nginx proxy container that Cloudflare Tunnel reaches at http://osticket:8080.

Architecture: osticket (nginx:alpine, port 8080) → osticket-app (campbellsoftwaresolutions/osticket, port 80) + osticket-db (mariadb:10.11)

~/kitestacks-live/docker/osticket/docker-compose.yml:

services:
  osticket:
    image: nginx:alpine
    container_name: osticket
    restart: unless-stopped
    depends_on:
      - osticket-app
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    ports:
      - "<port>:8080"
    networks:
      - default
      - kitestacks

  osticket-app:
    image: campbellsoftwaresolutions/osticket
    container_name: osticket-app
    restart: unless-stopped
    depends_on:
      - osticket-db
    environment:
      - INSTALL_NAME=KiteStacks Help Desk
      - INSTALL_EMAIL=helpdesk@kitestacks.com
      - INSTALL_URL=https://tasks.kitestacks.com/
      - ADMIN_FIRSTNAME=Ken
      - ADMIN_LASTNAME=Pat
      - ADMIN_EMAIL=<your_email>
      - ADMIN_USERNAME=<your_username>
      - ADMIN_PASSWORD=${OSTICKET_ADMIN_PASS}
      - INSTALL_SECRET=${OSTICKET_INSTALL_SECRET}
      - MYSQL_HOST=osticket-db
      - MYSQL_DATABASE=osticket
      - MYSQL_USER=osticket
      - MYSQL_PASSWORD=${OSTICKET_DB_PASS}
    volumes:
      - osticket_uploads:/data/upload/include/i18n
    networks:
      - default

  osticket-db:
    image: mariadb:10.11
    container_name: osticket-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${OSTICKET_DB_ROOT}
      - MYSQL_DATABASE=osticket
      - MYSQL_USER=osticket
      - MYSQL_PASSWORD=${OSTICKET_DB_PASS}
    volumes:
      - osticket_db:/var/lib/mysql
    networks:
      - default

volumes:
  osticket_uploads:
  osticket_db:

networks:
  kitestacks:
    external: true

nginx.conf:

server {
    listen 8080;
    location / {
        proxy_pass http://osticket-app:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Host $host;
        proxy_read_timeout 120s;
    }
}

.env keys: OSTICKET_DB_PASS, OSTICKET_DB_ROOT, OSTICKET_ADMIN_PASS, OSTICKET_INSTALL_SECRET

First boot: The install script runs automatically, connects to MariaDB, and seeds the database. Wait for Database installation successful in docker logs osticket-app before testing. The same .env values are used on kscloud1 so both instances share the same admin credentials (separate databases).

Cloudflare Tunnel: Route tasks.kitestacks.com → http://osticket:8080 in the tunnel config.

kscloud1 note: Same compose, host port <port>:8080 (8090 is used since port 80 is taken by caddy).

5.9 Grafana + Prometheus + Node Exporter

~/kitestacks-live/docker/prometheus/docker-compose.yml:

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - default
      - kitestacks

  node-exporter:
    image: prom/node-exporter
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks

volumes:
  prometheus-data:

networks:
  kitestacks:
    external: true

~/kitestacks-live/docker/prometheus/prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "t14-node"
    static_configs:
      - targets: ["node-exporter:<port>"]   # monk (this host)

  - job_name: "kscloud1-node"
    static_configs:
      - targets: ["<KSCLOUD1_PUBLIC_IP>:<port>"]     # kscloud1 (cloud replica)

~/kitestacks-live/docker/grafana/docker-compose.yml:

services:
  grafana:
    image: grafana/grafana-oss
    container_name: grafana
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.kitestacks.com
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_NAME=authentik
      - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
      - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.kitestacks.com/application/o/authorize/
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://authentik:<port>/application/o/token/
      - GF_AUTH_GENERIC_OAUTH_API_URL=http://authentik:<port>/application/o/userinfo/
      - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
      - GF_AUTH_OAUTH_AUTO_LOGIN=false
    volumes:
      - ./data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true

.env keys: GRAFANA_OAUTH_CLIENT_SECRET

Grafana provisioning (./provisioning/):

datasources/prometheus.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:<port>
    uid: "000000001"
    isDefault: true
    editable: true

dashboards/dashboard.yml:

apiVersion: 1
providers:
  - name: default
    folder: ''
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

Place node-exporter-full.json (Grafana dashboard ID 1860) in ./provisioning/dashboards/. The dashboard's instance picker dropdown will show both t14-node (monk) and kscloud1-node.

5.10 Uptime Kuma

~/kitestacks-live/docker/uptime-kuma/docker-compose.yml:

services:
  uptime-kuma:
    image: louislam/uptime-kuma:latest
    container_name: uptime-kuma
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - uptime-kuma:/app/data
    networks:
      - default
      - kitestacks

volumes:
  uptime-kuma:

networks:
  kitestacks:
    external: true

Configure monitors in the Uptime Kuma UI for all 9 public subdomains plus internal health checks.

5.11 Portainer

~/kitestacks-live/docker/portainer/docker-compose.yml:

services:
  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - default
      - kitestacks

volumes:
  portainer_data:

networks:
  kitestacks:
    external: true

SSO — CONFIGURED (2026-06-12): Authentik OAuth2 provider (Client ID: portainer) is live. OAuth configured via the Portainer API on both monk and kscloud1. Portal card updated to a live link on all 3 copies.

  • Cloudflare Tunnel: portainer.kitestacks.com → https://portainer:9443 (No TLS Verify) — add this in CF dashboard if not yet present
  • Access restricted to homelab-admin Authentik group via PolicyBinding

To re-configure OAuth via Portainer API (if settings get reset):

JWT=$(curl -sk -X POST https://localhost:9443/api/auth \
  -H "Content-Type: application/json" \
  -d '{"Username":"admin","Password":"<PORTAINER_ADMIN_PASSWORD>"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")

curl -sk -X PUT https://localhost:9443/api/settings \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "AuthenticationMethod": 3,
    "OAuthSettings": {
      "ClientID": "portainer",
      "ClientSecret": "<PORTAINER_OAUTH_CLIENT_SECRET>",
      "AuthorizationURI": "https://auth.kitestacks.com/application/o/authorize/",
      "AccessTokenURI": "https://auth.kitestacks.com/application/o/token/",
      "ResourceURI": "https://auth.kitestacks.com/application/o/userinfo/",
      "RedirectURI": "https://portainer.kitestacks.com",
      "UserIdentifier": "email",
      "Scopes": "openid email profile",
      "OAuthAutoCreateUsers": true,
      "SSO": true,
      "LogoutURI": "https://auth.kitestacks.com/application/o/portainer/end-session/"
    }
  }'

Password reset for monk Portainer (if admin password is lost): use a Go container with bbolt to patch BoltDB directly — see DEBUG-DOCUMENTATION.md for the full procedure.

kscloud1 Portainer: admin username kenpat7177, password in .env (same as OSticket admin). OAuth also configured via API.

5.12 Kite AI (LiteLLM + Open WebUI)

~/kitestacks-live/docker/kite-ai/docker-compose.yml:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: kite-litellm
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    command: ["--config", "/app/config.yaml", "--port", "<port>"]
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    env_file: .env
    networks:
      - default
      - kitestacks

  kite-openwebui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: kite-openwebui
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    environment:
      - WEBUI_NAME=Kite AI
      - WEBUI_URL=https://ai.kitestacks.com
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - ENABLE_SIGNUP=false
      - ENABLE_OAUTH_SIGNUP=true
      - OAUTH_PROVIDER_NAME=Authentik
      - OPENID_PROVIDER_URL=https://auth.kitestacks.com/application/o/open-webui/.well-known/openid-configuration
      - OAUTH_CLIENT_ID=open-webui
      - OAUTH_CLIENT_SECRET=${OPENWEBUI_OAUTH_CLIENT_SECRET}
      - OAUTH_SCOPES=openid email profile
      - OPENID_REDIRECT_URI=https://ai.kitestacks.com/oauth/oidc/callback
      - OPENAI_API_BASE_URL=http://litellm:<port>/v1
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
      - ENABLE_OAUTH_PERSISTENT_CONFIG=false
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - litellm
    networks:
      - default
      - kitestacks

volumes:
  open-webui:

networks:
  kitestacks:
    external: true

litellm_config.yaml:

model_list:
  - model_name: kite-free
    litellm_params:
      model: openrouter/openrouter/free
      api_key: os.environ/OPENROUTER_API_KEY
  - model_name: kite-llama
    litellm_params:
      model: openrouter/meta-llama/llama-3.1-8b-instruct:free
      api_key: os.environ/OPENROUTER_API_KEY
  - model_name: kite-deepseek
    litellm_params:
      model: openrouter/deepseek/deepseek-chat-v3-0324:free
      api_key: os.environ/OPENROUTER_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

.env keys: OPENROUTER_API_KEY, LITELLM_MASTER_KEY, WEBUI_SECRET_KEY, OPENWEBUI_OAUTH_CLIENT_SECRET

5.13 BookStack (Internal Wiki)

~/kitestacks-live/docker/bookstack/docker-compose.yml (internal only, no Cloudflare tunnel):

services:
  bookstack:
    image: lscr.io/linuxserver/bookstack:latest
    container_name: bookstack
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
      - APP_URL=http://<MONK_LAN_IP>:<port>
      - DB_HOST=bookstack-db
      - DB_PORT=<port>
      - DB_USER=bookstack
      - DB_PASS=${BOOKSTACK_DB_PASS}
      - DB_DATABASE=bookstackapp
    ports:
      - "<port>:<port>"
    volumes:
      - ./bookstack:/config
    depends_on:
      - bookstack-db

  bookstack-db:
    image: mariadb:11
    container_name: bookstack-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${BOOKSTACK_DB_ROOT_PASS}
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstack
      - MYSQL_PASSWORD=${BOOKSTACK_DB_PASS}
    volumes:
      - ./db:/var/lib/mysql

Access at http://<MONK_LAN_IP>:<port> (LAN only). No Cloudflare tunnel — internal wiki only.

Phase 6 — Authentik SSO Configuration

Log in to https://auth.kitestacks.com → Admin Interface. Create one OAuth2/OpenID Provider + Application for each service.

General Provider Settings (all providers)

  • Client type: Confidential
  • Signing Key: default RS256 key
  • Token validity: 10 min access, 30 days refresh
  • Scopes: openid, email, profile

6.1 Create Providers

App Client ID Redirect URI
Grafana grafana https://grafana.kitestacks.com/login/generic_oauth
Open WebUI open-webui https://ai.kitestacks.com/oauth/oidc/callback
Forgejo forgejo https://gitforge.kitestacks.com/user/oauth2/authentik/callback
Karakeep karakeep https://links.kitestacks.com/api/auth/callback/custom
Kavita kavita https://kavita.kitestacks.com/signin-oidc
Portainer portainer https://portainer.kitestacks.com

Note: OpenProject was removed 2026-06-12 and replaced by OSticket. The OpenProject Authentik provider and application have been deleted from the shared DB. OSticket does not currently have an Authentik SSO provider (local auth only).

Auth code TTL: All OAuth2 providers have access_code_validity = minutes=10 (bumped from 1 min on 2026-06-12) to prevent invalid_grant errors during monk reconnect. The default 60-second window was too short for monk's ~5-minute startup after going offline.

For each provider, create a matching Application (slug = Client ID, launch URL = https://<subdomain>.kitestacks.com).

6.2 Portainer — restrict to homelab-admin group

After creating the Portainer Application, add a Policy Binding:

  • Applications → Portainer → Policy/Group Bindings → Add
  • Select group: homelab-admin

6.3 Configure Forgejo OAuth source

In Forgejo Admin → Authentication Sources → Add:

  • Type: OAuth2
  • Name: authentik
  • Provider: OpenID Connect
  • Client ID: forgejo
  • Client Secret: (paste from Authentik)
  • Auto Discovery URL: https://auth.kitestacks.com/application/o/forgejo/.well-known/openid-configuration
  • Additional scopes: email profile

6.4 Karakeep — important redirect_uri fix

Karakeep uses NextAuth.js. The OAuth provider ID is custom, so the callback path is /api/auth/callback/custom, NOT /api/auth/callback/authentik. Ensure the Authentik Karakeep provider's redirect URI is exactly https://links.kitestacks.com/api/auth/callback/custom.

If you need to update this after initial setup via direct Postgres edit:

# Connect to the shared authentik-postgres on kscloud1
docker exec -it authentik-postgres psql -U authentik -d authentik
# Extract PG_PASS with: grep PG_PASS .env | cut -d= -f2-
BEGIN;
UPDATE authentik_providers_oauth2_oauth2provider
SET _redirect_uris = '["https://links.kitestacks.com/api/auth/callback/custom"]'
WHERE name = 'Karakeep';
COMMIT;
# Then restart authentik + authentik-worker on BOTH monk and kscloud1 and wait for healthy

Phase 7 — Deploy Full Replica on kscloud1

All 9 service directories live under /opt/kitestacks/docker/ on kscloud1. The same docker-compose patterns apply, with these differences:

  • OpenProject uses port <port>:<port> on host (port is taken by the pre-existing caddy)
  • ENABLE_SIGNUP=true on Open WebUI (can't SSO if Authentik has no providers yet)
  • FORGEJO_API_BASE=http://<MONK_TAILSCALE_IP>:<port> for metrics-api (monk's Forgejo over Tailscale)
  • Authentik on kscloud1 uses the same shared DB (it's the host — localhost resolves fine; use <KSCLOUD1_TAILSCALE_IP> for consistency)

7.1 Deploy cloudflared on kscloud1 (3rd connector)

Same docker-compose.yml as monk — same TUNNEL_TOKEN. Cloudflare assigns a new connector ID automatically.

ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>
cd /opt/kitestacks/docker/cloudflared && docker compose up -d

7.2 Deploy all other services

Follow the same patterns as Phase 5 for each service. Start with authentik (depends on shared Postgres+Redis which is already running), then remaining services in any order.

OpenProject on kscloud1 — first boot takes ~4–5 minutes (Postgres initdb + Rails migrations). Watch logs:

docker logs -f openproject 2>&1 | grep -E "Listening|ERROR"

7.3 Sync Authentik DB from kscloud1 to kscloud1

The shared DB is already on kscloud1, so kscloud1's own Authentik was bootstrapped with it. No sync needed. Monk's Authentik also uses it over Tailscale.

7.4 Sync Kavita DB from monk to kscloud1

The kavita.db must be synced so both connectors show the same users and OIDC settings. Direct file copy produces "database disk image is malformed" due to WAL mode — use sqlite3 backup:

# On monk: create a consistent backup
docker run --rm \
  -v ~/kitestacks-live/docker/kavita/config:/data \
  -v /tmp:/out \
  python:3-alpine sh -c "
    pip install -q sqlite3-api 2>/dev/null; python3 -c \"
import sqlite3, shutil
src = sqlite3.connect('/data/kavita.db')
dst = sqlite3.connect('/out/kavita-backup.db')
src.backup(dst)
dst.close(); src.close()
\""

scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-backup.db kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/

# On kscloud1: replace kavita.db
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
  cd /opt/kitestacks/docker/kavita
  docker compose stop kavita
  rm -f config/kavita.db config/kavita.db-wal config/kavita.db-shm
  cp /tmp/kavita-backup.db config/kavita.db
  chown 1000:1000 config/kavita.db   # kavita container runs as root, but 1000:1000 is fine
  docker compose start kavita
EOF

After sync, configure Kavita OIDC via SSH port-forward (NOT direct DB edit):

ssh -L 5099:localhost:<port> kenpat@<KSCLOUD1_PUBLIC_IP>
# Open http://localhost:<port> in browser
# Settings → OIDC: Authority=https://auth.kitestacks.com/application/o/kavita/ (trailing slash!)
# Client ID: kavita, Secret: (96-char hex from Authentik), Enabled: true

7.5 Sync Kavita cover images from monk to kscloud1

# On monk:
tar czf /tmp/kavita-covers.tar.gz -C ~/kitestacks-live/docker/kavita/config/covers .
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-covers.tar.gz kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/

# On kscloud1:
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
  mkdir -p /opt/kitestacks/docker/kavita/config/covers
  docker run --rm \
    -v /opt/kitestacks/docker/kavita/config/covers:/covers \
    -v /tmp/kavita-covers.tar.gz:/covers.tar.gz \
    alpine sh -c "cd /covers && tar xzf /covers.tar.gz && chown -R 1000:1000 ."
EOF

Note: book files themselves are not synced. Cover images load correctly; browsing the library when served by kscloud1 shows entries but no actual files — this is the accepted stale-data tradeoff.

7.6 Grafana provisioning on kscloud1

Same provisioning structure as monk. The Prometheus data source on kscloud1 points to kscloud1's own Prometheus (http://prometheus:<port>), which only scrapes kscloud1's node-exporter (monk is behind home NAT, not reachable from kscloud1 directly).

7.7 ufw — allow metrics API

echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from <IP_REDACTED>/12 to any port <port> proto tcp

Phase 8 — Portal UI

The portal lives at ~/kitestacks-live/docker/kitestacks-portal/public/index.html on monk and must be kept in sync across all 3 copies:

Host Path
monk (production) ~/kitestacks-live/docker/kitestacks-portal/public/index.html
monk (test) ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html
kscloud1 /opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html

Portal Panels

Panel Color Cards
INFRASTRUCTURE cyan Portainer (live), Authentik, Cloudflare
MONITORING magenta Grafana, Prometheus (coming-soon), Node Exporter (coming-soon), Uptime Kuma
AI & AUTOMATION purple Kite AI (coming-soon), LiteLLM (coming-soon), OpenRouter (coming-soon), FluxCD (coming-soon)
KNOWLEDGE BASE pink Kavita, Karakeep
DEVELOPMENT cyan Forgejo, OSticket
SYSTEM STATUS — live metrics from /api/metrics
RECENT ACTIVITY — last 8 commits from Forgejo via /api/activity
COMMUNITY — Discord invite link

Update workflow

Edit public/index.html on monk test portal first, verify at http://localhost:<port>, then copy to production and kscloud1.

# Test change at localhost:<port> first, then:
cp ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html \
   ~/kitestacks-live/docker/kitestacks-portal/public/index.html

scp ~/kitestacks-live/docker/kitestacks-portal/public/index.html \
    kenpat@<KSCLOUD1_PUBLIC_IP>:/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html

No container restarts needed — nginx serves the files directly from the bind-mounted volume.

Phase 9 — Monitoring

Prometheus scrapes

Monk's Prometheus scrapes both:

  • node-exporter:<port> (monk itself, via Docker DNS)
  • <KSCLOUD1_PUBLIC_IP>:<port> (kscloud1, direct public IP — kscloud1's node-exporter is <IP_REDACTED>:9100)

kscloud1's Prometheus only scrapes itself (monk is behind home NAT).

Grafana "Node Exporter Full" dashboard

Dashboard ID 1860 (provisioned from file). Use the instance dropdown to switch between monk and kscloud1 views. Both instances appear because Prometheus has both scrape jobs.

Phase 10 — Discord Integration

Discord card on portal

The Community panel on the portal has a Discord card linking to https://discord.gg/QbdveTb6Kw. No setup needed beyond keeping the invite valid.

Discord #recent-activities push webhook

A webhook posts to the Discord #recent-activities channel when Forgejo receives new commits. This is configured in Forgejo's webhook settings per-repository:

  1. In Forgejo, go to a repo → Settings → Webhooks → Add Webhook → Gitea (compatible format)
  2. Target URL: your Discord webhook URL (Discord → channel settings → Integrations → Webhooks → Copy Webhook URL)
  3. Content type: application/json
  4. Events: Push events (select relevant events)
  5. Active: enabled

The webhook URL is stored only in Forgejo's DB — do not commit it to this repo.

Phase 11 — Verification Checklist

Run after any major change or new host deployment.

Public subdomains

for sub in www auth gitforge tasks ai links kavita grafana status; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com")
  echo "$sub: $code"
done

Expected: all return 200 (or 301/302 for redirect-based logins).

SSO flows

  • https://auth.kitestacks.com → Authentik admin login works
  • https://grafana.kitestacks.com → "Sign in with authentik" → lands in Grafana
  • https://ai.kitestacks.com → "Sign in with Authentik" → lands in Open WebUI
  • https://gitforge.kitestacks.com → "Sign In with authentik" → lands in Forgejo
  • https://links.kitestacks.com → Karakeep login with Authentik → works
  • https://kavita.kitestacks.com → "Sign in with authentik" → works

Failover test (disconnect monk's internet)

With monk's home network off (phone hotspot or at a different location):

for sub in www auth gitforge tasks ai links kavita grafana status; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com")
  echo "$sub: $code"
done

All 9 should still return 200 (served by kscloud1).

Authentik shared DB health

# On monk — check authentik can reach shared DB
docker exec authentik sh -c 'python3 -c "import psycopg2; psycopg2.connect(host=\"<KSCLOUD1_TAILSCALE_IP>\", dbname=\"authentik\", user=\"authentik\", password=\"$AUTHENTIK_POSTGRESQL__PASSWORD\")"' && echo "DB reachable"

# Check both authentik containers are healthy
docker inspect --format '{{.Name}}: {{.State.Health.Status}}' authentik authentik-worker

Phase 12 — FluxCD GitOps Bootstrap (T14s)

Once K3s and Flux are installed, bootstrap the cluster to the homelab repository.

flux bootstrap git \
  --url=ssh://git@localhost:2222/kenpat/kitestacks-homelab.git \
  --branch=main \
  --path=clusters/T14s \
  --private-key-file=$HOME/.ssh/id_ed25519_gitforge

Flux will now automatically sync the manifests in apps/kavita/ to the cluster.

Key File Locations Reference

What Path on monk
All Docker services ~/kitestacks-live/docker/
Portal HTML (production) ~/kitestacks-live/docker/kitestacks-portal/public/index.html
Portal HTML (test) ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html
Metrics API source ~/kitestacks-live/docker/kitestacks-portal-test/api/main.py
Prometheus config ~/kitestacks-live/docker/prometheus/prometheus.yml
Grafana provisioning ~/kitestacks-live/docker/grafana/provisioning/
Book library ~/kitestacks-live/library/books/
Kavita config ~/kitestacks-live/docker/kavita/config/
Kavita cover images ~/kitestacks-live/docker/kavita/config/covers/
Authentik media ~/kitestacks-live/docker/authentik/media/
Homelab git repo ~/kitestacks-from-assassin-t14/opt/kitestacks-autosync/kitestacks-homelab/ (or cloned from gitforge.kitestacks.com)
What Path on kscloud1
All Docker services /opt/kitestacks/docker/
Shared Authentik Postgres data /opt/kitestacks/docker/authentik/postgres/
Portal HTML /opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html

Upcoming: Oracle Cloud Migration

kscloud1 (Hetzner) is planned to be replaced by an Oracle Cloud VPS. When the new VPS is ready:

  1. Provision Oracle VPS, install Docker + Tailscale
  2. Deploy all services using the same Phase 7 playbook
  3. Note: shared Authentik Postgres+Redis must move too — update both monk's and all connectors' AUTHENTIK_REDIS__HOST and AUTHENTIK_POSTGRESQL__HOST to the new Tailscale IP
  4. Add Oracle VPS cloudflared as a new tunnel connector (same TUNNEL_TOKEN)
  5. Verify all 9 subdomains work from Oracle VPS
  6. Remove kscloud1 cloudflared from the tunnel connectors
  7. Decommission kscloud1 Hetzner server

Secrets Inventory (keys only — values in .env files)

.env location Keys
authentik/.env AUTHENTIK_SECRET_KEY, PG_PASS
grafana/.env GRAFANA_OAUTH_CLIENT_SECRET
osticket/.env OSTICKET_DB_PASS, OSTICKET_DB_ROOT, OSTICKET_ADMIN_PASS, OSTICKET_INSTALL_SECRET
kite-ai/.env OPENROUTER_API_KEY, LITELLM_MASTER_KEY, WEBUI_SECRET_KEY, OPENWEBUI_OAUTH_CLIENT_SECRET
karakeep/.env KARAKEEP_VERSION, NEXTAUTH_SECRET, NEXTAUTH_URL, MEILI_MASTER_KEY, OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_WELLKNOWN_URL, OAUTH_PROVIDER_NAME, OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING
cloudflared/docker-compose.yml TUNNEL_TOKEN (inline)

Do not commit .env files, webhook URLs, or database files to this repo.

Phase 8: Forgejo Sync + osTicket Authentik LDAP SSO (2026-06-14/15)

Forgejo Sync (monk → kscloud1)

Monk is authoritative. kscloud1 Forgejo is a read replica synced every 6 hours.

Sync script: ~/kitestacks-live/docker/forgejo/sync-to-cloud.sh Cron: 0 */6 * * * on monk, logs to /tmp/forgejo-sync.log

Manual sync:

~/kitestacks-live/docker/forgejo/sync-to-cloud.sh

To re-do a full restore from scratch (e.g., after kscloud1 rebuild):

# On monk: create dump
docker exec -u git forgejo /app/gitea/gitea dump --type zip -f /tmp/forgejo-backup.zip
docker cp forgejo:/tmp/forgejo-backup.zip /tmp/forgejo-backup.zip
# Transfer and restore on cloud host — see claude-memory for detailed steps

osTicket Authentik LDAP SSO

Staff log into tasks.kitestacks.com/scp/ using their Authentik credentials (not a separate osTicket password).

Architecture:

osticket-app → authentik-ldap-proxy:389 (socat) → authentik-ldap:3389 → auth.kitestacks.com

Services deployed:

  • ~/kitestacks-live/docker/authentik-ldap/ — LDAP outpost + socat proxy on monk
  • /opt/kitestacks/docker/authentik-ldap/ — LDAP outpost on kscloud1

LDAP search account: cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io Password stored in Authentik and in osTicket's ost_config (namespace plugin.2, key bind_pw, encrypted).

auth-ldap.phar at /data/upload/include/plugins/auth-ldap.phar inside the osticket-app container has been patched (original at .phar.orig). Do NOT replace it with the upstream version — the patch is required for PHP 7.3 + PEAR compatibility.

If LDAP login stops working:

# Check LDAP outpost is running and connected
docker logs authentik-ldap --since 5m | grep -v debug
docker logs authentik-ldap-proxy 2>&1 | tail -5

# Test bind from osticket-app container
docker exec osticket-app php -r "
  \$c = @ldap_connect('authentik-ldap-proxy');
  ldap_set_option(\$c, LDAP_OPT_PROTOCOL_VERSION, 3);
  \$r = @ldap_bind(\$c, 'cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io', 'PASSWORD');
  echo \$r ? 'OK' : ldap_error(\$c);
"
# Verify kscloud1 outpost reachable
nc -zv 100.123.254.52 3389

Reset a staff member's Authentik password:

docker exec authentik ak shell -c "
from authentik.core.models import User
u = User.objects.get(username='kenpat7177')
u.set_password('NewPassword123!')
u.save()
print('done')
"

Clear osTicket login lockout:

docker run --rm --network host mariadb:10.11 mysql \
  -h 100.123.254.52 -u osticket -p<DB_PASS> osticket \
  -e "DELETE FROM ost_session;"