kitestacks-homelab/RUNBOOK.md

# KiteStacks Homelab — Complete Setup Runbook

**Last Updated:** 2026-06-12  
**Status:** Production (monk primary, kscloud1 Hetzner cloud replica)  
**Maintainer:** kenpat (kenpat7177@gmail.com)

---

## Architecture Overview

```
Internet
   │
   └── Cloudflare (DNS + Tunnel)
          │  Active-Active across 3 connectors
          ├── cloudflared on monk (primary home machine)
          ├── cloudflared on kscloud1 (Hetzner VPS, <KSCLOUD1_PUBLIC_IP>)
          └── cloudflared on T14s (currently OFF)

Tailscale overlay network (VPN mesh):
  monk      <MONK_TAILSCALE_IP>
  kscloud1  <KSCLOUD1_TAILSCALE_IP>  ← hosts shared Authentik Postgres + Redis
  T14s      <T14S_TAILSCALE_IP>    (off)
  pixel-6   <PIXEL6_TAILSCALE_IP>
  samurai   <SAMURAI_TAILSCALE_IP>
```

**Nine public subdomains** route through the same Cloudflare Tunnel token. Both monk and kscloud1 are connectors so the site stays up when either goes offline.

| Subdomain | Container | Port |
|-----------|-----------|------|
| www.kitestacks.com | homepage (nginx portal) | <port> |
| auth.kitestacks.com | authentik | <port> |
| gitforge.kitestacks.com | forgejo | <port> |
| tasks.kitestacks.com | osticket (nginx proxy → osticket-app) | 8080 |
| portainer.kitestacks.com | portainer | 9443 (HTTPS) |
| ai.kitestacks.com | kite-openwebui | <port> |
| links.kitestacks.com | karakeep | <port> |
| kavita.kitestacks.com | kavita | <port> |
| grafana.kitestacks.com | grafana | <port> |
| status.kitestacks.com | uptime-kuma | <port> |

**Important — active-active data model:** monk and kscloud1 each run their own copies of all stateful apps (Forgejo, Kavita, OSticket, etc.) with independent databases. Data is intentionally NOT synced between them (except for Authentik, which shares a single Postgres+Redis on kscloud1 over Tailscale). If kscloud1 serves a request, the user sees kscloud1's database. This is the accepted tradeoff for guaranteed uptime.

---

## Phase 0 — Prerequisites

**Accounts & services you need before starting:**

- Domain name (kitestacks.com) registered and nameservers pointed to Cloudflare
- Cloudflare account (free tier is fine — do NOT enable Zero Trust/Access, which costs money; the Tunnel UI lives under the "Zero Trust" nav but configuring tunnel hostnames is free)
- Hetzner Cloud account (for kscloud1 VPS)
- OpenRouter account + API key (for LiteLLM/AI services)
- Discord server (for community panel + #recent-activities webhook)
- SSH key pair (`~/.ssh/id_ed25519_kscloud1` for kscloud1 access)

**Software on monk (local machine):**

```bash
# Docker (official install script or distro package)
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER   # log out and back in

# Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
```

---

## Phase 1 — Monk (Primary Host) — Base Setup

Monk is the primary home server. All services live under `~/kitestacks-live/docker/`, one directory per app.

### 1.1 Create the shared Docker network

All containers join this network. `cloudflared` resolves container names within it for tunnel routing.

```bash
docker network create kitestacks
```

### 1.2 Directory structure

```
~/kitestacks-live/
└── docker/
    ├── authentik/
    ├── cloudflared/
    ├── forgejo/
    ├── grafana/
    ├── karakeep/
    ├── kavita/
    ├── kite-ai/
    ├── kitestacks-portal/        # production portal (container: homepage)
    ├── kitestacks-portal-test/   # dev portal + metrics API
    ├── openproject/
    ├── portainer/
    ├── prometheus/
    └── uptime-kuma/
```

### 1.3 Tailscale on monk

```bash
sudo tailscale up
# Accept the auth link, join tailnet as "monk"
# monk will get Tailscale IP <MONK_TAILSCALE_IP>
```

---

## Phase 2 — kscloud1 (Hetzner VPS) — Provision & Base Setup

### 2.1 Provision on Hetzner

- Server: Hetzner CX22 (or equivalent) — 3 vCPU, 3.7 GB RAM, 75 GB disk
- OS: Debian/Ubuntu
- Region: your choice
- Public IP: <KSCLOUD1_PUBLIC_IP>
- Add your SSH public key at provision time

### 2.2 SSH access

```bash
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>
```

Password for sudo: `<KSCLOUD1_SUDO_PASSWORD>` (non-interactive sudo: `echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S <cmd>`)

### 2.3 Install Docker on kscloud1

```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker kenpat
# Log out and back in
```

### 2.4 Create the kitestacks Docker network on kscloud1

```bash
docker network create kitestacks
```

All service directories live under `/opt/kitestacks/docker/` (same one-dir-per-app pattern as monk).

### 2.5 Tailscale on kscloud1

```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# Join the same tailnet; kscloud1 will get Tailscale IP <KSCLOUD1_TAILSCALE_IP>
```

### 2.6 ufw on kscloud1

kscloud1 has ufw active with default-deny. Fix docker-bridge-to-host traffic:

```bash
echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from <IP_REDACTED>/12 to any port <port> proto tcp
# Allows homepage metrics API to be reached from within docker containers
```

---

## Phase 3 — Cloudflare Tunnel Setup

### 3.1 Create the tunnel

In the Cloudflare dashboard: **Zero Trust → Networks → Tunnels → Create a tunnel**

- Type: Cloudflared
- Name: kitestacks (or any name)
- Copy the **Tunnel Token** — this is the `TUNNEL_TOKEN` used in all cloudflared containers

### 3.2 Add public hostnames

Still in the tunnel config, add one public hostname per subdomain. Use the container name as the service target (cloudflared resolves these within the `kitestacks` Docker network):

| Public Hostname | Service |
|----------------|---------|
| www.kitestacks.com | `http://homepage:<port>` |
| auth.kitestacks.com | `http://authentik:<port>` |
| gitforge.kitestacks.com | `http://forgejo:<port>` |
| tasks.kitestacks.com | `http://osticket:8080` |
| portainer.kitestacks.com | `https://portainer:9443` (enable "No TLS Verify") |
| ai.kitestacks.com | `http://kite-openwebui:<port>` |
| links.kitestacks.com | `http://karakeep:<port>` |
| kavita.kitestacks.com | `http://kavita:<port>` |
| grafana.kitestacks.com | `http://grafana:<port>` |
| status.kitestacks.com | `http://uptime-kuma:<port>` |
| portainer.kitestacks.com | `https://portainer:<port>` (enable "No TLS Verify") |

> **Note:** Portainer uses HTTPS internally. All others are plain HTTP (TLS is handled by Cloudflare at the edge).

---

## Phase 4 — Shared Authentik Database on kscloud1

Authentik on both monk and kscloud1 must share ONE Postgres+Redis to prevent `invalid_grant` errors. OAuth2 authorization codes are per-DB rows — if `/authorize` hits one connector and `/application/o/token/` hits another, the code is not found. The shared DB lives on kscloud1, bound only to the Tailscale interface.

### 4.1 Deploy shared Postgres + Redis on kscloud1

`/opt/kitestacks/docker/authentik/docker-compose.yml`:

```yaml
services:
  authentik-postgres:
    image: postgres:16
    container_name: authentik-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: authentik
      POSTGRES_PASSWORD: ${PG_PASS}
      POSTGRES_DB: authentik
    ports:
      - "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>"   # Tailscale IP only — not public
    volumes:
      - ./postgres:/var/lib/postgresql/data

  authentik-redis:
    image: redis:alpine
    container_name: authentik-redis
    restart: unless-stopped
    ports:
      - "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>"   # Tailscale IP only — not public
    volumes:
      - redis-data:/data

  authentik:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
      AUTHENTIK_BOOTSTRAP_EMAIL: <BOOTSTRAP_ADMIN_EMAIL>
      AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD}
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks
    depends_on:
      - authentik-postgres
      - authentik-redis

  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    networks:
      - default
      - kitestacks

volumes:
  redis-data:

networks:
  kitestacks:
    external: true
```

`.env` keys: `PG_PASS`, `AUTHENTIK_SECRET_KEY`, `AUTHENTIK_BOOTSTRAP_PASSWORD`

> **Wait ~2 minutes** after first boot for Postgres migrations and Authentik startup before testing.

---

## Phase 5 — Deploy Services on Monk

Start with cloudflared first so tunnel connectors register, then deploy services in any order (they all restart-unless-stopped).

### 5.1 cloudflared

`~/kitestacks-live/docker/cloudflared/docker-compose.yml`:

```yaml
services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    command: tunnel --no-autoupdate run
    environment:
      - TUNNEL_TOKEN=<your_tunnel_token>
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true
```

```bash
cd ~/kitestacks-live/docker/cloudflared && docker compose up -d
```

### 5.2 Authentik (monk side — points to shared DB on kscloud1)

`~/kitestacks-live/docker/authentik/docker-compose.yml`:

```yaml
services:
  authentik:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks

  authentik-worker:
    image: ghcr.io/goauthentik/server:latest
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true
```

`.env` keys: `AUTHENTIK_SECRET_KEY` (same value as kscloud1), `PG_PASS` (same value as kscloud1)

> **Critical:** `AUTHENTIK_SECRET_KEY` and `PG_PASS` must be IDENTICAL between monk and kscloud1 or shared-DB authentication will fail.

> **Rollback note:** If Tailscale connectivity breaks, monk's authentik will fail to connect to <KSCLOUD1_TAILSCALE_IP>. To roll back: restore a local postgres+redis to monk's compose, `docker start authentik-postgres authentik-redis`, then `docker compose up -d`. Do a fresh `pg_dump` from kscloud1 first to avoid losing any logins made since the shared-DB migration.

### 5.3 Forgejo

`~/kitestacks-live/docker/forgejo/docker-compose.yml`:

```yaml
services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:11
    container_name: forgejo
    restart: unless-stopped
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__DOMAIN=gitforge.kitestacks.com
      - FORGEJO__server__ROOT_URL=https://gitforge.kitestacks.com/
      - FORGEJO__server__SSH_DOMAIN=gitforge.kitestacks.com
      - FORGEJO__server__SSH_PORT=<port>
    ports:
      - "<port>:<port>"
      - "<port>:<port>"
    volumes:
      - ./data:/data
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true
```

### 5.4 KiteStacks Portal (Production)

The portal is a custom nginx-served static site (cyberpunk-themed). The container is named `homepage` to match the Cloudflare tunnel route `http://homepage:<port>`.

`~/kitestacks-live/docker/kitestacks-portal/docker-compose.yml`:

```yaml
services:
  homepage:
    image: nginx:alpine
    container_name: homepage
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks
    volumes:
      - ./public:/usr/share/nginx/html:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  kitestacks:
    external: true
```

`nginx.conf`:

```nginx
server {
    listen <port>;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://host.docker.internal:<port>;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 10s;
    }

    location /images/ {
        expires 7d;
        add_header Cache-Control "public, max-age=604800";
    }
}
```

The `/api/` path proxies to the Metrics API (port <port> on the host).

### 5.5 Metrics API (Portal Backend)

The Metrics API provides real-time system stats (CPU, RAM, storage, network, uptime), weather, and Forgejo recent-activity data to the portal. It runs in `network_mode: host` with `pid: host` to read actual host metrics via psutil.

`~/kitestacks-live/docker/kitestacks-portal-test/docker-compose.yml` (this is the compose that owns the shared metrics-api container):

```yaml
services:
  kitestacks-portal-test:
    image: nginx:alpine
    container_name: kitestacks-portal-test
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./public:/usr/share/nginx/html:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - metrics-api

  metrics-api:
    build: ./api
    container_name: kitestacks-metrics-api
    restart: unless-stopped
    pid: host
    network_mode: host
    environment:
      - HOST_PROC=/host/proc
      - HOST_SYS=/host/sys
      - HOST_ETC=/host/etc
      - FORGEJO_API_BASE=http://localhost:<port>
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /etc/hostname:/host/etc/hostname:ro
      - /etc/localtime:/host/etc/localtime:ro
      - /:/host:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
```

`api/main.py` serves four endpoints:
- `GET /api/metrics` — CPU, RAM, storage, network, uptime, system info
- `GET /api/weather` — weather via ipapi.co + open-meteo.com (cached 10 min)
- `GET /api/activity` — recent Forgejo commits (cached 60s)
- `GET /api/health` — `{"ok": true}`

`FORGEJO_API_BASE` on monk points to `http://localhost:<port>` (monk's own Forgejo). On kscloud1 it points to `http://<MONK_TAILSCALE_IP>:<port>` (monk's Forgejo over Tailscale) so both connectors show consistent recent activity from the same source.

**Python deps:** `fastapi`, `uvicorn`, `psutil`, `httpx`

```bash
cd ~/kitestacks-live/docker/kitestacks-portal-test && docker compose up -d
cd ~/kitestacks-live/docker/kitestacks-portal && docker compose up -d
```

### 5.6 Karakeep (Bookmarks)

`~/kitestacks-live/docker/karakeep/docker-compose.yml`:

```yaml
services:
  karakeep:
    image: ghcr.io/karakeep-app/karakeep:${KARAKEEP_VERSION:-release}
    container_name: karakeep
    restart: unless-stopped
    environment:
      - PORT=<port>
      - MEILI_ADDR=http://karakeep-meilisearch:<port>
      - BROWSER_WEB_URL=http://karakeep-chrome:<port>
      - DATA_DIR=/data
    volumes:
      - ./data:/data
    networks:
      - default
      - kitestacks
      - internal

  karakeep-chrome:
    image: gcr.io/zenika-hub/alpine-chrome:124
    container_name: karakeep-chrome
    restart: unless-stopped
    command: chromium-browser --headless --remote-debugging-address=<IP_REDACTED> --remote-debugging-port=<port>
    networks:
      - internal

  karakeep-meilisearch:
    image: getmeili/meilisearch:v1.41.0
    container_name: karakeep-meilisearch
    restart: unless-stopped
    environment:
      - MEILI_NO_ANALYTICS=true
    volumes:
      - ./meilisearch:/meili_data
    networks:
      - internal

networks:
  kitestacks:
    external: true
  internal:
```

`.env` keys: `KARAKEEP_VERSION`, `NEXTAUTH_SECRET`, `NEXTAUTH_URL`, `MEILI_MASTER_KEY`, `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`, `OAUTH_WELLKNOWN_URL`, `OAUTH_PROVIDER_NAME`, `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING`

**SSO note for Karakeep:** The `OAUTH_CLIENT_ID` is `karakeep`. The Authentik redirect URI must be `https://links.kitestacks.com/api/auth/callback/custom` (NextAuth.js uses provider id `custom`, NOT `authentik` — the callback path is `/api/auth/callback/custom`).

### 5.7 Kavita (eBook Reader)

`~/kitestacks-live/docker/kavita/docker-compose.yml`:

```yaml
services:
  kavita:
    image: ghcr.io/kareadita/kavita:latest
    container_name: kavita
    restart: unless-stopped
    environment:
      - TZ=UTC
    ports:
      - "<port>:<port>"
    volumes:
      - ./config:/kavita/config
      - ../../library/books:/books
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true
```

**SSO config for Kavita:** Do NOT edit `config/kavita.db` directly — settings written to the `ServerSetting` table are overwritten by Kavita on restart. Always configure OIDC through Kavita's Settings UI:
- Go to `http://localhost:<port>` (or SSH tunnel `ssh -L 5099:localhost:<port> kenpat@<host>` for kscloud1)
- Settings → OIDC:
  - Authority: `https://auth.kitestacks.com/application/o/kavita/` (trailing slash required — must exactly match the `issuer` field in Authentik's discovery doc)
  - Client ID: `kavita`
  - Client Secret: (96-char hex from Authentik)
  - Enabled: true, ProviderName: `authentik`

### 5.8 OSticket (Help Desk) — replaced OpenProject 2026-06-12

OSticket is a PHP-based help desk ticketing system. It runs behind an nginx proxy container that Cloudflare Tunnel reaches at `http://osticket:8080`.

**Architecture:** `osticket` (nginx:alpine, port 8080) → `osticket-app` (campbellsoftwaresolutions/osticket, port 80) + `osticket-db` (mariadb:10.11)

`~/kitestacks-live/docker/osticket/docker-compose.yml`:

```yaml
services:
  osticket:
    image: nginx:alpine
    container_name: osticket
    restart: unless-stopped
    depends_on:
      - osticket-app
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    ports:
      - "<port>:8080"
    networks:
      - default
      - kitestacks

  osticket-app:
    image: campbellsoftwaresolutions/osticket
    container_name: osticket-app
    restart: unless-stopped
    depends_on:
      - osticket-db
    environment:
      - INSTALL_NAME=KiteStacks Help Desk
      - INSTALL_EMAIL=helpdesk@kitestacks.com
      - INSTALL_URL=https://tasks.kitestacks.com/
      - ADMIN_FIRSTNAME=Ken
      - ADMIN_LASTNAME=Pat
      - ADMIN_EMAIL=<your_email>
      - ADMIN_USERNAME=<your_username>
      - ADMIN_PASSWORD=${OSTICKET_ADMIN_PASS}
      - INSTALL_SECRET=${OSTICKET_INSTALL_SECRET}
      - MYSQL_HOST=osticket-db
      - MYSQL_DATABASE=osticket
      - MYSQL_USER=osticket
      - MYSQL_PASSWORD=${OSTICKET_DB_PASS}
    volumes:
      - osticket_uploads:/data/upload/include/i18n
    networks:
      - default

  osticket-db:
    image: mariadb:10.11
    container_name: osticket-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${OSTICKET_DB_ROOT}
      - MYSQL_DATABASE=osticket
      - MYSQL_USER=osticket
      - MYSQL_PASSWORD=${OSTICKET_DB_PASS}
    volumes:
      - osticket_db:/var/lib/mysql
    networks:
      - default

volumes:
  osticket_uploads:
  osticket_db:

networks:
  kitestacks:
    external: true
```

`nginx.conf`:
```nginx
server {
    listen 8080;
    location / {
        proxy_pass http://osticket-app:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Host $host;
        proxy_read_timeout 120s;
    }
}
```

`.env` keys: `OSTICKET_DB_PASS`, `OSTICKET_DB_ROOT`, `OSTICKET_ADMIN_PASS`, `OSTICKET_INSTALL_SECRET`

**First boot:** The install script runs automatically, connects to MariaDB, and seeds the database. Wait for `Database installation successful` in `docker logs osticket-app` before testing. The same `.env` values are used on kscloud1 so both instances share the same admin credentials (separate databases).

**Cloudflare Tunnel:** Route `tasks.kitestacks.com → http://osticket:8080` in the tunnel config.

**kscloud1 note:** Same compose, host port `<port>:8080` (8090 is used since port 80 is taken by caddy).

### 5.9 Grafana + Prometheus + Node Exporter

`~/kitestacks-live/docker/prometheus/docker-compose.yml`:

```yaml
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - default
      - kitestacks

  node-exporter:
    image: prom/node-exporter
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    networks:
      - default
      - kitestacks

volumes:
  prometheus-data:

networks:
  kitestacks:
    external: true
```

`~/kitestacks-live/docker/prometheus/prometheus.yml`:

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "t14-node"
    static_configs:
      - targets: ["node-exporter:<port>"]   # monk (this host)

  - job_name: "kscloud1-node"
    static_configs:
      - targets: ["<KSCLOUD1_PUBLIC_IP>:<port>"]     # kscloud1 (cloud replica)
```

`~/kitestacks-live/docker/grafana/docker-compose.yml`:

```yaml
services:
  grafana:
    image: grafana/grafana-oss
    container_name: grafana
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.kitestacks.com
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_NAME=authentik
      - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
      - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.kitestacks.com/application/o/authorize/
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://authentik:<port>/application/o/token/
      - GF_AUTH_GENERIC_OAUTH_API_URL=http://authentik:<port>/application/o/userinfo/
      - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
      - GF_AUTH_OAUTH_AUTO_LOGIN=false
    volumes:
      - ./data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    networks:
      - default
      - kitestacks

networks:
  kitestacks:
    external: true
```

`.env` keys: `GRAFANA_OAUTH_CLIENT_SECRET`

**Grafana provisioning** (`./provisioning/`):

`datasources/prometheus.yml`:
```yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:<port>
    uid: "000000001"
    isDefault: true
    editable: true
```

`dashboards/dashboard.yml`:
```yaml
apiVersion: 1
providers:
  - name: default
    folder: ''
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards
```

Place `node-exporter-full.json` (Grafana dashboard ID 1860) in `./provisioning/dashboards/`. The dashboard's instance picker dropdown will show both `t14-node` (monk) and `kscloud1-node`.

### 5.10 Uptime Kuma

`~/kitestacks-live/docker/uptime-kuma/docker-compose.yml`:

```yaml
services:
  uptime-kuma:
    image: louislam/uptime-kuma:latest
    container_name: uptime-kuma
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - uptime-kuma:/app/data
    networks:
      - default
      - kitestacks

volumes:
  uptime-kuma:

networks:
  kitestacks:
    external: true
```

Configure monitors in the Uptime Kuma UI for all 9 public subdomains plus internal health checks.

### 5.11 Portainer

`~/kitestacks-live/docker/portainer/docker-compose.yml`:

```yaml
services:
  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - default
      - kitestacks

volumes:
  portainer_data:

networks:
  kitestacks:
    external: true
```

**SSO — CONFIGURED (2026-06-12):** Authentik OAuth2 provider (Client ID: `portainer`) is live. OAuth configured via the Portainer API on both monk and kscloud1. Portal card updated to a live link on all 3 copies.

- Cloudflare Tunnel: `portainer.kitestacks.com → https://portainer:9443` (No TLS Verify) — **add this in CF dashboard if not yet present**
- Access restricted to `homelab-admin` Authentik group via PolicyBinding

**To re-configure OAuth via Portainer API** (if settings get reset):
```bash
JWT=$(curl -sk -X POST https://localhost:9443/api/auth \
  -H "Content-Type: application/json" \
  -d '{"Username":"admin","Password":"<PORTAINER_ADMIN_PASSWORD>"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")

curl -sk -X PUT https://localhost:9443/api/settings \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "AuthenticationMethod": 3,
    "OAuthSettings": {
      "ClientID": "portainer",
      "ClientSecret": "<PORTAINER_OAUTH_CLIENT_SECRET>",
      "AuthorizationURI": "https://auth.kitestacks.com/application/o/authorize/",
      "AccessTokenURI": "https://auth.kitestacks.com/application/o/token/",
      "ResourceURI": "https://auth.kitestacks.com/application/o/userinfo/",
      "RedirectURI": "https://portainer.kitestacks.com",
      "UserIdentifier": "email",
      "Scopes": "openid email profile",
      "OAuthAutoCreateUsers": true,
      "SSO": true,
      "LogoutURI": "https://auth.kitestacks.com/application/o/portainer/end-session/"
    }
  }'
```

**Password reset for monk Portainer** (if admin password is lost): use a Go container with bbolt to patch BoltDB directly — see DEBUG-DOCUMENTATION.md for the full procedure.

**kscloud1 Portainer:** admin username `kenpat7177`, password in `.env` (same as OSticket admin). OAuth also configured via API.

### 5.12 Kite AI (LiteLLM + Open WebUI)

`~/kitestacks-live/docker/kite-ai/docker-compose.yml`:

```yaml
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: kite-litellm
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    command: ["--config", "/app/config.yaml", "--port", "<port>"]
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    env_file: .env
    networks:
      - default
      - kitestacks

  kite-openwebui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: kite-openwebui
    restart: unless-stopped
    ports:
      - "<port>:<port>"
    environment:
      - WEBUI_NAME=Kite AI
      - WEBUI_URL=https://ai.kitestacks.com
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - ENABLE_SIGNUP=false
      - ENABLE_OAUTH_SIGNUP=true
      - OAUTH_PROVIDER_NAME=Authentik
      - OPENID_PROVIDER_URL=https://auth.kitestacks.com/application/o/open-webui/.well-known/openid-configuration
      - OAUTH_CLIENT_ID=open-webui
      - OAUTH_CLIENT_SECRET=${OPENWEBUI_OAUTH_CLIENT_SECRET}
      - OAUTH_SCOPES=openid email profile
      - OPENID_REDIRECT_URI=https://ai.kitestacks.com/oauth/oidc/callback
      - OPENAI_API_BASE_URL=http://litellm:<port>/v1
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
      - ENABLE_OAUTH_PERSISTENT_CONFIG=false
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - litellm
    networks:
      - default
      - kitestacks

volumes:
  open-webui:

networks:
  kitestacks:
    external: true
```

`litellm_config.yaml`:

```yaml
model_list:
  - model_name: kite-free
    litellm_params:
      model: openrouter/openrouter/free
      api_key: os.environ/OPENROUTER_API_KEY
  - model_name: kite-llama
    litellm_params:
      model: openrouter/meta-llama/llama-3.1-8b-instruct:free
      api_key: os.environ/OPENROUTER_API_KEY
  - model_name: kite-deepseek
    litellm_params:
      model: openrouter/deepseek/deepseek-chat-v3-0324:free
      api_key: os.environ/OPENROUTER_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
```

`.env` keys: `OPENROUTER_API_KEY`, `LITELLM_MASTER_KEY`, `WEBUI_SECRET_KEY`, `OPENWEBUI_OAUTH_CLIENT_SECRET`

### 5.13 BookStack (Internal Wiki)

`~/kitestacks-live/docker/bookstack/docker-compose.yml` (internal only, no Cloudflare tunnel):

```yaml
services:
  bookstack:
    image: lscr.io/linuxserver/bookstack:latest
    container_name: bookstack
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
      - APP_URL=http://<MONK_LAN_IP>:<port>
      - DB_HOST=bookstack-db
      - DB_PORT=<port>
      - DB_USER=bookstack
      - DB_PASS=${BOOKSTACK_DB_PASS}
      - DB_DATABASE=bookstackapp
    ports:
      - "<port>:<port>"
    volumes:
      - ./bookstack:/config
    depends_on:
      - bookstack-db

  bookstack-db:
    image: mariadb:11
    container_name: bookstack-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${BOOKSTACK_DB_ROOT_PASS}
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstack
      - MYSQL_PASSWORD=${BOOKSTACK_DB_PASS}
    volumes:
      - ./db:/var/lib/mysql
```

Access at `http://<MONK_LAN_IP>:<port>` (LAN only). No Cloudflare tunnel — internal wiki only.

---

## Phase 6 — Authentik SSO Configuration

Log in to `https://auth.kitestacks.com` → Admin Interface. Create one OAuth2/OpenID Provider + Application for each service.

### General Provider Settings (all providers)

- Client type: Confidential
- Signing Key: default RS256 key
- Token validity: 10 min access, 30 days refresh
- Scopes: `openid`, `email`, `profile`

### 6.1 Create Providers

| App | Client ID | Redirect URI |
|-----|-----------|-------------|
| Grafana | `grafana` | `https://grafana.kitestacks.com/login/generic_oauth` |
| Open WebUI | `open-webui` | `https://ai.kitestacks.com/oauth/oidc/callback` |
| Forgejo | `forgejo` | `https://gitforge.kitestacks.com/user/oauth2/authentik/callback` |
| Karakeep | `karakeep` | `https://links.kitestacks.com/api/auth/callback/custom` |
| Kavita | `kavita` | `https://kavita.kitestacks.com/signin-oidc` |
| Portainer | `portainer` | `https://portainer.kitestacks.com` |

> **Note:** OpenProject was removed 2026-06-12 and replaced by OSticket. The OpenProject Authentik provider and application have been deleted from the shared DB. OSticket does not currently have an Authentik SSO provider (local auth only).

> **Auth code TTL:** All OAuth2 providers have `access_code_validity = minutes=10` (bumped from 1 min on 2026-06-12) to prevent `invalid_grant` errors during monk reconnect. The default 60-second window was too short for monk's ~5-minute startup after going offline.

For each provider, create a matching Application (slug = Client ID, launch URL = `https://<subdomain>.kitestacks.com`).

### 6.2 Portainer — restrict to homelab-admin group

After creating the Portainer Application, add a Policy Binding:
- Applications → Portainer → Policy/Group Bindings → Add
- Select group: `homelab-admin`

### 6.3 Configure Forgejo OAuth source

In Forgejo Admin → Authentication Sources → Add:
- Type: OAuth2
- Name: `authentik`
- Provider: OpenID Connect
- Client ID: `forgejo`
- Client Secret: (paste from Authentik)
- Auto Discovery URL: `https://auth.kitestacks.com/application/o/forgejo/.well-known/openid-configuration`
- Additional scopes: `email profile`

### 6.4 Karakeep — important redirect_uri fix

Karakeep uses NextAuth.js. The OAuth provider ID is `custom`, so the callback path is `/api/auth/callback/custom`, NOT `/api/auth/callback/authentik`. Ensure the Authentik Karakeep provider's redirect URI is exactly `https://links.kitestacks.com/api/auth/callback/custom`.

If you need to update this after initial setup via direct Postgres edit:

```bash
# Connect to the shared authentik-postgres on kscloud1
docker exec -it authentik-postgres psql -U authentik -d authentik
# Extract PG_PASS with: grep PG_PASS .env | cut -d= -f2-
BEGIN;
UPDATE authentik_providers_oauth2_oauth2provider
SET _redirect_uris = '["https://links.kitestacks.com/api/auth/callback/custom"]'
WHERE name = 'Karakeep';
COMMIT;
# Then restart authentik + authentik-worker on BOTH monk and kscloud1 and wait for healthy
```

---

## Phase 7 — Deploy Full Replica on kscloud1

All 9 service directories live under `/opt/kitestacks/docker/` on kscloud1. The same docker-compose patterns apply, with these differences:

- OpenProject uses port `<port>:<port>` on host (port <port> is taken by the pre-existing caddy)
- `ENABLE_SIGNUP=true` on Open WebUI (can't SSO if Authentik has no providers yet)
- `FORGEJO_API_BASE=http://<MONK_TAILSCALE_IP>:<port>` for metrics-api (monk's Forgejo over Tailscale)
- Authentik on kscloud1 uses the same shared DB (it's the host — localhost resolves fine; use `<KSCLOUD1_TAILSCALE_IP>` for consistency)

### 7.1 Deploy cloudflared on kscloud1 (3rd connector)

Same `docker-compose.yml` as monk — same `TUNNEL_TOKEN`. Cloudflare assigns a new connector ID automatically.

```bash
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>
cd /opt/kitestacks/docker/cloudflared && docker compose up -d
```

### 7.2 Deploy all other services

Follow the same patterns as Phase 5 for each service. Start with authentik (depends on shared Postgres+Redis which is already running), then remaining services in any order.

**OpenProject on kscloud1** — first boot takes ~4–5 minutes (Postgres initdb + Rails migrations). Watch logs:
```bash
docker logs -f openproject 2>&1 | grep -E "Listening|ERROR"
```

### 7.3 Sync Authentik DB from kscloud1 to kscloud1

The shared DB is already on kscloud1, so kscloud1's own Authentik was bootstrapped with it. No sync needed. Monk's Authentik also uses it over Tailscale.

### 7.4 Sync Kavita DB from monk to kscloud1

The kavita.db must be synced so both connectors show the same users and OIDC settings. Direct file copy produces "database disk image is malformed" due to WAL mode — use sqlite3 backup:

```bash
# On monk: create a consistent backup
docker run --rm \
  -v ~/kitestacks-live/docker/kavita/config:/data \
  -v /tmp:/out \
  python:3-alpine sh -c "
    pip install -q sqlite3-api 2>/dev/null; python3 -c \"
import sqlite3, shutil
src = sqlite3.connect('/data/kavita.db')
dst = sqlite3.connect('/out/kavita-backup.db')
src.backup(dst)
dst.close(); src.close()
\""

scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-backup.db kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/

# On kscloud1: replace kavita.db
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
  cd /opt/kitestacks/docker/kavita
  docker compose stop kavita
  rm -f config/kavita.db config/kavita.db-wal config/kavita.db-shm
  cp /tmp/kavita-backup.db config/kavita.db
  chown 1000:1000 config/kavita.db   # kavita container runs as root, but 1000:1000 is fine
  docker compose start kavita
EOF
```

After sync, configure Kavita OIDC via SSH port-forward (NOT direct DB edit):

```bash
ssh -L 5099:localhost:<port> kenpat@<KSCLOUD1_PUBLIC_IP>
# Open http://localhost:<port> in browser
# Settings → OIDC: Authority=https://auth.kitestacks.com/application/o/kavita/ (trailing slash!)
# Client ID: kavita, Secret: (96-char hex from Authentik), Enabled: true
```

### 7.5 Sync Kavita cover images from monk to kscloud1

```bash
# On monk:
tar czf /tmp/kavita-covers.tar.gz -C ~/kitestacks-live/docker/kavita/config/covers .
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-covers.tar.gz kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/

# On kscloud1:
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
  mkdir -p /opt/kitestacks/docker/kavita/config/covers
  docker run --rm \
    -v /opt/kitestacks/docker/kavita/config/covers:/covers \
    -v /tmp/kavita-covers.tar.gz:/covers.tar.gz \
    alpine sh -c "cd /covers && tar xzf /covers.tar.gz && chown -R 1000:1000 ."
EOF
```

Note: book files themselves are not synced. Cover images load correctly; browsing the library when served by kscloud1 shows entries but no actual files — this is the accepted stale-data tradeoff.

### 7.6 Grafana provisioning on kscloud1

Same provisioning structure as monk. The Prometheus data source on kscloud1 points to kscloud1's own Prometheus (`http://prometheus:<port>`), which only scrapes kscloud1's node-exporter (monk is behind home NAT, not reachable from kscloud1 directly).

### 7.7 ufw — allow metrics API

```bash
echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from <IP_REDACTED>/12 to any port <port> proto tcp
```

---

## Phase 8 — Portal UI

The portal lives at `~/kitestacks-live/docker/kitestacks-portal/public/index.html` on monk and must be kept in sync across all 3 copies:

| Host | Path |
|------|------|
| monk (production) | `~/kitestacks-live/docker/kitestacks-portal/public/index.html` |
| monk (test) | `~/kitestacks-live/docker/kitestacks-portal-test/public/index.html` |
| kscloud1 | `/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html` |

### Portal Panels

| Panel | Color | Cards |
|-------|-------|-------|
| INFRASTRUCTURE | cyan | Portainer (live), Authentik, Cloudflare |
| MONITORING | magenta | Grafana, Prometheus (coming-soon), Node Exporter (coming-soon), Uptime Kuma |
| AI & AUTOMATION | purple | Kite AI (coming-soon), LiteLLM (coming-soon), OpenRouter (coming-soon), FluxCD (coming-soon) |
| KNOWLEDGE BASE | pink | Kavita, Karakeep |
| DEVELOPMENT | cyan | Forgejo, OSticket |
| SYSTEM STATUS | — | live metrics from /api/metrics |
| RECENT ACTIVITY | — | last 8 commits from Forgejo via /api/activity |
| COMMUNITY | — | Discord invite link |

### Update workflow

Edit `public/index.html` on monk test portal first, verify at `http://localhost:<port>`, then copy to production and kscloud1.

```bash
# Test change at localhost:<port> first, then:
cp ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html \
   ~/kitestacks-live/docker/kitestacks-portal/public/index.html

scp ~/kitestacks-live/docker/kitestacks-portal/public/index.html \
    kenpat@<KSCLOUD1_PUBLIC_IP>:/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html
```

No container restarts needed — nginx serves the files directly from the bind-mounted volume.

---

## Phase 9 — Monitoring

### Prometheus scrapes

Monk's Prometheus scrapes both:
- `node-exporter:<port>` (monk itself, via Docker DNS)
- `<KSCLOUD1_PUBLIC_IP>:<port>` (kscloud1, direct public IP — kscloud1's node-exporter is <IP_REDACTED>:9100)

kscloud1's Prometheus only scrapes itself (monk is behind home NAT).

### Grafana "Node Exporter Full" dashboard

Dashboard ID 1860 (provisioned from file). Use the `instance` dropdown to switch between `monk` and `kscloud1` views. Both instances appear because Prometheus has both scrape jobs.

---

## Phase 10 — Discord Integration

### Discord card on portal

The Community panel on the portal has a Discord card linking to `https://discord.gg/QbdveTb6Kw`. No setup needed beyond keeping the invite valid.

### Discord #recent-activities push webhook

A webhook posts to the Discord `#recent-activities` channel when Forgejo receives new commits. This is configured in Forgejo's webhook settings per-repository:

1. In Forgejo, go to a repo → Settings → Webhooks → Add Webhook → Gitea (compatible format)
2. Target URL: your Discord webhook URL (Discord → channel settings → Integrations → Webhooks → Copy Webhook URL)
3. Content type: `application/json`
4. Events: Push events (select relevant events)
5. Active: enabled

The webhook URL is stored only in Forgejo's DB — do not commit it to this repo.

---

## Phase 11 — Verification Checklist

Run after any major change or new host deployment.

### Public subdomains

```bash
for sub in www auth gitforge tasks ai links kavita grafana status; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com")
  echo "$sub: $code"
done
```

Expected: all return 200 (or 301/302 for redirect-based logins).

### SSO flows

- [ ] `https://auth.kitestacks.com` → Authentik admin login works
- [ ] `https://grafana.kitestacks.com` → "Sign in with authentik" → lands in Grafana
- [ ] `https://ai.kitestacks.com` → "Sign in with Authentik" → lands in Open WebUI
- [ ] `https://gitforge.kitestacks.com` → "Sign In with authentik" → lands in Forgejo
- [ ] `https://links.kitestacks.com` → Karakeep login with Authentik → works
- [ ] `https://kavita.kitestacks.com` → "Sign in with authentik" → works

### Failover test (disconnect monk's internet)

With monk's home network off (phone hotspot or at a different location):
```bash
for sub in www auth gitforge tasks ai links kavita grafana status; do
  code=$(curl -sk -o /dev/null -w "%{http_code}" "https://${sub}.kitestacks.com")
  echo "$sub: $code"
done
```

All 9 should still return 200 (served by kscloud1).

### Authentik shared DB health

```bash
# On monk — check authentik can reach shared DB
docker exec authentik sh -c 'python3 -c "import psycopg2; psycopg2.connect(host=\"<KSCLOUD1_TAILSCALE_IP>\", dbname=\"authentik\", user=\"authentik\", password=\"$AUTHENTIK_POSTGRESQL__PASSWORD\")"' && echo "DB reachable"

# Check both authentik containers are healthy
docker inspect --format '{{.Name}}: {{.State.Health.Status}}' authentik authentik-worker
```

---

## Key File Locations Reference

| What | Path on monk |
|------|-------------|
| All Docker services | `~/kitestacks-live/docker/` |
| Portal HTML (production) | `~/kitestacks-live/docker/kitestacks-portal/public/index.html` |
| Portal HTML (test) | `~/kitestacks-live/docker/kitestacks-portal-test/public/index.html` |
| Metrics API source | `~/kitestacks-live/docker/kitestacks-portal-test/api/main.py` |
| Prometheus config | `~/kitestacks-live/docker/prometheus/prometheus.yml` |
| Grafana provisioning | `~/kitestacks-live/docker/grafana/provisioning/` |
| Book library | `~/kitestacks-live/library/books/` |
| Kavita config | `~/kitestacks-live/docker/kavita/config/` |
| Kavita cover images | `~/kitestacks-live/docker/kavita/config/covers/` |
| Authentik media | `~/kitestacks-live/docker/authentik/media/` |
| Homelab git repo | `~/kitestacks-from-assassin-t14/opt/kitestacks-autosync/kitestacks-homelab/` (or cloned from gitforge.kitestacks.com) |

| What | Path on kscloud1 |
|------|-----------------|
| All Docker services | `/opt/kitestacks/docker/` |
| Shared Authentik Postgres data | `/opt/kitestacks/docker/authentik/postgres/` |
| Portal HTML | `/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html` |

---

## Upcoming: Oracle Cloud Migration

kscloud1 (Hetzner) is planned to be replaced by an Oracle Cloud VPS. When the new VPS is ready:

1. Provision Oracle VPS, install Docker + Tailscale
2. Deploy all services using the same Phase 7 playbook
3. Note: shared Authentik Postgres+Redis must move too — update both monk's and all connectors' `AUTHENTIK_REDIS__HOST` and `AUTHENTIK_POSTGRESQL__HOST` to the new Tailscale IP
4. Add Oracle VPS cloudflared as a new tunnel connector (same TUNNEL_TOKEN)
5. Verify all 9 subdomains work from Oracle VPS
6. Remove kscloud1 cloudflared from the tunnel connectors
7. Decommission kscloud1 Hetzner server

---

## Secrets Inventory (keys only — values in .env files)

| .env location | Keys |
|---------------|------|
| `authentik/.env` | `AUTHENTIK_SECRET_KEY`, `PG_PASS` |
| `grafana/.env` | `GRAFANA_OAUTH_CLIENT_SECRET` |
| `osticket/.env` | `OSTICKET_DB_PASS`, `OSTICKET_DB_ROOT`, `OSTICKET_ADMIN_PASS`, `OSTICKET_INSTALL_SECRET` |
| `kite-ai/.env` | `OPENROUTER_API_KEY`, `LITELLM_MASTER_KEY`, `WEBUI_SECRET_KEY`, `OPENWEBUI_OAUTH_CLIENT_SECRET` |
| `karakeep/.env` | `KARAKEEP_VERSION`, `NEXTAUTH_SECRET`, `NEXTAUTH_URL`, `MEILI_MASTER_KEY`, `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`, `OAUTH_WELLKNOWN_URL`, `OAUTH_PROVIDER_NAME`, `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING` |
| `cloudflared/docker-compose.yml` | `TUNNEL_TOKEN` (inline) |

Do not commit `.env` files, webhook URLs, or database files to this repo.

---

## Phase 8: Forgejo Sync + osTicket Authentik LDAP SSO (2026-06-14/15)

### Forgejo Sync (monk → kscloud1)

Monk is authoritative. kscloud1 Forgejo is a read replica synced every 6 hours.

**Sync script:** `~/kitestacks-live/docker/forgejo/sync-to-cloud.sh`
**Cron:** `0 */6 * * *` on monk, logs to `/tmp/forgejo-sync.log`

Manual sync:
```bash
~/kitestacks-live/docker/forgejo/sync-to-cloud.sh
```

To re-do a full restore from scratch (e.g., after kscloud1 rebuild):
```bash
# On monk: create dump
docker exec -u git forgejo /app/gitea/gitea dump --type zip -f /tmp/forgejo-backup.zip
docker cp forgejo:/tmp/forgejo-backup.zip /tmp/forgejo-backup.zip
# Transfer and restore on cloud host — see claude-memory for detailed steps
```

### osTicket Authentik LDAP SSO

Staff log into `tasks.kitestacks.com/scp/` using their **Authentik credentials** (not a separate osTicket password).

**Architecture:**
```
osticket-app → authentik-ldap-proxy:389 (socat) → authentik-ldap:3389 → auth.kitestacks.com
```

**Services deployed:**
- `~/kitestacks-live/docker/authentik-ldap/` — LDAP outpost + socat proxy on monk
- `/opt/kitestacks/docker/authentik-ldap/` — LDAP outpost on kscloud1

**LDAP search account:** `cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io`
Password stored in Authentik and in osTicket's `ost_config` (namespace `plugin.2`, key `bind_pw`, encrypted).

**auth-ldap.phar** at `/data/upload/include/plugins/auth-ldap.phar` inside the osticket-app container has been patched (original at `.phar.orig`). Do NOT replace it with the upstream version — the patch is required for PHP 7.3 + PEAR compatibility.

**If LDAP login stops working:**
```bash
# Check LDAP outpost is running and connected
docker logs authentik-ldap --since 5m | grep -v debug
docker logs authentik-ldap-proxy 2>&1 | tail -5

# Test bind from osticket-app container
docker exec osticket-app php -r "
  \$c = @ldap_connect('authentik-ldap-proxy');
  ldap_set_option(\$c, LDAP_OPT_PROTOCOL_VERSION, 3);
  \$r = @ldap_bind(\$c, 'cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io', 'PASSWORD');
  echo \$r ? 'OK' : ldap_error(\$c);
"
# Verify kscloud1 outpost reachable
nc -zv 100.123.254.52 3389
```

**Reset a staff member's Authentik password:**
```bash
docker exec authentik ak shell -c "
from authentik.core.models import User
u = User.objects.get(username='kenpat7177')
u.set_password('NewPassword123!')
u.save()
print('done')
"
```

**Clear osTicket login lockout:**
```bash
docker run --rm --network host mariadb:10.11 mysql \
  -h 100.123.254.52 -u osticket -p<DB_PASS> osticket \
  -e "DELETE FROM ost_session;"
```