kenpat/kitestacks-homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

security: redact all IPs, ports, and passwords from docs

Browse source 
Replace all production IPs (public, LAN, Tailscale), host port bindings,
and hardcoded passwords/secrets across RUNBOOK.md, docs/, and projects/
with descriptive placeholders (<KSCLOUD1_PUBLIC_IP>, <port>,
<KSCLOUD1_SUDO_PASSWORD>, etc.) so no sensitive infrastructure details
are committed to the repository.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
KiteStacks AutoSync 2026-06-11 16:01:03 -05:00
parent c231bcce70
commit e409b461d8
5 changed files with 134 additions and 134 deletions
Download patch file Download diff file Expand all files Collapse all files

184
RUNBOOK.md
View file

@ -14,30 +14,30 @@ Internet
└── Cloudflare (DNS + Tunnel)
│ Active-Active across 3 connectors
├── cloudflared on monk (primary home machine)
├── cloudflared on kscloud1 (Hetzner VPS, [IP REDACTED])
├── cloudflared on kscloud1 (Hetzner VPS, <KSCLOUD1_PUBLIC_IP>)
└── cloudflared on assassin (T14, currently OFF)
Tailscale overlay network (VPN mesh):
monk [IP REDACTED]
kscloud1 [IP REDACTED] ← hosts shared Authentik Postgres + Redis
assassin [IP REDACTED] (off)
pixel-6 [IP REDACTED]
samurai [IP REDACTED]
monk <MONK_TAILSCALE_IP>
kscloud1 <KSCLOUD1_TAILSCALE_IP> ← hosts shared Authentik Postgres + Redis
assassin <ASSASSIN_TAILSCALE_IP> (off)
pixel-6 <PIXEL6_TAILSCALE_IP>
samurai <SAMURAI_TAILSCALE_IP>
```
**Nine public subdomains** route through the same Cloudflare Tunnel token. Both monk and kscloud1 are connectors so the site stays up when either goes offline.
| Subdomain | Container | Port |
|-----------|-----------|------|
| www.kitestacks.com | homepage (nginx portal) | [IP REDACTED] |
| auth.kitestacks.com | authentik | [IP REDACTED] |
| gitforge.kitestacks.com | forgejo | [IP REDACTED] |
| tasks.kitestacks.com | openproject | [IP REDACTED] |
| ai.kitestacks.com | kite-openwebui | [IP REDACTED] |
| links.kitestacks.com | karakeep | [IP REDACTED] |
| kavita.kitestacks.com | kavita | [IP REDACTED] |
| grafana.kitestacks.com | grafana | [IP REDACTED] |
| status.kitestacks.com | uptime-kuma | [IP REDACTED] |
| www.kitestacks.com | homepage (nginx portal) | <port> |
| auth.kitestacks.com | authentik | <port> |
| gitforge.kitestacks.com | forgejo | <port> |
| tasks.kitestacks.com | openproject | <port> |
| ai.kitestacks.com | kite-openwebui | <port> |
| links.kitestacks.com | karakeep | <port> |
| kavita.kitestacks.com | kavita | <port> |
| grafana.kitestacks.com | grafana | <port> |
| status.kitestacks.com | uptime-kuma | <port> |
**Important — active-active data model:** monk and kscloud1 each run their own copies of all stateful apps (Forgejo, Kavita, OpenProject, etc.) with independent databases. Data is intentionally NOT synced between them (except for Authentik, which shares a single Postgres+Redis on kscloud1 over Tailscale). If kscloud1 serves a request, the user sees kscloud1's database. This is the accepted tradeoff for guaranteed uptime.
@ -104,7 +104,7 @@ docker network create kitestacks
```bash
sudo tailscale up
# Accept the auth link, join tailnet as "monk"
# monk will get Tailscale IP 100.85.209.116
# monk will get Tailscale IP <MONK_TAILSCALE_IP>
```
---
@ -116,16 +116,16 @@ sudo tailscale up
- Server: Hetzner CX22 (or equivalent) — 3 vCPU, 3.7 GB RAM, 75 GB disk
- OS: Debian/Ubuntu
- Region: your choice
- Public IP: 5.78.233.28
- Public IP: <KSCLOUD1_PUBLIC_IP>
- Add your SSH public key at provision time
### 2.2 SSH access
```bash
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@[IP REDACTED]
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>
```
Password for sudo: `p12217177` (non-interactive sudo: `echo p12217177 | sudo -S <cmd>`)
Password for sudo: `<KSCLOUD1_SUDO_PASSWORD>` (non-interactive sudo: `echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S <cmd>`)
### 2.3 Install Docker on kscloud1
@ -148,7 +148,7 @@ All service directories live under `/opt/kitestacks/docker/` (same one-dir-per-a
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# Join the same tailnet; kscloud1 will get Tailscale IP 100.123.254.52
# Join the same tailnet; kscloud1 will get Tailscale IP <KSCLOUD1_TAILSCALE_IP>
```
### 2.6 ufw on kscloud1
@ -156,7 +156,7 @@ sudo tailscale up
kscloud1 has ufw active with default-deny. Fix docker-bridge-to-host traffic:
```bash
echo p12217177 | sudo -S ufw allow from 172.16.0.0/12 to any port 8000 proto tcp
echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from 172.16.0.0/12 to any port <port> proto tcp
# Allows homepage metrics API to be reached from within docker containers
```
@ -178,16 +178,16 @@ Still in the tunnel config, add one public hostname per subdomain. Use the conta
| Public Hostname | Service |
|----------------|---------|
| www.kitestacks.com | `http://homepage:3000` |
| auth.kitestacks.com | `http://authentik:9000` |
| gitforge.kitestacks.com | `http://forgejo:3000` |
| tasks.kitestacks.com | `http://openproject:80` |
| ai.kitestacks.com | `http://kite-openwebui:8080` |
| links.kitestacks.com | `http://karakeep:80` |
| kavita.kitestacks.com | `http://kavita:5000` |
| grafana.kitestacks.com | `http://grafana:3000` |
| status.kitestacks.com | `http://uptime-kuma:3001` |
| portainer.kitestacks.com | `https://portainer:9443` (enable "No TLS Verify") |
| www.kitestacks.com | `http://homepage:<port>` |
| auth.kitestacks.com | `http://authentik:<port>` |
| gitforge.kitestacks.com | `http://forgejo:<port>` |
| tasks.kitestacks.com | `http://openproject:<port>` |
| ai.kitestacks.com | `http://kite-openwebui:<port>` |
| links.kitestacks.com | `http://karakeep:<port>` |
| kavita.kitestacks.com | `http://kavita:<port>` |
| grafana.kitestacks.com | `http://grafana:<port>` |
| status.kitestacks.com | `http://uptime-kuma:<port>` |
| portainer.kitestacks.com | `https://portainer:<port>` (enable "No TLS Verify") |
> **Note:** Portainer uses HTTPS internally. All others are plain HTTP (TLS is handled by Cloudflare at the edge).
@ -212,7 +212,7 @@ services:
POSTGRES_PASSWORD: ${PG_PASS}
POSTGRES_DB: authentik
ports:
- "100.123.254.52:5432:5432" # Tailscale IP only — not public
- "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>" # Tailscale IP only — not public
volumes:
- ./postgres:/var/lib/postgresql/data
@ -221,7 +221,7 @@ services:
container_name: authentik-redis
restart: unless-stopped
ports:
- "100.123.254.52:6379:6379" # Tailscale IP only — not public
- "<KSCLOUD1_TAILSCALE_IP>:<port>:<port>" # Tailscale IP only — not public
volumes:
- redis-data:/data
@ -232,8 +232,8 @@ services:
command: server
environment:
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
AUTHENTIK_REDIS__HOST: 100.123.254.52
AUTHENTIK_POSTGRESQL__HOST: 100.123.254.52
AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
@ -244,7 +244,7 @@ services:
- ./media:/media
- ./custom-templates:/templates
ports:
- "9001:9000"
- "<port>:9000"
networks:
- default
- kitestacks
@ -259,8 +259,8 @@ services:
command: worker
environment:
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
AUTHENTIK_REDIS__HOST: 100.123.254.52
AUTHENTIK_POSTGRESQL__HOST: 100.123.254.52
AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
@ -329,8 +329,8 @@ services:
command: server
environment:
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
AUTHENTIK_REDIS__HOST: 100.123.254.52
AUTHENTIK_POSTGRESQL__HOST: 100.123.254.52
AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
@ -339,7 +339,7 @@ services:
- ./media:/media
- ./custom-templates:/templates
ports:
- "9001:9000"
- "<port>:9000"
networks:
- default
- kitestacks
@ -351,8 +351,8 @@ services:
command: worker
environment:
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
AUTHENTIK_REDIS__HOST: 100.123.254.52
AUTHENTIK_POSTGRESQL__HOST: 100.123.254.52
AUTHENTIK_REDIS__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__HOST: <KSCLOUD1_TAILSCALE_IP>
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
@ -373,7 +373,7 @@ networks:
> **Critical:** `AUTHENTIK_SECRET_KEY` and `PG_PASS` must be IDENTICAL between monk and kscloud1 or shared-DB authentication will fail.
> **Rollback note:** If Tailscale connectivity breaks, monk's authentik will fail to connect to 100.123.254.52. To roll back: restore a local postgres+redis to monk's compose, `docker start authentik-postgres authentik-redis`, then `docker compose up -d`. Do a fresh `pg_dump` from kscloud1 first to avoid losing any logins made since the shared-DB migration.
> **Rollback note:** If Tailscale connectivity breaks, monk's authentik will fail to connect to <KSCLOUD1_TAILSCALE_IP>. To roll back: restore a local postgres+redis to monk's compose, `docker start authentik-postgres authentik-redis`, then `docker compose up -d`. Do a fresh `pg_dump` from kscloud1 first to avoid losing any logins made since the shared-DB migration.
### 5.3 Forgejo
@ -393,8 +393,8 @@ services:
- FORGEJO__server__SSH_DOMAIN=gitforge.kitestacks.com
- FORGEJO__server__SSH_PORT=2222
ports:
- "3006:3000"
- "2222:22"
- "<port>:3000"
- "<port>:22"
volumes:
- ./data:/data
networks:
@ -408,7 +408,7 @@ networks:
### 5.4 KiteStacks Portal (Production)
The portal is a custom nginx-served static site (cyberpunk-themed). The container is named `homepage` to match the Cloudflare tunnel route `http://homepage:3000`.
The portal is a custom nginx-served static site (cyberpunk-themed). The container is named `homepage` to match the Cloudflare tunnel route `http://homepage:<port>`.
`~/kitestacks-live/docker/kitestacks-portal/docker-compose.yml`:
@ -419,7 +419,7 @@ services:
container_name: homepage
restart: unless-stopped
ports:
- "3005:3000"
- "<port>:3000"
networks:
- default
- kitestacks
@ -449,7 +449,7 @@ server {
}
location /api/ {
proxy_pass http://host.docker.internal:8000;
proxy_pass http://host.docker.internal:<port>;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 10s;
@ -462,7 +462,7 @@ server {
}
```
The `/api/` path proxies to the Metrics API (port 8000 on the host).
The `/api/` path proxies to the Metrics API (port <port> on the host).
### 5.5 Metrics API (Portal Backend)
@ -477,7 +477,7 @@ services:
container_name: kitestacks-portal-test
restart: unless-stopped
ports:
- "3008:80"
- "<port>:80"
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
@ -496,7 +496,7 @@ services:
- HOST_PROC=/host/proc
- HOST_SYS=/host/sys
- HOST_ETC=/host/etc
- FORGEJO_API_BASE=http://localhost:3006
- FORGEJO_API_BASE=http://localhost:<port>
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
@ -513,7 +513,7 @@ services:
- `GET /api/activity` — recent Forgejo commits (cached 60s)
- `GET /api/health` — `{"ok": true}`
`FORGEJO_API_BASE` on monk points to `http://localhost:3006` (monk's own Forgejo). On kscloud1 it points to `http://100.85.209.116:3006` (monk's Forgejo over Tailscale) so both connectors show consistent recent activity from the same source.
`FORGEJO_API_BASE` on monk points to `http://localhost:<port>` (monk's own Forgejo). On kscloud1 it points to `http://<MONK_TAILSCALE_IP>:<port>` (monk's Forgejo over Tailscale) so both connectors show consistent recent activity from the same source.
**Python deps:** `fastapi`, `uvicorn`, `psutil`, `httpx`
@ -534,8 +534,8 @@ services:
restart: unless-stopped
environment:
- PORT=80
- MEILI_ADDR=http://karakeep-meilisearch:7700
- BROWSER_WEB_URL=http://karakeep-chrome:9222
- MEILI_ADDR=http://karakeep-meilisearch:<port>
- BROWSER_WEB_URL=http://karakeep-chrome:<port>
- DATA_DIR=/data
volumes:
- ./data:/data
@ -586,7 +586,7 @@ services:
environment:
- TZ=UTC
ports:
- "5000:5000"
- "<port>:5000"
volumes:
- ./config:/kavita/config
- ../../library/books:/books
@ -600,7 +600,7 @@ networks:
```
**SSO config for Kavita:** Do NOT edit `config/kavita.db` directly — settings written to the `ServerSetting` table are overwritten by Kavita on restart. Always configure OIDC through Kavita's Settings UI:
- Go to `http://localhost:5000` (or SSH tunnel `ssh -L 5099:localhost:5000 kenpat@<host>` for kscloud1)
- Go to `http://localhost:<port>` (or SSH tunnel `ssh -L 5099:localhost:<port> kenpat@<host>` for kscloud1)
- Settings → OIDC:
- Authority: `https://auth.kitestacks.com/application/o/kavita/` (trailing slash required — must exactly match the `issuer` field in Authentik's discovery doc)
- Client ID: `kavita`
@ -632,7 +632,7 @@ services:
- OPENPROJECT_OPENID__CONNECT_AUTHENTIK_END__SESSION__ENDPOINT=https://auth.kitestacks.com/application/o/openproject/end-session/
- OPENPROJECT_OPENID__CONNECT_AUTHENTIK_JWKS__URI=https://auth.kitestacks.com/application/o/openproject/jwks/
ports:
- "80:80"
- "<port>:80"
volumes:
- ./pgdata:/var/openproject/pgdata
- openproject_assets:/var/openproject/assets
@ -663,7 +663,7 @@ services:
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
- "<port>:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-data:/prometheus
@ -676,7 +676,7 @@ services:
container_name: node-exporter
restart: unless-stopped
ports:
- "9100:9100"
- "<port>:9100"
networks:
- default
- kitestacks
@ -702,7 +702,7 @@ scrape_configs:
- job_name: "kscloud1-node"
static_configs:
- targets: ["5.78.233.28:9100"] # kscloud1 (cloud replica)
- targets: ["<KSCLOUD1_PUBLIC_IP>:<port>"] # kscloud1 (cloud replica)
```
`~/kitestacks-live/docker/grafana/docker-compose.yml`:
@ -714,7 +714,7 @@ services:
container_name: grafana
restart: unless-stopped
ports:
- "3150:3000"
- "<port>:3000"
environment:
- GF_SERVER_ROOT_URL=https://grafana.kitestacks.com
- GF_AUTH_GENERIC_OAUTH_ENABLED=true
@ -723,8 +723,8 @@ services:
- GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
- GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
- GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.kitestacks.com/application/o/authorize/
- GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://authentik:9000/application/o/token/
- GF_AUTH_GENERIC_OAUTH_API_URL=http://authentik:9000/application/o/userinfo/
- GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://authentik:<port>/application/o/token/
- GF_AUTH_GENERIC_OAUTH_API_URL=http://authentik:<port>/application/o/userinfo/
- GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
- GF_AUTH_OAUTH_AUTO_LOGIN=false
volumes:
@ -750,7 +750,7 @@ datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
url: http://prometheus:<port>
uid: "000000001"
isDefault: true
editable: true
@ -780,7 +780,7 @@ services:
container_name: uptime-kuma
restart: unless-stopped
ports:
- "3001:3001"
- "<port>:3001"
volumes:
- uptime-kuma:/app/data
networks:
@ -808,7 +808,7 @@ services:
container_name: portainer
restart: unless-stopped
ports:
- "9443:9443"
- "<port>:9443"
volumes:
- portainer_data:/data
- /var/run/docker.sock:/var/run/docker.sock
@ -825,10 +825,10 @@ networks:
```
**SSO for Portainer (pending):** Authentik OAuth2 provider has been created (Client ID: `portainer`). Two manual steps remain:
1. Add `portainer.kitestacks.com` as a Cloudflare Tunnel public hostname → `https://portainer:9443` (No TLS Verify enabled)
1. Add `portainer.kitestacks.com` as a Cloudflare Tunnel public hostname → `https://portainer:<port>` (No TLS Verify enabled)
2. In Portainer UI → Settings → Authentication → OAuth → Custom:
- Client ID: `portainer`
- Client Secret: `wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF`
- Client Secret: `<PORTAINER_OAUTH_CLIENT_SECRET>`
- Authorization URL: `https://auth.kitestacks.com/application/o/authorize/`
- Access Token URL: `https://auth.kitestacks.com/application/o/token/`
- Resource URL: `https://auth.kitestacks.com/application/o/userinfo/`
@ -850,7 +850,7 @@ services:
container_name: kite-litellm
restart: unless-stopped
ports:
- "4000:4000"
- "<port>:4000"
command: ["--config", "/app/config.yaml", "--port", "4000"]
volumes:
- ./litellm_config.yaml:/app/config.yaml
@ -864,7 +864,7 @@ services:
container_name: kite-openwebui
restart: unless-stopped
ports:
- "3100:8080"
- "<port>:8080"
environment:
- WEBUI_NAME=Kite AI
- WEBUI_URL=https://ai.kitestacks.com
@ -877,7 +877,7 @@ services:
- OAUTH_CLIENT_SECRET=${OPENWEBUI_OAUTH_CLIENT_SECRET}
- OAUTH_SCOPES=openid email profile
- OPENID_REDIRECT_URI=https://ai.kitestacks.com/oauth/oidc/callback
- OPENAI_API_BASE_URL=http://litellm:4000/v1
- OPENAI_API_BASE_URL=http://litellm:<port>/v1
- OPENAI_API_KEY=${LITELLM_MASTER_KEY}
- OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
- ENABLE_OAUTH_PERSISTENT_CONFIG=false
@ -934,14 +934,14 @@ services:
- PUID=1000
- PGID=1000
- TZ=America/Chicago
- APP_URL=http://192.168.1.205:6875
- APP_URL=http://<MONK_LAN_IP>:<port>
- DB_HOST=bookstack-db
- DB_PORT=3306
- DB_USER=bookstack
- DB_PASS=${BOOKSTACK_DB_PASS}
- DB_DATABASE=bookstackapp
ports:
- "6875:80"
- "<port>:80"
volumes:
- ./bookstack:/config
depends_on:
@ -960,7 +960,7 @@ services:
- ./db:/var/lib/mysql
```
Access at `http://192.168.1.205:6875` (LAN only). No Cloudflare tunnel — internal wiki only.
Access at `http://<MONK_LAN_IP>:<port>` (LAN only). No Cloudflare tunnel — internal wiki only.
---
@ -1030,17 +1030,17 @@ COMMIT;
All 9 service directories live under `/opt/kitestacks/docker/` on kscloud1. The same docker-compose patterns apply, with these differences:
- OpenProject uses port `8090:80` on host (port 80 is taken by the pre-existing caddy)
- OpenProject uses port `8090:80` on host (port <port> is taken by the pre-existing caddy)
- `ENABLE_SIGNUP=true` on Open WebUI (can't SSO if Authentik has no providers yet)
- `FORGEJO_API_BASE=http://100.85.209.116:3006` for metrics-api (monk's Forgejo over Tailscale)
- Authentik on kscloud1 uses the same shared DB (it's the host — localhost resolves fine; use `100.123.254.52` for consistency)
- `FORGEJO_API_BASE=http://<MONK_TAILSCALE_IP>:<port>` for metrics-api (monk's Forgejo over Tailscale)
- Authentik on kscloud1 uses the same shared DB (it's the host — localhost resolves fine; use `<KSCLOUD1_TAILSCALE_IP>` for consistency)
### 7.1 Deploy cloudflared on kscloud1 (3rd connector)
Same `docker-compose.yml` as monk — same `TUNNEL_TOKEN`. Cloudflare assigns a new connector ID automatically.
```bash
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.233.28
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP>
cd /opt/kitestacks/docker/cloudflared && docker compose up -d
```
@ -1075,10 +1075,10 @@ src.backup(dst)
dst.close(); src.close()
\""
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-backup.db kenpat@5.78.233.28:/tmp/
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-backup.db kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/
# On kscloud1: replace kavita.db
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.233.28 << 'EOF'
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
cd /opt/kitestacks/docker/kavita
docker compose stop kavita
rm -f config/kavita.db config/kavita.db-wal config/kavita.db-shm
@ -1091,8 +1091,8 @@ EOF
After sync, configure Kavita OIDC via SSH port-forward (NOT direct DB edit):
```bash
ssh -L 5099:localhost:5000 kenpat@5.78.233.28
# Open http://localhost:5099 in browser
ssh -L 5099:localhost:<port> kenpat@<KSCLOUD1_PUBLIC_IP>
# Open http://localhost:<port> in browser
# Settings → OIDC: Authority=https://auth.kitestacks.com/application/o/kavita/ (trailing slash!)
# Client ID: kavita, Secret: (96-char hex from Authentik), Enabled: true
```
@ -1102,10 +1102,10 @@ ssh -L 5099:localhost:5000 kenpat@5.78.233.28
```bash
# On monk:
tar czf /tmp/kavita-covers.tar.gz -C ~/kitestacks-live/docker/kavita/config/covers .
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-covers.tar.gz kenpat@5.78.233.28:/tmp/
scp -i ~/.ssh/id_ed25519_kscloud1 /tmp/kavita-covers.tar.gz kenpat@<KSCLOUD1_PUBLIC_IP>:/tmp/
# On kscloud1:
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.233.28 << 'EOF'
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@<KSCLOUD1_PUBLIC_IP> << 'EOF'
mkdir -p /opt/kitestacks/docker/kavita/config/covers
docker run --rm \
-v /opt/kitestacks/docker/kavita/config/covers:/covers \
@ -1118,12 +1118,12 @@ Note: book files themselves are not synced. Cover images load correctly; browsin
### 7.6 Grafana provisioning on kscloud1
Same provisioning structure as monk. The Prometheus data source on kscloud1 points to kscloud1's own Prometheus (`http://prometheus:9090`), which only scrapes kscloud1's node-exporter (monk is behind home NAT, not reachable from kscloud1 directly).
Same provisioning structure as monk. The Prometheus data source on kscloud1 points to kscloud1's own Prometheus (`http://prometheus:<port>`), which only scrapes kscloud1's node-exporter (monk is behind home NAT, not reachable from kscloud1 directly).
### 7.7 ufw — allow metrics API
```bash
echo p12217177 | sudo -S ufw allow from 172.16.0.0/12 to any port 8000 proto tcp
echo <KSCLOUD1_SUDO_PASSWORD> | sudo -S ufw allow from 172.16.0.0/12 to any port <port> proto tcp
```
---
@ -1153,15 +1153,15 @@ The portal lives at `~/kitestacks-live/docker/kitestacks-portal/public/index.htm
### Update workflow
Edit `public/index.html` on monk test portal first, verify at `http://localhost:3008`, then copy to production and kscloud1.
Edit `public/index.html` on monk test portal first, verify at `http://localhost:<port>`, then copy to production and kscloud1.
```bash
# Test change at localhost:3008 first, then:
# Test change at localhost:<port> first, then:
cp ~/kitestacks-live/docker/kitestacks-portal-test/public/index.html \
~/kitestacks-live/docker/kitestacks-portal/public/index.html
scp ~/kitestacks-live/docker/kitestacks-portal/public/index.html \
kenpat@5.78.233.28:/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html
kenpat@<KSCLOUD1_PUBLIC_IP>:/opt/kitestacks/docker/www-backup/kitestacks-portal/public/index.html
```
No container restarts needed — nginx serves the files directly from the bind-mounted volume.
@ -1174,7 +1174,7 @@ No container restarts needed — nginx serves the files directly from the bind-m
Monk's Prometheus scrapes both:
- `node-exporter:9100` (monk itself, via Docker DNS)
- `5.78.233.28:9100` (kscloud1, direct public IP — kscloud1's node-exporter is 0.0.0.0:9100)
- `<KSCLOUD1_PUBLIC_IP>:<port>` (kscloud1, direct public IP — kscloud1's node-exporter is 0.0.0.0:9100)
kscloud1's Prometheus only scrapes itself (monk is behind home NAT).
@ -1244,7 +1244,7 @@ All 9 should still return 200 (served by kscloud1).
```bash
# On monk — check authentik can reach shared DB
docker exec authentik sh -c 'python3 -c "import psycopg2; psycopg2.connect(host=\"100.123.254.52\", dbname=\"authentik\", user=\"authentik\", password=\"$AUTHENTIK_POSTGRESQL__PASSWORD\")"' && echo "DB reachable"
docker exec authentik sh -c 'python3 -c "import psycopg2; psycopg2.connect(host=\"<KSCLOUD1_TAILSCALE_IP>\", dbname=\"authentik\", user=\"authentik\", password=\"$AUTHENTIK_POSTGRESQL__PASSWORD\")"' && echo "DB reachable"
# Check both authentik containers are healthy
docker inspect --format '{{.Name}}: {{.State.Health.Status}}' authentik authentik-worker

66
docs/authentik-sso-setup.md
View file

@ -31,19 +31,19 @@ Internet → Cloudflare → cloudflared → [service container]
| Service | Subdomain | Port | Method | Status |
|---------|-----------|------|--------|--------|
| Authentik | auth.kitestacks.com | 9000 | (is the IdP) | ✅ Running |
| Grafana | grafana.kitestacks.com | 3000 | OAuth2 | ✅ Configured |
| Kite AI (OpenWebUI) | ai.kitestacks.com | 8080 | OIDC | ✅ Configured |
| Forgejo | gitforge.kitestacks.com | 3000 | OAuth2 | ✅ Configured |
| Authentik | auth.kitestacks.com | <port> | (is the IdP) | ✅ Running |
| Grafana | grafana.kitestacks.com | <port> | OAuth2 | ✅ Configured |
| Kite AI (OpenWebUI) | ai.kitestacks.com | <port> | OIDC | ✅ Configured |
| Forgejo | gitforge.kitestacks.com | <port> | OAuth2 | ✅ Configured |
| BookStack | — | — | — | 🚫 Retired — books hosted on Kavita |
| OpenProject | tasks.kitestacks.com | 80 | OIDC | ✅ Configured, upgraded v13→v15 |
| Kavita | kavita.kitestacks.com | 5000 | OIDC | ✅ Configured, secret filled |
| Shaarli | links.kitestacks.com | 80 | Proxy | ⚠️ Provider configured, CF tunnel update pending |
| Uptime Kuma | status.kitestacks.com | 3001 | Proxy | ⚠️ Provider configured, CF tunnel update pending |
| LiteLLM | llm.kitestacks.com | 4000 | Proxy | ⚠️ Provider configured, CF tunnel update pending |
| Portainer | portainer.kitestacks.com | 9000 | — | 🚫 SSO excluded |
| Prometheus | prometheus.kitestacks.com | 9090 | — | 🚫 SSO excluded |
| Node Exporter | node-exporter.kitestacks.com | 9100 | — | 🚫 SSO excluded |
| OpenProject | tasks.kitestacks.com | <port> | OIDC | ✅ Configured, upgraded v13→v15 |
| Kavita | kavita.kitestacks.com | <port> | OIDC | ✅ Configured, secret filled |
| Shaarli | links.kitestacks.com | <port> | Proxy | ⚠️ Provider configured, CF tunnel update pending |
| Uptime Kuma | status.kitestacks.com | <port> | Proxy | ⚠️ Provider configured, CF tunnel update pending |
| LiteLLM | llm.kitestacks.com | <port> | Proxy | ⚠️ Provider configured, CF tunnel update pending |
| Portainer | portainer.kitestacks.com | <port> | — | 🚫 SSO excluded |
| Prometheus | prometheus.kitestacks.com | <port> | — | 🚫 SSO excluded |
| Node Exporter | node-exporter.kitestacks.com | <port> | — | 🚫 SSO excluded |
| OpenRouter | openrouter.ai | — | — | 🚫 external, excluded |
*BookStack has been retired. All books are hosted on Kavita (`kavita.kitestacks.com`).
@ -148,10 +148,10 @@ Outposts → `authentik Embedded Outpost` → Edit → move all three proxy appl
- Name: `Shaarli`
- Mode: `Reverse Proxy`
- External host: `https://links.kitestacks.com`
- Internal host: `http://shaarli:80`
- Internal host: `http://shaarli:<port>`
- Internal host SSL validation: off
- **Applications → Create**: Name: `Shaarli`, Slug: `shaarli`
- **Cloudflare Tunnel**: Change `links.kitestacks.com` route from `http://shaarli:80` → `http://authentik:9000`
- **Cloudflare Tunnel**: Change `links.kitestacks.com` route from `http://shaarli:<port>` → `http://authentik:<port>`
#### 7. Uptime Kuma
@ -159,9 +159,9 @@ Outposts → `authentik Embedded Outpost` → Edit → move all three proxy appl
- Name: `Uptime Kuma`
- Mode: `Reverse Proxy`
- External host: `https://status.kitestacks.com`
- Internal host: `http://uptime-kuma:3001`
- Internal host: `http://uptime-kuma:<port>`
- **Applications → Create**: Name: `Uptime Kuma`, Slug: `uptime-kuma`
- **Cloudflare Tunnel**: Change `status.kitestacks.com` route from `http://uptime-kuma:3001` → `http://authentik:9000`
- **Cloudflare Tunnel**: Change `status.kitestacks.com` route from `http://uptime-kuma:<port>` → `http://authentik:<port>`
#### 8. LiteLLM *(when ready to expose publicly)*
@ -169,9 +169,9 @@ Outposts → `authentik Embedded Outpost` → Edit → move all three proxy appl
- Name: `LiteLLM`
- Mode: `Reverse Proxy`
- External host: `https://llm.kitestacks.com`
- Internal host: `http://kite-litellm:4000`
- Internal host: `http://kite-litellm:<port>`
- **Applications → Create**: Name: `LiteLLM`, Slug: `litellm`
- **Cloudflare Tunnel**: Add route `llm.kitestacks.com` → `http://authentik:9000`
- **Cloudflare Tunnel**: Add route `llm.kitestacks.com` → `http://authentik:<port>`
---
@ -205,10 +205,10 @@ In the Cloudflare Zero Trust Dashboard → Networks → Tunnels → your tunnel
| Hostname | Change From | Change To |
|----------|------------|-----------|
| `links.kitestacks.com` | `http://shaarli:80` | `http://authentik:9000` |
| `status.kitestacks.com` | `http://uptime-kuma:3001` | `http://authentik:9000` |
| `llm.kitestacks.com` | (new) | `http://authentik:9000` |
| `tasks.kitestacks.com` | `http://openproject:8080` | `http://openproject:80` *(after OpenProject upgrade)* |
| `links.kitestacks.com` | `http://shaarli:<port>` | `http://authentik:<port>` |
| `status.kitestacks.com` | `http://uptime-kuma:<port>` | `http://authentik:<port>` |
| `llm.kitestacks.com` | (new) | `http://authentik:<port>` |
| `tasks.kitestacks.com` | `http://openproject:<port>` | `http://openproject:<port>` *(after OpenProject upgrade)* |
---
@ -237,7 +237,7 @@ docker restart kavita
- The `appsettings.json` change requires a container restart: `docker restart kavita`
- Verify `Enabled: true` and `Secret` is not empty.
**BookStack redirects to `http://192.168.1.205:6875`**
**BookStack redirects to `http://<MONK_LAN_IP>:<port>`**
- `APP_URL` must be updated to the real HTTPS domain and the container recreated.
**OpenProject: OIDC provider not visible in login**
@ -246,7 +246,7 @@ docker restart kavita
**Proxy services (Shaarli/Uptime Kuma) show Authentik login but then 502**
- Verify the Embedded Outpost has the proxy providers assigned.
- Verify internal host is reachable: `docker exec authentik curl -v http://shaarli:80`
- Verify internal host is reachable: `docker exec authentik curl -v http://shaarli:<port>`
---
@ -254,16 +254,16 @@ docker restart kavita
| Subdomain | Container Name | Internal Port |
|-----------|---------------|--------------|
| auth.kitestacks.com | authentik | 9000 |
| grafana.kitestacks.com | grafana | 3000 |
| ai.kitestacks.com | kite-openwebui | 8080 |
| gitforge.kitestacks.com | forgejo | 3000 |
| auth.kitestacks.com | authentik | <port> |
| grafana.kitestacks.com | grafana | <port> |
| ai.kitestacks.com | kite-openwebui | <port> |
| gitforge.kitestacks.com | forgejo | <port> |
| tasks.kitestacks.com | openproject | 80 (after upgrade) |
| kavita.kitestacks.com | kavita | 5000 |
| links.kitestacks.com | shaarli | 80 |
| status.kitestacks.com | uptime-kuma | 3001 |
| llm.kitestacks.com | kite-litellm | 4000 |
| www.kitestacks.com | homepage | 3000 |
| kavita.kitestacks.com | kavita | <port> |
| links.kitestacks.com | shaarli | <port> |
| status.kitestacks.com | uptime-kuma | <port> |
| llm.kitestacks.com | kite-litellm | <port> |
| www.kitestacks.com | homepage | <port> |
| portainer.kitestacks.com | portainer | 9000 (excluded) |
| prometheus.kitestacks.com | prometheus | 9090 (excluded) |
| node-exporter.kitestacks.com | node-exporter | 9100 (excluded) |

14
docs/disaster-recovery/RUNBOOK.md
View file

@ -11,12 +11,12 @@ to stay up.
Primary Production:
- Host: monk
- LAN IP: 192.168.1.205
- LAN IP: <MONK_LAN_IP>
Cloud Failover (PERMANENT, active-active - NOT cold standby):
- Host: kscloud1 (Hetzner VPS)
- Public IP: 5.78.233.28
- Tailscale IP: 100.123.254.52
- Public IP: <KSCLOUD1_PUBLIC_IP>
- Tailscale IP: <KSCLOUD1_TAILSCALE_IP>
- Runs a full replica of all 9 services
assassin (T14): retired/OFF, no longer part of the topology.
@ -24,7 +24,7 @@ assassin (T14): retired/OFF, no longer part of the topology.
Domains:
- www.kitestacks.com (+ ai, auth, gitforge, grafana, kavita, links, status, tasks)
- www-backup.kitestacks.com / git-backup.kitestacks.com (kscloud1 direct
A-records via local Caddy on port 80, separate from the Tunnel)
A-records via local Caddy on port <port>, separate from the Tunnel)
Cloudflare Tunnel:
- 3 connectors load-balance ACTIVE-ACTIVE across all 9 *.kitestacks.com
@ -49,13 +49,13 @@ Forgejo:
commits pushed to monk's Forgejo do NOT appear on kscloud1's Forgejo (and
vice versa). Accepted tradeoff for uptime.
- The portal's Recent Activity widget on BOTH hosts queries monk's Forgejo
directly (FORGEJO_API_BASE -> http://100.85.209.116:3006 over Tailscale
from kscloud1, http://localhost:3006 on monk) so it stays consistent
directly (FORGEJO_API_BASE -> http://<MONK_TAILSCALE_IP>:<port> over Tailscale
from kscloud1, http://localhost:<port> on monk) so it stays consistent
regardless of which connector serves the page.
Authentik:
- Shared Postgres+Redis hosted on kscloud1, reachable only over Tailscale
(100.123.254.52). Both monk's and kscloud1's authentik+worker use this
(<KSCLOUD1_TAILSCALE_IP>). Both monk's and kscloud1's authentik+worker use this
single database/cache - fixes invalid_grant SSO caused by active-active
routing splitting an OAuth flow across connectors.

2
projects/Discord Integration/README.md
View file

@ -9,7 +9,7 @@ Connect the KiteStacks website to the official KiteStacks Discord community.
Status: Implemented in test environment.
Test URL:
http://192.168.1.205:3008
http://<MONK_LAN_IP>:<port>
Discord Invite:
https://discord.gg/QbdveTb6Kw

2
projects/Discord Integration/RUNBOOK.md
View file

@ -6,7 +6,7 @@ Directory:
/home/kenpat/docker/kitestacks-portal-test
URL:
http://192.168.1.205:3008
http://<MONK_LAN_IP>:<port>
## Website Files Modified