kitestacks-homelab/RUNBOOK.md

KiteStacks Homelab — Complete Setup Runbook

Last Updated: 2026-06-18
Status: Production (monk primary, kscloud1 Hetzner cloud replica)
Maintainer: kenpat

---

Architecture Overview

Internet
   │
   └── Cloudflare (DNS + Tunnel)
          │  Active-Active across 2 connectors
          ├── cloudflared on monk (primary home machine, Docker container)
          └── cloudflared on kscloud1 (Hetzner VPS, <KSCLOUD1_PUBLIC_IP>)

Tailscale overlay network (VPN mesh):
  monk      <MONK_TAILSCALE_IP>
  kscloud1  <KSCLOUD1_TAILSCALE_IP>  ← hosts shared Authentik Postgres + Redis

Public subdomains route through the same Cloudflare Tunnel token. Both monk and kscloud1 are connectors so the site stays up if either goes offline.

Subdomain	Service	Port
auth.kitestacks.com	Authentik	9000
portainer.kitestacks.com	Portainer	9443
wiki.kitestacks.com	BookStack	6875 (monk) / 6877 (kscloud1)
grafana.kitestacks.com	Grafana	3000
gitforge.kitestacks.com	Forgejo	3006
links.kitestacks.com	Karakeep	3100
status.kitestacks.com	Uptime Kuma	3001
tasks.kitestacks.com	OSTicket	8080
flux.kitestacks.com	FluxCD	—

---

Service Inventory

Running on monk

authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy, bookstack, bookstack-db, cloudflared, flux, forgejo, grafana, karakeep, karakeep-chrome, karakeep-meilisearch, kavita, kite-litellm, kite-openwebui, kitestacks-metrics-api, kitestacks-portal, node-exporter, ntfy, osticket, osticket-app, osticket-db, portainer, prometheus, uptime-kuma, blackbox-exporter

Running on kscloud1 (extras)

bookstack, bookstack-db-ks, kite-monitor, osticket-app-118, osticket-db-118, www-backup, homepage-backup, cloudflared, authentik-postgresql, authentik-redis

Shared infrastructure on kscloud1

• PostgreSQL :5432 — Authentik DB used by both hosts (Tailscale only)
• Redis :6379 — Authentik session cache (Tailscale only)

---

Cloudflare Tunnel

How it works

Both monk and kscloud1 run cloudflared as Docker containers using the same tunnel token. Cloudflare load-balances across both connectors (active-active). The tunnel token is stored in:

• monk: ~/kitestacks-live/docker/cloudflared/.env → TUNNEL_TOKEN
• kscloud1: /opt/kitestacks/docker/cloudflared/.env → TUNNEL_TOKEN

Fix: Phantom 3rd Replica

If cloudflared tunnel info shows 3 connectors instead of 2, the native cloudflared systemd service on monk is running alongside the Docker container.

# Check systemd cloudflared on monk
systemctl status cloudflared
# Disable it — Docker container is the correct one
sudo systemctl disable --now cloudflared

Adding a new hostname route

In Cloudflare Zero Trust → Networks → Tunnels → your tunnel → Edit → Public Hostname:

• Subdomain: newservice
• Domain: kitestacks.com
• Service: http://container-name:port

Both monk and kscloud1 must have the container running on the same port.

---

Authentik SSO

Architecture

Authentik uses a shared database hosted on kscloud1. monk's Authentik containers connect via Tailscale.

• monk containers: authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy
• DB: PostgreSQL on kscloud1 at <KSCLOUD1_TAILSCALE_IP>:5432
• Redis: kscloud1 at <KSCLOUD1_TAILSCALE_IP>:6379

Adding OIDC SSO for a new app

1. In Authentik admin (https://auth.kitestacks.com/if/admin/):

   • Providers → Create → OAuth2/OpenID Provider
   • Name the provider after the app (e.g. bookstack)
   • Set issuer_mode based on the app's requirements (see Debug doc)
   • Note the Client ID and Client Secret

2. Application → Create → link to the provider

3. Policy Binding → bind the default-authentication-flow to the application

4. Configure the app with:

   • OIDC_ISSUER = discovery base URL
   • OIDC_CLIENT_ID / OIDC_CLIENT_SECRET
   • Callback URL = https://yourapp.kitestacks.com/auth/callback

Checking OIDC discovery URL

# Per-provider (issuer_mode=per_provider)
curl -s https://auth.kitestacks.com/application/o/<slug>/.well-known/openid-configuration | python3 -m json.tool

# Global (issuer_mode=global)
# Note: global issuer URL does NOT serve a JSON discovery doc at /.well-known/
# Use per-provider mode for apps that auto-discover endpoints (BookStack, etc.)

Changing provider issuer_mode via SQL

docker run --rm --network host \
  -e PGPASSWORD="<REDACTED>" \
  postgres:16 psql -h <KSCLOUD1_TAILSCALE_IP> -U authentik authentik -c \
  "UPDATE authentik_providers_oauth2_oauth2provider SET issuer_mode='per_provider' WHERE provider_ptr_id=<ID>;"

---

Portainer

OAuth setup (Authentik)

Portainer CE uses AuthenticationMethod=3 (OAuth). Configured via the BoltDB.

Key settings:

• OAuthLoginURI: https://auth.kitestacks.com/application/o/authorize/
• OAuthTokenURI: https://auth.kitestacks.com/application/o/token/
• OAuthUserURI: https://auth.kitestacks.com/application/o/userinfo/
• OAuthClientID: portainer
• OAuthRedirectURI: https://portainer.kitestacks.com
• OAuthAutoCreateUsers: true
• OAuthDefaultTeamID: 0

Pre-creating an admin user before first OAuth login

OAuth auto-created users default to Role:2 (regular user) and can't see environments. Pre-create them as Role:1 (admin) via the API before they log in:

# Get auth token
TOKEN=$(curl -sk -X POST https://portainer.kitestacks.com/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"<REDACTED>"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")

# Create user as admin (Role:1), no password needed for OAuth users
curl -sk -X POST "https://portainer.kitestacks.com/api/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"username":"user@example.com","role":1}'

Reset admin password (if locked out)

# Stop Portainer
docker stop portainer

# Reset password (shows new temp password)
docker run --rm -v portainer_data:/data portainer/helper-reset-password

# Restart
docker start portainer

---

BookStack

Setup (both monk and kscloud1)

Location:

• monk: ~/kitestacks-live/docker/bookstack/docker-compose.yml
• kscloud1: /opt/kitestacks/docker/bookstack/docker-compose.yml

Key environment variables:

- APP_URL=https://wiki.kitestacks.com
- DB_HOST=bookstack-db
- AUTH_METHOD=oidc
- OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/
- OIDC_ISSUER_DISCOVER=true
- OIDC_CLIENT_ID=bookstack
- OIDC_CLIENT_SECRET=<REDACTED>
- OIDC_USER_ATTRIBUTE=email
- APP_KEY=<REDACTED>

Generate APP_KEY

docker run --rm --entrypoint /bin/bash lscr.io/linuxserver/bookstack:latest appkey

OIDC Configuration

BookStack uses OIDC_ISSUER_DISCOVER=true to auto-discover all endpoints from Authentik. The OIDC_ISSUER must match the per-app discovery URL base (not the global Authentik URL).

The Authentik bookstack provider must have issuer_mode='per_provider' so its discovery document returns the correct per-app issuer URL. See Debug doc for full troubleshooting.

Fix cache permissions after artisan runs

Running php artisan as root creates root-owned cache dirs that block the app:

docker exec bookstack chown -R abc:users /config/www/framework/cache/

Clear Laravel config/cache

docker exec bookstack php /app/www/artisan config:clear
docker exec bookstack php /app/www/artisan cache:clear

---

kscloud1 Access

SSH

ssh -i ~/.ssh/id_ed25519_kscloud1 root@<KSCLOUD1_TAILSCALE_IP>

If SSH key is lost / not working

1. Open Hetzner Cloud console: console.hetzner.cloud → your server → Console tab
2. Log in as root (Linux user password)
3. Serve the key from monk over Tailscale:

   # On monk — start temporary HTTP server
   cat ~/.ssh/id_ed25519_kscloud1.pub > ~/key.txt
   python3 -m http.server 7777 --directory ~/
   

4. In Hetzner console, type:

   curl http://<MONK_TAILSCALE_IP>:7777/key.txt > /root/.ssh/authorized_keys
   

5. Enable root SSH (if needed):

   sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
   systemctl restart ssh
   

---

OSTicket SMTP

Config: smtp.gmail.com:587, STARTTLS
From: kitestacks.helpdesk@gmail.com (app password stored in DB)

To test email delivery: Admin Panel → Diagnostics → Send Test Email

---

Forgejo

Runs on monk at localhost:3006 (port 2222 for SSH git).

Generate API token for automation

docker exec -u git forgejo forgejo admin user generate-access-token \
  --username kenpat --token-name "my-token" --raw \
  --scopes "read:user,write:user,read:repository,write:repository"

---

Common Docker Operations

# View logs for a service
docker logs <container> --tail 50 -f

# Restart a service
cd ~/kitestacks-live/docker/<service> && docker compose restart

# Full stack restart
docker compose down && docker compose up -d

# Update a container image
docker compose pull && docker compose up -d

# Check all running containers
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"