kenpat/kitestacks-homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Reorganize repos into kitestacks-homelab + plain-English doc rewrite

Browse source 
- Rewrote RUNBOOK.md and DEBUG-DOCUMENTATION.md in simple 5th-grade language
  with real-world analogies for every technical concept
- Updated README.md with current service inventory and folder map
- Added cloud-migration/ subdirectory (from kitestacks-cloud-migration repo)
- Added autosync/ subdirectory (from kitestacks-homelab-autosync-test repo)
- Added osticket/ subdirectory (from OSTicketSystem repo)
- Added cloud/ placeholder for future cloud configs
- Excluded binary DB/postgres files from autosync subdirectory

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
kenpat 2026-06-18 18:37:58 -05:00
parent f79478158d
commit fb822d5142
75 changed files with 11711 additions and 338 deletions
Download patch file Download diff file Expand all files Collapse all files

359
RUNBOOK.md
View file

@ -1,221 +1,174 @@
# KiteStacks Homelab — Complete Setup Runbook
# KiteStacks Homelab — How Everything Works
**Last Updated:** 2026-06-18
**Status:** Production (monk primary, kscloud1 Hetzner cloud replica)
**Maintainer:** kenpat
**Last Updated:** 2026-06-18
**Status:** Up and running
**Owner:** kenpat
---
## Architecture Overview
## The Big Picture
KiteStacks is a personal homelab — a small set of programs (called "services") that run
on two computers. One computer sits at home (called **monk**), and one rents space in
a data center in Germany (called **kscloud1**).
People on the internet can reach every website without knowing where the computers are,
because all traffic goes through **Cloudflare** — a free service that acts like a secret
post-office. Cloudflare knows the address; the rest of the world doesn't.
```
Internet
│
└── Cloudflare (DNS + Tunnel)
│ Active-Active across 2 connectors
├── cloudflared on monk (primary home machine, Docker container)
└── cloudflared on kscloud1 (Hetzner VPS, <KSCLOUD1_PUBLIC_IP>)
Tailscale overlay network (VPN mesh):
monk <MONK_TAILSCALE_IP>
kscloud1 <KSCLOUD1_TAILSCALE_IP> ← hosts shared Authentik Postgres + Redis
You (browser)
│
└─► Cloudflare (the post office)
│
├─► monk (home machine, runs most services)
└─► kscloud1 (cloud backup machine in Germany)
```
**Public subdomains** route through the same Cloudflare Tunnel token.
Both monk and kscloud1 are connectors so the site stays up if either goes offline.
| Subdomain | Service | Port |
|-----------|---------|------|
| auth.kitestacks.com | Authentik | 9000 |
| portainer.kitestacks.com | Portainer | 9443 |
| wiki.kitestacks.com | BookStack | 6875 (monk) / 6877 (kscloud1) |
| grafana.kitestacks.com | Grafana | 3000 |
| gitforge.kitestacks.com | Forgejo | 3006 |
| links.kitestacks.com | Karakeep | 3100 |
| status.kitestacks.com | Uptime Kuma | 3001 |
| tasks.kitestacks.com | OSTicket | 8080 |
| flux.kitestacks.com | FluxCD | — |
If monk goes offline, kscloud1 keeps serving the sites — Cloudflare automatically
switches traffic over. This is called **active-active** (both doors are always open).
---
## Service Inventory
## What Runs Where
### Running on monk
authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy,
bookstack, bookstack-db, cloudflared, flux, forgejo, grafana,
karakeep, karakeep-chrome, karakeep-meilisearch, kavita,
kite-litellm, kite-openwebui, kitestacks-metrics-api, kitestacks-portal,
node-exporter, ntfy, osticket, osticket-app, osticket-db,
portainer, prometheus, uptime-kuma, blackbox-exporter
### Services on monk
| What it is | What it does | Website |
|------------|--------------|---------|
| Authentik | Login manager — handles all usernames and passwords | auth.kitestacks.com |
| Portainer | Dashboard to manage all the Docker containers | portainer.kitestacks.com |
| Forgejo | Git — stores all the code and scripts | gitforge.kitestacks.com |
| BookStack | Wiki — where all the notes and guides live | wiki.kitestacks.com |
| Grafana | Charts showing how healthy the servers are | grafana.kitestacks.com |
| Karakeep | Saves and organizes bookmarks | links.kitestacks.com |
| Kavita | Reads ebooks and manga | kavita.kitestacks.com |
| OSTicket | Help-desk ticket system | tasks.kitestacks.com |
| Open WebUI | Chat with AI models (GPT, Claude, local models) | ai.kitestacks.com |
| Uptime Kuma | Watches every service and alerts if one goes down | status.kitestacks.com |
| KiteStacks Portal | The main homepage with links to everything | www.kitestacks.com |
### Running on kscloud1 (extras)
bookstack, bookstack-db-ks, kite-monitor, osticket-app-118,
osticket-db-118, www-backup, homepage-backup, cloudflared,
authentik-postgresql, authentik-redis
### Shared infrastructure on kscloud1
- PostgreSQL `:5432` — Authentik DB used by both hosts (Tailscale only)
- Redis `:6379` — Authentik session cache (Tailscale only)
### Services on kscloud1 (cloud backup)
- A copy of BookStack
- A copy of the main Portal
- The login database (PostgreSQL) and session memory (Redis) that Authentik uses
- The Cloudflare connector (so the site keeps running if monk is off)
---
## Cloudflare Tunnel
## Cloudflare Tunnel (the secret post office)
### How it works
Both monk and kscloud1 run `cloudflared` as Docker containers using the **same tunnel token**. Cloudflare load-balances across both connectors (active-active). The tunnel token is stored in:
- monk: `~/kitestacks-live/docker/cloudflared/.env` → `TUNNEL_TOKEN`
- kscloud1: `/opt/kitestacks/docker/cloudflared/.env` → `TUNNEL_TOKEN`
### Why it exists
Normal websites need a router setting called "port forwarding" and a fixed home IP address.
Cloudflare Tunnel removes both requirements — monk connects **out** to Cloudflare, and
Cloudflare forwards visitor traffic back in. Your home address is never exposed.
### Fix: Phantom 3rd Replica
If `cloudflared tunnel info` shows 3 connectors instead of 2, the native cloudflared systemd service on monk is running alongside the Docker container.
### How to check it's healthy
Go to Cloudflare Zero Trust → Networks → Tunnels. You should see **2 healthy connectors**
(one from monk, one from kscloud1).
```bash
# Check systemd cloudflared on monk
systemctl status cloudflared
# Disable it — Docker container is the correct one
sudo systemctl disable --now cloudflared
```
### Adding a new hostname route
### Adding a new website
In Cloudflare Zero Trust → Networks → Tunnels → your tunnel → Edit → Public Hostname:
- Subdomain: `newservice`
- Domain: `kitestacks.com`
- Service: `http://container-name:port`
- Service URL: `http://container-name:port`
Both monk and kscloud1 must have the container running on the same port.
Both monk and kscloud1 need to be running that container on the same port.
---
### Fix: If you see 3 connectors instead of 2
The old cloudflared system service on monk is probably running alongside the Docker one.
Run this on monk to fix it:
## Authentik SSO
### Architecture
Authentik uses a **shared database** hosted on kscloud1. monk's Authentik containers connect via Tailscale.
- monk containers: `authentik`, `authentik-worker`, `authentik-ldap`, `authentik-ldap-proxy`
- DB: PostgreSQL on kscloud1 at `<KSCLOUD1_TAILSCALE_IP>:5432`
- Redis: kscloud1 at `<KSCLOUD1_TAILSCALE_IP>:6379`
### Adding OIDC SSO for a new app
1. In Authentik admin (`https://auth.kitestacks.com/if/admin/`):
- **Providers** → Create → OAuth2/OpenID Provider
- Name the provider after the app (e.g. `bookstack`)
- Set `issuer_mode` based on the app's requirements (see Debug doc)
- Note the Client ID and Client Secret
2. **Application** → Create → link to the provider
3. **Policy Binding** → bind the `default-authentication-flow` to the application
4. Configure the app with:
- `OIDC_ISSUER` = discovery base URL
- `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET`
- Callback URL = `https://yourapp.kitestacks.com/auth/callback`
### Checking OIDC discovery URL
```bash
# Per-provider (issuer_mode=per_provider)
curl -s https://auth.kitestacks.com/application/o/<slug>/.well-known/openid-configuration | python3 -m json.tool
# Global (issuer_mode=global)
# Note: global issuer URL does NOT serve a JSON discovery doc at /.well-known/
# Use per-provider mode for apps that auto-discover endpoints (BookStack, etc.)
```
### Changing provider issuer_mode via SQL
```bash
docker run --rm --network host \
-e PGPASSWORD="<REDACTED>" \
postgres:16 psql -h <KSCLOUD1_TAILSCALE_IP> -U authentik authentik -c \
"UPDATE authentik_providers_oauth2_oauth2provider SET issuer_mode='per_provider' WHERE provider_ptr_id=<ID>;"
sudo systemctl disable --now cloudflared
```
---
## Portainer
## Authentik (the login manager)
### OAuth setup (Authentik)
Portainer CE uses AuthenticationMethod=3 (OAuth). Configured via the BoltDB.
### What it does
Every website on KiteStacks uses Authentik for login. Instead of each website having its
own username and password, Authentik is the one source of truth. You log in once and
all the websites trust that login. This system is called **SSO** (Single Sign-On).
Key settings:
- `OAuthLoginURI`: `https://auth.kitestacks.com/application/o/authorize/`
- `OAuthTokenURI`: `https://auth.kitestacks.com/application/o/token/`
- `OAuthUserURI`: `https://auth.kitestacks.com/application/o/userinfo/`
- `OAuthClientID`: `portainer`
- `OAuthRedirectURI`: `https://portainer.kitestacks.com`
- `OAuthAutoCreateUsers`: `true`
- `OAuthDefaultTeamID`: `0`
### Where the database lives
Authentik's user database lives on **kscloud1** (not on monk). Both machines share it
through a private encrypted network called **Tailscale**.
### Pre-creating an admin user before first OAuth login
OAuth auto-created users default to Role:2 (regular user) and can't see environments.
Pre-create them as Role:1 (admin) via the API before they log in:
### Adding a new app to SSO
1. Go to `https://auth.kitestacks.com/if/admin/`
2. **Providers** → Create → OAuth2/OpenID Provider
3. Name it after the app (e.g., `myapp`)
4. Note the Client ID and Client Secret
5. **Application** → Create → link it to the provider
6. Set up the app with:
- Login URL (your app's OIDC issuer URL)
- Client ID and Client Secret
- Callback URL: `https://yourapp.kitestacks.com/auth/callback`
---
## Portainer (the container dashboard)
### What it does
Portainer is a web dashboard that shows all running Docker containers. Think of Docker
containers like small self-contained boxes — each one runs one program. Portainer lets
you start, stop, restart, and view logs for all the boxes without typing commands.
### If you get locked out
```bash
# Stop Portainer
docker stop portainer
# Reset the password (the command will print a new temporary password)
docker run --rm -v portainer_data:/data portainer/helper-reset-password
# Start it again
docker start portainer
```
### First-time OAuth login issue
When someone logs into Portainer through Authentik for the first time, they get created
as a regular user (not admin). They won't be able to see any servers. To fix this,
create their account as admin **before** their first login:
```bash
# Get auth token
# Step 1: Get a login token
TOKEN=$(curl -sk -X POST https://portainer.kitestacks.com/api/auth \
-H "Content-Type: application/json" \
-d '{"username":"admin","password":"<REDACTED>"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")
-d '{"username":"admin","password":"YOUR_PASSWORD"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")
# Create user as admin (Role:1), no password needed for OAuth users
# Step 2: Create the user as admin (role 1 = admin)
curl -sk -X POST "https://portainer.kitestacks.com/api/users" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"username":"user@example.com","role":1}'
```
### Reset admin password (if locked out)
```bash
# Stop Portainer
docker stop portainer
# Reset password (shows new temp password)
docker run --rm -v portainer_data:/data portainer/helper-reset-password
# Restart
docker start portainer
```
---
## BookStack
## BookStack (the wiki)
### Setup (both monk and kscloud1)
Location:
- monk: `~/kitestacks-live/docker/bookstack/docker-compose.yml`
- kscloud1: `/opt/kitestacks/docker/bookstack/docker-compose.yml`
### What it does
BookStack is a self-hosted wiki — like a private Wikipedia just for this homelab.
All notes, runbooks, and guides live here.
Key environment variables:
```yaml
- APP_URL=https://wiki.kitestacks.com
- DB_HOST=bookstack-db
- AUTH_METHOD=oidc
- OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/
- OIDC_ISSUER_DISCOVER=true
- OIDC_CLIENT_ID=bookstack
- OIDC_CLIENT_SECRET=<REDACTED>
- OIDC_USER_ATTRIBUTE=email
- APP_KEY=<REDACTED>
```
### Important settings
BookStack uses Authentik for login. Two settings must be correct:
### Generate APP_KEY
```bash
docker run --rm --entrypoint /bin/bash lscr.io/linuxserver/bookstack:latest appkey
```
- `OIDC_ISSUER_DISCOVER=true` — tells BookStack to automatically find all login endpoints
- `OIDC_ISSUER` — must point to the per-app Authentik URL, like:
`https://auth.kitestacks.com/application/o/bookstack/`
### OIDC Configuration
BookStack uses `OIDC_ISSUER_DISCOVER=true` to auto-discover all endpoints from Authentik.
The `OIDC_ISSUER` must match the per-app discovery URL base (not the global Authentik URL).
### Fix: If cache breaks after running a PHP command
Sometimes running admin commands inside the container breaks file permissions:
The Authentik bookstack provider must have `issuer_mode='per_provider'` so its discovery
document returns the correct per-app issuer URL. See Debug doc for full troubleshooting.
### Fix cache permissions after artisan runs
Running `php artisan` as root creates root-owned cache dirs that block the app:
```bash
docker exec bookstack chown -R abc:users /config/www/framework/cache/
```
### Clear Laravel config/cache
### Clear BookStack's config cache
```bash
docker exec bookstack php /app/www/artisan config:clear
docker exec bookstack php /app/www/artisan cache:clear
@ -223,27 +176,27 @@ docker exec bookstack php /app/www/artisan cache:clear
---
## kscloud1 Access
## kscloud1 (the cloud backup machine)
### SSH
### SSH access
```bash
ssh -i ~/.ssh/id_ed25519_kscloud1 root@<KSCLOUD1_TAILSCALE_IP>
ssh -i ~/.ssh/id_ed25519_kscloud1 root@KSCLOUD1_TAILSCALE_IP
```
### If SSH key is lost / not working
1. Open Hetzner Cloud console: `console.hetzner.cloud` → your server → Console tab
2. Log in as `root` (Linux user password)
3. Serve the key from monk over Tailscale:
### If you can't SSH in (key was lost)
1. Open Hetzner Cloud console → your server → **Console** tab (this is like a TV remote for the server)
2. Log in as `root` using the Linux root password
3. On monk, share your public SSH key temporarily:
```bash
# On monk — start temporary HTTP server
cat ~/.ssh/id_ed25519_kscloud1.pub > ~/key.txt
python3 -m http.server 7777 --directory ~/
```
4. In Hetzner console, type:
4. In the Hetzner console, type:
```bash
curl http://<MONK_TAILSCALE_IP>:7777/key.txt > /root/.ssh/authorized_keys
curl http://MONK_TAILSCALE_IP:7777/key.txt > /root/.ssh/authorized_keys
```
5. Enable root SSH (if needed):
5. If root SSH is disabled:
```bash
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart ssh
@ -251,43 +204,67 @@ ssh -i ~/.ssh/id_ed25519_kscloud1 root@<KSCLOUD1_TAILSCALE_IP>
---
## OSTicket SMTP
## OSTicket (help desk)
**Config:** smtp.gmail.com:587, STARTTLS
**From:** `kitestacks.helpdesk@gmail.com` (app password stored in DB)
OSTicket is the ticket/task system at `tasks.kitestacks.com`.
Emails sent to `kitestacks.helpdesk@gmail.com` become tickets automatically.
To test email delivery: Admin Panel → Diagnostics → Send Test Email
To test that email is working: Admin Panel → Diagnostics → Send Test Email
---
## Forgejo
## Forgejo (code storage)
Runs on monk at `localhost:3006` (port 2222 for SSH git).
Forgejo is the Git server — all scripts, configs, and docs live here.
### Generate API token for automation
### Create an API token for automation
```bash
docker exec -u git forgejo forgejo admin user generate-access-token \
--username kenpat --token-name "my-token" --raw \
--username kenpat \
--token-name "my-token" \
--raw \
--scopes "read:user,write:user,read:repository,write:repository"
```
Note: SSH to gitforge.kitestacks.com only works from inside the local network,
not through Cloudflare (Cloudflare blocks non-HTTPS ports).
For git operations from monk, use `ssh://git@localhost:2222/kenpat/repo.git`.
---
## Common Docker Operations
## Everyday Docker Commands
```bash
# View logs for a service
docker logs <container> --tail 50 -f
# See all running containers and their status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
# View recent logs for a service
docker logs CONTAINER_NAME --tail 50 -f
# Restart a service
cd ~/kitestacks-live/docker/<service> && docker compose restart
cd ~/kitestacks-live/docker/SERVICE_NAME
docker compose restart
# Full stack restart
# Stop and restart a service (harder reset)
docker compose down && docker compose up -d
# Update a container image
# Pull latest image and restart
docker compose pull && docker compose up -d
# Check all running containers
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```
---
## Tailscale (the private tunnel between machines)
Tailscale creates an encrypted private network between monk and kscloud1.
Nothing on this network is visible to the public internet.
Used for:
- monk connecting to kscloud1's PostgreSQL and Redis (for Authentik)
- SSH from monk to kscloud1
- Prometheus on monk scraping metrics from kscloud1
To check connection status:
```bash
tailscale status
```