kitestacks-homelab/RUNBOOK.md

KiteStacks Homelab — How Everything Works

Last Updated: 2026-06-18 Status: Up and running Owner: kenpat

---

The Big Picture

KiteStacks is a personal homelab — a small set of programs (called "services") that run on two computers. One computer sits at home (called monk), and one rents space in a data center in Germany (called kscloud1).

People on the internet can reach every website without knowing where the computers are, because all traffic goes through Cloudflare — a free service that acts like a secret post-office. Cloudflare knows the address; the rest of the world doesn't.

You (browser)
    │
    └─► Cloudflare (the post office)
             │
             ├─► monk  (home machine, runs most services)
             └─► kscloud1  (cloud backup machine in Germany)

If monk goes offline, kscloud1 keeps serving the sites — Cloudflare automatically switches traffic over. This is called active-active (both doors are always open).

---

What Runs Where

Services on monk

What it is	What it does	Website
Authentik	Login manager — handles all usernames and passwords	auth.kitestacks.com
Portainer	Dashboard to manage all the Docker containers	portainer.kitestacks.com
Forgejo	Git — stores all the code and scripts	gitforge.kitestacks.com
BookStack	Wiki — where all the notes and guides live	wiki.kitestacks.com
Grafana	Charts showing how healthy the servers are	grafana.kitestacks.com
Karakeep	Saves and organizes bookmarks	links.kitestacks.com
Kavita	Reads ebooks and manga	kavita.kitestacks.com
OSTicket	Help-desk ticket system	tasks.kitestacks.com
Open WebUI	Chat with AI models (GPT, Claude, local models)	ai.kitestacks.com
Uptime Kuma	Watches every service and alerts if one goes down	status.kitestacks.com
KiteStacks Portal	The main homepage with links to everything	www.kitestacks.com

Services on kscloud1 (cloud backup)

• A copy of BookStack
• A copy of the main Portal
• The login database (PostgreSQL) and session memory (Redis) that Authentik uses
• The Cloudflare connector (so the site keeps running if monk is off)

---

Cloudflare Tunnel (the secret post office)

Why it exists

Normal websites need a router setting called "port forwarding" and a fixed home IP address. Cloudflare Tunnel removes both requirements — monk connects out to Cloudflare, and Cloudflare forwards visitor traffic back in. Your home address is never exposed.

How to check it's healthy

Go to Cloudflare Zero Trust → Networks → Tunnels. You should see 2 healthy connectors (one from monk, one from kscloud1).

Adding a new website

In Cloudflare Zero Trust → Networks → Tunnels → your tunnel → Edit → Public Hostname:

• Subdomain: newservice
• Domain: kitestacks.com
• Service URL: http://container-name:port

Both monk and kscloud1 need to be running that container on the same port.

Fix: If you see 3 connectors instead of 2

The old cloudflared system service on monk is probably running alongside the Docker one. Run this on monk to fix it:

sudo systemctl disable --now cloudflared

---

Authentik (the login manager)

What it does

Every website on KiteStacks uses Authentik for login. Instead of each website having its own username and password, Authentik is the one source of truth. You log in once and all the websites trust that login. This system is called SSO (Single Sign-On).

Where the database lives

Authentik's user database lives on kscloud1 (not on monk). Both machines share it through a private encrypted network called Tailscale.

Adding a new app to SSO

1. Go to https://auth.kitestacks.com/if/admin/
2. Providers → Create → OAuth2/OpenID Provider
3. Name it after the app (e.g., myapp)
4. Note the Client ID and Client Secret
5. Application → Create → link it to the provider
6. Set up the app with:
   • Login URL (your app's OIDC issuer URL)
   • Client ID and Client Secret
   • Callback URL: https://yourapp.kitestacks.com/auth/callback

---

Portainer (the container dashboard)

What it does

Portainer is a web dashboard that shows all running Docker containers. Think of Docker containers like small self-contained boxes — each one runs one program. Portainer lets you start, stop, restart, and view logs for all the boxes without typing commands.

If you get locked out

# Stop Portainer
docker stop portainer

# Reset the password (the command will print a new temporary password)
docker run --rm -v portainer_data:/data portainer/helper-reset-password

# Start it again
docker start portainer

First-time OAuth login issue

When someone logs into Portainer through Authentik for the first time, they get created as a regular user (not admin). They won't be able to see any servers. To fix this, create their account as admin before their first login:

# Step 1: Get a login token
TOKEN=$(curl -sk -X POST https://portainer.kitestacks.com/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"YOUR_PASSWORD"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")

# Step 2: Create the user as admin (role 1 = admin)
curl -sk -X POST "https://portainer.kitestacks.com/api/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"username":"user@example.com","role":1}'

---

BookStack (the wiki)

What it does

BookStack is a self-hosted wiki — like a private Wikipedia just for this homelab. All notes, runbooks, and guides live here.

Important settings

BookStack uses Authentik for login. Two settings must be correct:

• OIDC_ISSUER_DISCOVER=true — tells BookStack to automatically find all login endpoints
• OIDC_ISSUER — must point to the per-app Authentik URL, like: https://auth.kitestacks.com/application/o/bookstack/

Fix: If cache breaks after running a PHP command

Sometimes running admin commands inside the container breaks file permissions:

docker exec bookstack chown -R abc:users /config/www/framework/cache/

Clear BookStack's config cache

docker exec bookstack php /app/www/artisan config:clear
docker exec bookstack php /app/www/artisan cache:clear

---

kscloud1 (the cloud backup machine)

SSH access

ssh -i ~/.ssh/id_ed25519_kscloud1 root@KSCLOUD1_TAILSCALE_IP

If you can't SSH in (key was lost)

1. Open Hetzner Cloud console → your server → Console tab (this is like a TV remote for the server)
2. Log in as root using the Linux root password
3. On monk, share your public SSH key temporarily:

   cat ~/.ssh/id_ed25519_kscloud1.pub > ~/key.txt
   python3 -m http.server 7777 --directory ~/
   

4. In the Hetzner console, type:

   curl http://MONK_TAILSCALE_IP:7777/key.txt > /root/.ssh/authorized_keys
   

5. If root SSH is disabled:

   sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
   systemctl restart ssh
   

---

OSTicket (help desk)

OSTicket is the ticket/task system at tasks.kitestacks.com. Emails sent to kitestacks.helpdesk@gmail.com become tickets automatically.

To test that email is working: Admin Panel → Diagnostics → Send Test Email

---

Forgejo (code storage)

Forgejo is the Git server — all scripts, configs, and docs live here.

Create an API token for automation

docker exec -u git forgejo forgejo admin user generate-access-token \
  --username kenpat \
  --token-name "my-token" \
  --raw \
  --scopes "read:user,write:user,read:repository,write:repository"

Note: SSH to gitforge.kitestacks.com only works from inside the local network, not through Cloudflare (Cloudflare blocks non-HTTPS ports). For git operations from monk, use ssh://git@localhost:2222/kenpat/repo.git.

---

Everyday Docker Commands

# See all running containers and their status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# View recent logs for a service
docker logs CONTAINER_NAME --tail 50 -f

# Restart a service
cd ~/kitestacks-live/docker/SERVICE_NAME
docker compose restart

# Stop and restart a service (harder reset)
docker compose down && docker compose up -d

# Pull latest image and restart
docker compose pull && docker compose up -d

---

Tailscale (the private tunnel between machines)

Tailscale creates an encrypted private network between monk and kscloud1. Nothing on this network is visible to the public internet.

Used for:

• monk connecting to kscloud1's PostgreSQL and Redis (for Authentik)
• SSH from monk to kscloud1
• Prometheus on monk scraping metrics from kscloud1

To check connection status:

tailscale status