kitestacks-homelab/RUNBOOK.md

# KiteStacks Homelab — How Everything Works

**Last Updated:** 2026-06-18
**Status:** Up and running
**Owner:** kenpat

---

## The Big Picture

KiteStacks is a personal homelab — a small set of programs (called "services") that run
on two computers. One computer sits at home (called **monk**), and one rents space in
a data center in Germany (called **kscloud1**).

People on the internet can reach every website without knowing where the computers are,
because all traffic goes through **Cloudflare** — a free service that acts like a secret
post-office. Cloudflare knows the address; the rest of the world doesn't.

```
You (browser)
    │
    └─► Cloudflare (the post office)
             │
             ├─► monk  (home machine, runs most services)
             └─► kscloud1  (cloud backup machine in Germany)
```

If monk goes offline, kscloud1 keeps serving the sites — Cloudflare automatically
switches traffic over. This is called **active-active** (both doors are always open).

---

## What Runs Where

### Services on monk
| What it is | What it does | Website |
|------------|--------------|---------|
| Authentik | Login manager — handles all usernames and passwords | auth.kitestacks.com |
| Portainer | Dashboard to manage all the Docker containers | portainer.kitestacks.com |
| Forgejo | Git — stores all the code and scripts | gitforge.kitestacks.com |
| BookStack | Wiki — where all the notes and guides live | wiki.kitestacks.com |
| Grafana | Charts showing how healthy the servers are | grafana.kitestacks.com |
| Karakeep | Saves and organizes bookmarks | links.kitestacks.com |
| Kavita | Reads ebooks and manga | kavita.kitestacks.com |
| OSTicket | Help-desk ticket system | tasks.kitestacks.com |
| Open WebUI | Chat with AI models (GPT, Claude, local models) | ai.kitestacks.com |
| Uptime Kuma | Watches every service and alerts if one goes down | status.kitestacks.com |
| KiteStacks Portal | The main homepage with links to everything | www.kitestacks.com |

### Services on kscloud1 (cloud backup)
- A copy of BookStack
- A copy of the main Portal
- The login database (PostgreSQL) and session memory (Redis) that Authentik uses
- The Cloudflare connector (so the site keeps running if monk is off)

---

## Cloudflare Tunnel (the secret post office)

### Why it exists
Normal websites need a router setting called "port forwarding" and a fixed home IP address.
Cloudflare Tunnel removes both requirements — monk connects **out** to Cloudflare, and
Cloudflare forwards visitor traffic back in. Your home address is never exposed.

### How to check it's healthy
Go to Cloudflare Zero Trust → Networks → Tunnels. You should see **2 healthy connectors**
(one from monk, one from kscloud1).

### Adding a new website
In Cloudflare Zero Trust → Networks → Tunnels → your tunnel → Edit → Public Hostname:
- Subdomain: `newservice`
- Domain: `kitestacks.com`
- Service URL: `http://container-name:port`

Both monk and kscloud1 need to be running that container on the same port.

### Fix: If you see 3 connectors instead of 2
The old cloudflared system service on monk is probably running alongside the Docker one.
Run this on monk to fix it:

```bash
sudo systemctl disable --now cloudflared
```

---

## Authentik (the login manager)

### What it does
Every website on KiteStacks uses Authentik for login. Instead of each website having its
own username and password, Authentik is the one source of truth. You log in once and
all the websites trust that login. This system is called **SSO** (Single Sign-On).

### Where the database lives
Authentik's user database lives on **kscloud1** (not on monk). Both machines share it
through a private encrypted network called **Tailscale**.

### Adding a new app to SSO

1. Go to `https://auth.kitestacks.com/if/admin/`
2. **Providers** → Create → OAuth2/OpenID Provider
3. Name it after the app (e.g., `myapp`)
4. Note the Client ID and Client Secret
5. **Application** → Create → link it to the provider
6. Set up the app with:
   - Login URL (your app's OIDC issuer URL)
   - Client ID and Client Secret
   - Callback URL: `https://yourapp.kitestacks.com/auth/callback`

---

## Portainer (the container dashboard)

### What it does
Portainer is a web dashboard that shows all running Docker containers. Think of Docker
containers like small self-contained boxes — each one runs one program. Portainer lets
you start, stop, restart, and view logs for all the boxes without typing commands.

### If you get locked out
```bash
# Stop Portainer
docker stop portainer

# Reset the password (the command will print a new temporary password)
docker run --rm -v portainer_data:/data portainer/helper-reset-password

# Start it again
docker start portainer
```

### First-time OAuth login issue
When someone logs into Portainer through Authentik for the first time, they get created
as a regular user (not admin). They won't be able to see any servers. To fix this,
create their account as admin **before** their first login:

```bash
# Step 1: Get a login token
TOKEN=$(curl -sk -X POST https://portainer.kitestacks.com/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"YOUR_PASSWORD"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")

# Step 2: Create the user as admin (role 1 = admin)
curl -sk -X POST "https://portainer.kitestacks.com/api/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"username":"user@example.com","role":1}'
```

---

## BookStack (the wiki)

### What it does
BookStack is a self-hosted wiki — like a private Wikipedia just for this homelab.
All notes, runbooks, and guides live here.

### Important settings
BookStack uses Authentik for login. Two settings must be correct:

- `OIDC_ISSUER_DISCOVER=true` — tells BookStack to automatically find all login endpoints
- `OIDC_ISSUER` — must point to the per-app Authentik URL, like:
  `https://auth.kitestacks.com/application/o/bookstack/`

### Fix: If cache breaks after running a PHP command
Sometimes running admin commands inside the container breaks file permissions:

```bash
docker exec bookstack chown -R abc:users /config/www/framework/cache/
```

### Clear BookStack's config cache
```bash
docker exec bookstack php /app/www/artisan config:clear
docker exec bookstack php /app/www/artisan cache:clear
```

---

## kscloud1 (the cloud backup machine)

### SSH access
```bash
ssh -i ~/.ssh/id_ed25519_kscloud1 root@KSCLOUD1_TAILSCALE_IP
```

### If you can't SSH in (key was lost)

1. Open Hetzner Cloud console → your server → **Console** tab (this is like a TV remote for the server)
2. Log in as `root` using the Linux root password
3. On monk, share your public SSH key temporarily:
   ```bash
   cat ~/.ssh/id_ed25519_kscloud1.pub > ~/key.txt
   python3 -m http.server 7777 --directory ~/
   ```
4. In the Hetzner console, type:
   ```bash
   curl http://MONK_TAILSCALE_IP:7777/key.txt > /root/.ssh/authorized_keys
   ```
5. If root SSH is disabled:
   ```bash
   sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
   systemctl restart ssh
   ```

---

## OSTicket (help desk)

OSTicket is the ticket/task system at `tasks.kitestacks.com`.
Emails sent to `kitestacks.helpdesk@gmail.com` become tickets automatically.

To test that email is working: Admin Panel → Diagnostics → Send Test Email

---

## Forgejo (code storage)

Forgejo is the Git server — all scripts, configs, and docs live here.

### Create an API token for automation
```bash
docker exec -u git forgejo forgejo admin user generate-access-token \
  --username kenpat \
  --token-name "my-token" \
  --raw \
  --scopes "read:user,write:user,read:repository,write:repository"
```

Note: SSH to gitforge.kitestacks.com only works from inside the local network,
not through Cloudflare (Cloudflare blocks non-HTTPS ports).
For git operations from monk, use `ssh://git@localhost:2222/kenpat/repo.git`.

---

## Everyday Docker Commands

```bash
# See all running containers and their status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# View recent logs for a service
docker logs CONTAINER_NAME --tail 50 -f

# Restart a service
cd ~/kitestacks-live/docker/SERVICE_NAME
docker compose restart

# Stop and restart a service (harder reset)
docker compose down && docker compose up -d

# Pull latest image and restart
docker compose pull && docker compose up -d
```

---

## Tailscale (the private tunnel between machines)

Tailscale creates an encrypted private network between monk and kscloud1.
Nothing on this network is visible to the public internet.

Used for:
- monk connecting to kscloud1's PostgreSQL and Redis (for Authentik)
- SSH from monk to kscloud1
- Prometheus on monk scraping metrics from kscloud1

To check connection status:
```bash
tailscale status
```