kitestacks-homelab/DEBUG-DOCUMENTATION.md

# KiteStacks Homelab — Problems We've Seen and How We Fixed Them

Newest problems at the top.

---

## 2026-06-18 — Can't SSH into kscloud1

**What happened:** Trying to connect to the cloud machine (kscloud1) gave a
"Permission denied" error. The SSH key was missing from the machine.

**How we found it:** The error message said `publickey,password` — meaning it tried
the SSH key first and then tried a password, both failed.

**How we fixed it:**
1. Used Hetzner's browser console (like a TV remote for the server) to log in as root
2. Served the SSH public key from monk as a temporary download:
   ```bash
   # On monk — share the key file over a mini web server
   cat ~/.ssh/id_ed25519_kscloud1.pub > ~/key.txt
   python3 -m http.server 7777 --directory ~/
   ```
3. Downloaded it from the Hetzner console:
   ```bash
   curl http://MONK_TAILSCALE_IP:7777/key.txt > /root/.ssh/authorized_keys
   ```
4. If the machine had root SSH login disabled:
   ```bash
   sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
   systemctl restart ssh
   ```

**Why this works:** The Hetzner console bypasses SSH entirely — it's like plugging a
keyboard and monitor directly into the server. So even when SSH is broken, you can still
type commands.

---

## 2026-06-18 — BookStack Login Said "An Error Occurred"

**What happened:** Clicking "Login with Authentik" on the wiki showed a generic error.
No details, no clues — just "An unknown error occurred."

**Why it happened (three problems at once):**

**Problem 1 — Missing setting in BookStack**
BookStack needs `OIDC_ISSUER_DISCOVER=true` to automatically find all the login
endpoints from Authentik. Without it, BookStack can't verify login tokens.

**Problem 2 — Authentik was using the wrong login URL format**
Authentik can either use one shared URL for all apps or a unique URL per app.
BookStack expects a per-app URL. When the wrong type was set, BookStack tried to
download login instructions from a URL that returned an HTML page instead of data,
and then crashed trying to read it.

**Problem 3 — File permission error hidden by BookStack**
Running a setup command inside the BookStack container as root created some folders
that only root could write to. When the normal BookStack process tried to save
a login session, it couldn't — and BookStack showed a generic error instead of
the real one.

**How we fixed it:**

Step 1 — Change Authentik to use per-app URLs (run this once):
```bash
docker run --rm --network host \
  -e PGPASSWORD="YOUR_DB_PASSWORD" \
  postgres:16 psql -h KSCLOUD1_TAILSCALE_IP -U authentik authentik -c \
  "UPDATE authentik_providers_oauth2_oauth2provider SET issuer_mode='per_provider' WHERE provider_ptr_id=PROVIDER_ID;"
```

Step 2 — Make sure BookStack's settings include:
```
OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/
OIDC_ISSUER_DISCOVER=true
```

Step 3 — Fix the file permission problem:
```bash
docker exec bookstack chown -R abc:users /config/www/framework/cache/
```

Step 4 — Restart BookStack:
```bash
docker compose up -d
```

---

## 2026-06-18 — Portainer OAuth Login Couldn't See Any Servers

**What happened:** Logged in through Authentik, got into Portainer, but no environments
(no servers, nothing to manage) were visible.

**Why it happened:** Portainer creates new SSO users as "regular users." Regular users
can't see environments — only admins can. The fix is to create the user as an admin
**before** they log in for the first time.

**How we fixed it:**

Create the user as admin before first login:
```bash
# Get a temporary auth token
TOKEN=$(curl -sk -X POST https://portainer.kitestacks.com/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"YOUR_PASSWORD"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])")

# Create the user with admin role (role 1 = admin)
curl -sk -X POST "https://portainer.kitestacks.com/api/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"username":"user@example.com","role":1}'
```

If they already logged in as a regular user, promote them:
```bash
curl -sk -X PUT "https://portainer.kitestacks.com/api/users/USER_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"role":1}'
```

---

## 2026-06-17 — Three Cloudflare Connectors Instead of Two

**What happened:** The Cloudflare dashboard was showing 3 tunnel connectors when there
should only be 2 (one from monk, one from kscloud1). This caused Authentik logins to
fail randomly — about half the time, the code from the login form would reach the wrong
connector and get rejected.

**Why it happened:** The system's built-in cloudflared service was still running on monk,
alongside the Docker container version. So monk was connecting to Cloudflare twice.

**How we fixed it:**
```bash
sudo systemctl disable --now cloudflared
```

That stopped the duplicate. Now only the Docker container runs.

After fixing: verified only 2 connectors in Cloudflare Zero Trust → Networks → Tunnels.

---

## 2026-06-17 — BookStack Database Kept Crashing

**What happened:** The BookStack database container (bookstack-db) kept restarting
and never stayed running. Logs showed: `Table 'mysql.db' doesn't exist`

**Why it happened:** The database's data folder had leftover files from a previous
incomplete setup. When MariaDB started, it saw partial old data and crashed trying
to use it.

**How we fixed it:**
```bash
# Wipe the broken database files (they're owned by root inside the container)
docker run --rm -v $(pwd)/db:/db alpine sh -c 'rm -rf /db/*'

# Start fresh
docker compose up -d
```

---

## 2026-06-17 — BookStack Said It Couldn't Find the Database

**What happened:** BookStack started but immediately errored saying it couldn't connect
to the database (bookstack-db).

**Why it happened:** BookStack was too fast. It started before the database was fully
ready, and when it tried to find `bookstack-db` on the internal network, Docker hadn't
finished registering it yet.

**How we fixed it:**
```bash
# Just wait a few seconds and restart BookStack
docker restart bookstack
```

That's it — the database had finished starting up by then.

---

## Quick Diagnostic Commands

```bash
# See which containers are running (and which are crashing)
docker ps --format "table {{.Names}}\t{{.Status}}"

# Follow the live logs of any service
docker logs CONTAINER_NAME --tail 50 -f

# Read BookStack's PHP error log
docker exec bookstack cat /app/www/storage/logs/laravel.log | tail -50

# Test if BookStack's login redirect works
curl -sc /tmp/c.txt http://localhost:6875/login -o /tmp/l.html && \
  CSRF=$(grep -oP 'name="_token" value="\K[^"]+' /tmp/l.html | head -1) && \
  curl -v -b /tmp/c.txt -X POST http://localhost:6875/oidc/login \
  -d "_token=$CSRF" --max-redirs 0 2>&1 | grep -E "HTTP|Location"
# Should show: Location: https://auth.kitestacks.com/application/o/authorize/?...

# Check Tailscale connections between machines
tailscale status

# See if both Cloudflare connectors are working
docker exec cloudflared cloudflared tunnel info TUNNEL_ID
```