kenpat/kitestacks-homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
kitestacks-homelab/docs/DEBUGGING.md

4.5 KiB
Raw Blame History

KiteStacks Homelab - Debugging & Troubleshooting

This document contains solutions and diagnostic steps for known issues that have occurred during the setup and operation of the KiteStacks homelab.

1. osTicket: New User Activation Emails Not Sending

Symptom: When a new user registers for a Help Desk account, they do not receive the activation email. Root Cause: osTicket runs in a Docker container without a local Mail Transfer Agent (MTA) like Postfix or Sendmail. By default, PHP's internal mail() function silently fails because it cannot route the email. Fix: You must configure an external SMTP server in the osTicket Admin Panel.

  1. Log into the osTicket Staff Control Panel (/scp/).
  2. Go to Emails > Emails.
  3. Select the default outbound email address (e.g., noreply@kitestacks.com).
  4. Scroll down to SMTP Settings and configure it to use a real mail provider (e.g., SendGrid, Mailgun, Amazon SES, or Gmail SMTP).
  5. Ensure Authentication Required is set to Yes.
  6. Save and send a test email.

2. Cloudflare Tunnel "Hmm. We're having trouble finding that site"

Symptom: A subdomain is correctly configured in Cloudflare Zero Trust, but visiting the site returns a Cloudflare error. Root Cause 1: The internal service is down or restarting. Check the Docker container logs. Root Cause 2: Multi-node load balancing cache. Cloudflare balances requests between monk and kscloud1. If you update a container on monk but forget to update it on kscloud1, 50% of requests will fail or show stale data. Fix: Ensure Docker containers on both hosts are perfectly mirrored or explicitly configure Cloudflare to route only to the active host for that specific subdomain.

3. Authentik "invalid_grant" or "Code does not exist"

Symptom: Logging into a service via Authentik SSO randomly fails with "invalid_grant". Root Cause: Initially, monk and kscloud1 ran separate Authentik Postgres databases. Auth codes were generated on one node and consumed on the other, failing validation. Fix: The Authentik databases are now Unified over Tailscale. monk points its Postgres and Redis connections to 100.123.254.52 (kscloud1). Do not run local Postgres for Authentik on monk. If Tailscale goes down, SSO will fail.

4. Kavita "Sign in with Authentik" Button Missing

Symptom: The Authentik OIDC login button does not appear on the Kavita login screen. Root Cause: Kavita stores OIDC settings in its internal SQLite database (kavita.db), not in an environment variable. Fix: The OIDC settings must be configured manually via the Kavita UI (Admin Settings -> OIDC). Direct SQL edits are overwritten by the Kavita container upon restart.

5. Portainer Password Reset

Symptom: Admin password is lost. Fix: Stop the Portainer container. You must use a Go container with bbolt to patch the underlying BoltDB directly, or temporarily pass the --admin-password flag to the container entrypoint to reset it.

6. Uptime Kuma "Reconnecting to server..." Loop

Symptom: The Uptime Kuma UI constantly shows "Reconnecting...". Fix: Uptime Kuma requires WebSockets. Ensure Cloudflare Tunnel does not aggressively cache the HTML/JS and that Nginx proxy timeouts are not aggressively closing the WebSocket connection.

7. Forgejo Authentication Failures (LDAP/OIDC)

Symptom: Forgejo throws a 500 or redirect error after SSO login. Fix: Ensure the ROOT_URL in Forgejo's app.ini exactly matches the public domain (https://gitforge.kitestacks.com/), and that the Authentik Application Launch URL strictly matches the OAuth redirect URI configured in Forgejo.

8. Random 502 Errors on New Subdomains (e.g., ntfy)

Symptom: Accessing a newly created subdomain (like ntfy.kitestacks.com) randomly returns a 502 Bad Gateway error from Cloudflare, even though it works internally. Root Cause: The KiteStacks architecture uses a single Cloudflare Tunnel with multiple connectors (monk and kscloud1) for active-active high availability. Cloudflare load balances traffic across all active connectors blindly. If you deploy a new service (like ntfy) only on monk, any request that Cloudflare sends to the kscloud1 connector will fail with a 502 because the container doesn't exist on that node. Fix: For a multi-connector tunnel setup, you must deploy the identical service stack on all nodes. Deploy the missing container (e.g., ntfy) to the kscloud1 replica to ensure both connectors can route the traffic successfully.