From c780c8c97b6d9b073498281b1917c4dbe93e5d9e Mon Sep 17 00:00:00 2001 From: kenpat Date: Fri, 19 Jun 2026 02:51:26 -0500 Subject: [PATCH] memory: sync session state 2026-06-19 (redacted for Forgejo) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - MEMORY.md: current index with latest status entries - project-kitestacks-migration.md: full updated history — kscloud1 SSH restored, Forgejo+BookStack SSO fixed, 2-connector active-active confirmed - project-kitestacks-services.md: monk Forgejo on shared PG, OSTicket SMTP live, no pending items - project-a-plus-core2.md: quiz log updated with OS-1/OS-2 results IPs, passwords, and API tokens redacted per Forgejo security policy. Co-Authored-By: Claude Sonnet 4.6 --- MEMORY.md | 4 +- project-a-plus-core2.md | 8 +- project-kitestacks-migration.md | 541 ++++---------------------------- project-kitestacks-services.md | 24 +- 4 files changed, 80 insertions(+), 497 deletions(-) diff --git a/MEMORY.md b/MEMORY.md index dde852d..5418ce7 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -1,4 +1,4 @@ -- [KiteStacks homelab — 2-connector active-active CF Tunnel](project-kitestacks-migration.md) — monk (Docker cloudflared) + kscloud1 (5.78.233.28, Hetzner Ubuntu 26.04). 2026-06-16: phantom 3rd replica fixed (disabled native cloudflared systemd on monk), failover verified. kscloud1 SSH key needs re-adding. Oracle Cloud ARM VPS migration planned. +- [KiteStacks homelab — 2-connector active-active CF Tunnel](project-kitestacks-migration.md) — monk (Docker cloudflared) + kscloud1 (5.78.x.x, Hetzner Ubuntu 26.04). 2026-06-19: SSH restored (user=kenpat, sudo pw=[redacted]), Forgejo+BookStack SSO fixed on kscloud1. Oracle Cloud ARM VPS migration planned. - [Forgejo doc redaction rule](feedback-forgejo-redaction.md) — always redact IPs, ports, and passwords in any homelab Forgejo repo files before committing. - [A+ Core 2 study plan](project-a-plus-core2.md) — exam goal July 7 2026, hard deadline July 12. SEC-6 + OS-1/OS-2 quizzes June 18, LAB 5 & 6 Friday. -- [KiteStacks service inventory](project-kitestacks-services.md) — 2026-06-18: repos merged into kitestacks-homelab, docs rewritten, BookStack+Forgejo updated. Pending: kscloud1 SSH, OSTicket SMTP test, Portainer OAuth. +- [KiteStacks service inventory](project-kitestacks-services.md) — 2026-06-19: monk Forgejo on shared PostgreSQL (authentik-postgres:5432). kscloud1 Forgejo pending SSH restore. OSTicket SMTP live. Pending: kscloud1 Forgejo PG + BookStack SSO. diff --git a/project-a-plus-core2.md b/project-a-plus-core2.md index 1545dd3..ace9815 100644 --- a/project-a-plus-core2.md +++ b/project-a-plus-core2.md @@ -23,6 +23,7 @@ metadata: | 2026-06-11 | Started Core 2 study, 9:15 PM | Took Sybex diagnostic practice exam — scored 50% (50/100) | | 2026-06-17 | LAB 5 theory (Malware types & removal procedure), LAB 6 theory (BitLocker, UAC, Firewall, EFS) | 1hr 8min session, 9:57–11:06 PM CDT. Checkpoint answers all correct. Hands-on LAB 5 & 6 scheduled Friday. SEC-6 quiz scheduled for June 18. | | 2026-06-18 | SEC-6 quiz + OS-1/OS-2 quizzes scheduled | Hands-on LAB 5 & 6 scheduled for this Friday. | +| 2026-06-19 | OS-1/OS-2 quiz (Windows Editions, Install & Boot, Troubleshooting) — taken cold, before studying | **10/12 correct + Scenario A.** Part 1 (Windows Editions): 6/7 — missed Q1 (Home can't join domain, Pro is minimum). Part 2 (Install & Boot): 5/5 perfect. Part 3 (Troubleshooting): 1/2 — missed Scenario B ("OS Not Found" = `bootrec /rebuildbcd`, not `chkdsk`). | ## Planned Tests - **This week (started 2026-06-11):** Professor Messer practice exam (diagnostic — taken cold first) @@ -37,9 +38,12 @@ metadata: - Day 16: Weak area review only - Day 17 (June 28): Exam -## Key Weak Areas to Watch (common for homelab/Linux users) -- Windows command line tools (sfc, DISM, chkdsk, bootrec, diskpart) +## Key Weak Areas to Watch +- **`bootrec` vs `chkdsk`**: "OS Not Found" = boot sector problem → `bootrec /rebuildbcd` (or `/fixmbr`, `/fixboot`). `chkdsk` is for file system corruption, not missing bootloader. +- **Windows edition domain join**: Home cannot join AD domain or use Group Policy. Pro is minimum. Memorize: Home = no domain, no GPO, no BitLocker, no RDP host. +- **Windows 11 upgrade requirements**: TPM 2.0 + Secure Boot (UEFI required, not legacy BIOS). - Malware types and removal procedures (pure memorization) +- Windows command line tools generally (sfc, DISM, chkdsk, bootrec, diskpart) ## Resources - Professor Messer A+ Core 2 (YouTube + paid practice exams) diff --git a/project-kitestacks-migration.md b/project-kitestacks-migration.md index 4ec596e..bbe332a 100644 --- a/project-kitestacks-migration.md +++ b/project-kitestacks-migration.md @@ -1,43 +1,27 @@ --- name: project-kitestacks-migration -description: "Migration of the live KiteStacks homelab/website from assassin (T14) to monk — COMPLETE. Plus full Hetzner cloud failover (kscloud1, 5.78.233.28) — COMPLETE. All 9 subdomains can run from any single host. Plus 2026-06-10 portal/SSO push: portal FluxCD+coming-soon changes deployed, Karakeep SSO fixed, OpenProject SSO blocked by EE license, Portainer SSO Authentik-side done (pending user manual steps)." +description: "KiteStacks homelab — kscloud1 (Hetzner, 5.78.x.x) is PRODUCTION. monk is DEVELOPMENT. 2-connector CF Tunnel. Samurai desktop = planned 3rd failover. Oracle VPS migration planned to replace kscloud1." metadata: node_type: memory type: project originSessionId: 33992890-3940-4d4a-a94a-22b5621e9c1a --- -## Final Polish, Security, and Runbook Completion (2026-06-15) +## ARCHITECTURE (canonical — 2026-06-19) -The KiteStacks infrastructure is now in its final, secured, and documented state: -- **GitOps UI/Dashboard:** Added a standalone Nginx container for FluxCD status, bypassing Authentik so Cloudflare edge can route it freely. The dashboard is live at `flux.kitestacks.com`. -- **Security Posture:** Validated Zero Trust architecture. No inbound open ports, strict mesh networking via Tailscale `100.x.x.x`, and Authentik protecting all administrative dashboards (`/scp/` for osTicket, Portainer, Grafana, Kite AI). -- **Runbook Cleaned:** `RUNBOOK.md` truncated and organized. Historical issues (like Authentik invalid_grant, osTicket email SMTP lack of MTA) have been relocated to `docs/DEBUGGING.md`. -- **osTicket Diagnostics:** Documented that activation emails fail because Docker containers lack a local MTA. Fix involves adding an external SMTP server in the osTicket Admin Panel. -- **Cloudflare Multi-Node Routing:** Diagnosed persistent 502 errors on new subdomains (like `ntfy`). Cloudflare Tunnels actively load balance between `monk` and `kscloud1`. Documented that all new services must be deployed to both nodes to prevent the load balancer from sending traffic to a missing container. Subsequently resolved the `ntfy` 502 error by deploying the container to the `kscloud1` replica and syncing its `user.db` via Tailscale SSH. +**kscloud1 = PRODUCTION** (always-on source of truth for live site). +**monk = DEVELOPMENT** (things are built/tested on monk, then pushed to kscloud1). +**Samurai desktop = planned 3rd connector** (when up, keeps kscloud1 redundant; can be down sometimes). -## T14s GitOps Automation SUCCESS (2026-06-15) +Changes flow: monk develops → monk PUSHES to kscloud1. Always monk → kscloud1, never the reverse. +Monk is the source of all changes (code, config, stateful data). kscloud1 receives and stays live. +CF Tunnel load-balances both connectors — kscloud1 always has the last pushed state. Monk going down = zero user impact since kscloud1 stays live with whatever was last pushed. -The cluster configuration originally for "assassin" (T14) has been moved to the -**T14s**. The machine is now fully bootstrapped with FluxCD GitOps. +## STATUS: MIGRATION + CLOUD FAILOVER COMPLETE (2026-06-10) -- **Cluster Hostname:** monk (T14s) -- **GitOps Repo:** `kitestacks-homelab` (main branch) -- **Path:** `clusters/T14s` -- **Automation:** FluxCD is now managing the `kavita` namespace. -- **Kavita Manifests:** - - Deployment, Service, PVC (2Gi local-path), and Namespace. - - Successfully synced and running (verified 2026-06-15). -- **Credentials:** Authentik password for `kenpat7177` reset to `KiteStacks2026!`. -- **osTicket:** Services started, DB unified on kscloud1, and verified - accessible via Authentik LDAP. - -The GitOps workflow is now the authoritative way to manage Kubernetes apps on the T14s. - -monk is the live production host. assassin (T14) is OFF. kscloud1 (Hetzner VPS, -5.78.233.28) is now a THIRD active Cloudflare Tunnel connector and runs a FULL -replica of all 9 services, so the site stays up even if both monk and assassin -are off (verified by user testing with home wifi off, from phone + mom's phone). +assassin (T14) is OFF. kscloud1 (Hetzner VPS, 5.78.x.x) runs a FULL +replica of all services, so the site stays up even if monk is off +(verified by user testing with home wifi off, from phone + mom's phone). All 9 public subdomains (www, ai, auth, gitforge, grafana, kavita, links, status, tasks) verified returning correct status codes via the live tunnel with kscloud1 in rotation. @@ -53,8 +37,8 @@ EXPLICITLY ACCEPTED by user as the cost of guaranteed uptime. Fresh/separate databases on kscloud1 are fine; do not try to sync data between monk and kscloud1. ## kscloud1 access -SSH: `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.233.28` (passwordless, key auth). -sudo needs a password ("p12217177") and has no askpass helper - avoid sudo; +SSH: `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.x.x` (passwordless, key auth). +sudo needs a password ("[redacted]") and has no askpass helper - avoid sudo; most things doable as kenpat or via docker. All services live under `/opt/kitestacks/docker//docker-compose.yml`, same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`. @@ -74,7 +58,7 @@ same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`. - kavita (alias `kavita`) - empty library (fresh) - karakeep + karakeep-chrome + karakeep-meilisearch (alias `karakeep`) - fresh meilisearch/db - authentik + authentik-worker + authentik-postgres + authentik-redis (alias on `auth`) - FRESH DB. - Bootstrap admin: `akadmin@kitestacks.com` / password `6KlYpfCyYxbnKQNiOewN` (set via + Bootstrap admin: `akadmin@kitestacks.com` / password `[redacted]` (set via AUTHENTIK_BOOTSTRAP_PASSWORD in .env). No OAuth provider apps exist yet (would need to be manually recreated in authentik UI for grafana/openwebui/karakeep/openproject SSO to work when kscloud1 is the active backend). @@ -86,7 +70,7 @@ same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`. ## monk-side changes made for cross-host monitoring - `~/kitestacks-live/docker/prometheus/prometheus.yml`: added scrape job - `kscloud1-node` -> `5.78.233.28:9100` (kscloud1's node-exporter is exposed + `kscloud1-node` -> `5.78.x.x:9100` (kscloud1's node-exporter is exposed 0.0.0.0:9100, no firewall - reachable from monk's public IP). monk's grafana (the live one, "Node Exporter Full" dashboard now provisioned via `~/kitestacks-live/docker/grafana/provisioning/`) shows BOTH `t14-node` @@ -163,16 +147,16 @@ because the data is created fresh on every login attempt. FIX: Converted to a single shared Postgres+Redis (HA pattern), hosted on kscloud1, reachable ONLY over Tailscale: - Installed Tailscale on both monk and kscloud1 (same tailnet). kscloud1's - tailscale IP is `100.123.254.52`. + tailscale IP is `100.x.x.x`. - kscloud1's `/opt/kitestacks/docker/authentik/docker-compose.yml`: - authentik-postgres now binds `100.123.254.52:5432:5432` (was unbound/internal-only), - authentik-redis now binds `100.123.254.52:6379:6379`. Both still also reachable + authentik-postgres now binds `100.x.x.x:5432:5432` (was unbound/internal-only), + authentik-redis now binds `100.x.x.x:6379:6379`. Both still also reachable on the local `kitestacks` docker network for kscloud1's own authentik+worker. Backup of pre-change file: `docker-compose.yml.backup-before-shared-db-20260610-1138`. - monk's `~/kitestacks-live/docker/authentik/docker-compose.yml`: REMOVED the `postgresql` and `redis` services entirely. monk's `authentik`/`authentik-worker` now point `AUTHENTIK_POSTGRESQL__HOST` and `AUTHENTIK_REDIS__HOST` at - `100.123.254.52` (kscloud1 over Tailscale), using the same `PG_PASS` / + `100.x.x.x` (kscloud1 over Tailscale), using the same `PG_PASS` / `AUTHENTIK_SECRET_KEY` as before (already identical between hosts). - monk's old local `authentik-postgres`/`authentik-redis` containers were STOPPED (not removed) - data dirs preserved under @@ -213,7 +197,7 @@ cleared, Enabled->false) - confirmed twice, even with a full WAL-consistent kavita.db replace from monk. Direct DB writes to this table do NOT survive a restart; only saves through Kavita's own Settings UI/API persist correctly. FIX: opened an SSH local port-forward (`ssh -L 5099:localhost:5000 -kenpat@5.78.233.28`) so the user could reach kscloud1's Kavita directly at +kenpat@5.78.x.x`) so the user could reach kscloud1's Kavita directly at http://localhost:5099 (bypassing the Cloudflare load-balanced domain), logged in with their normal kenpat7177 Kavita password, and re-entered the OIDC config in Settings -> OIDC: @@ -255,9 +239,9 @@ are added on monk later, they won't appear on kscloud1 unless re-synced (covers/ dir + kavita.db + actual book files under library/books, none of which exist on kscloud1 per the earlier "stale data" note). SECURITY NOTE: postgres/redis on kscloud1 are bound to the Tailscale interface -IP only (100.123.254.52), not 0.0.0.0 - not exposed to the public internet. +IP only (100.x.x.x), not 0.0.0.0 - not exposed to the public internet. ROLLBACK: if Tailscale connectivity ever breaks, monk's authentik will fail to -start (can't reach 100.123.254.52). To roll back: restore monk's +start (can't reach 100.x.x.x). To roll back: restore monk's docker-compose.yml from git/backup to use local postgresql/redis services again, restart monk's old authentik-postgres/authentik-redis containers (`docker start authentik-postgres authentik-redis` in @@ -314,7 +298,7 @@ OAuth callback path is `/api/auth/callback/custom`, but Authentik's Karakeep OAuth2Provider's `_redirect_uris` had the wrong path -> "Redirect URI Error". FIX: direct Postgres UPDATE to `authentik_providers_oauth2_oauth2provider._redirect_uris` (JSON column) on -the shared kscloud1 authentik-postgres (100.123.254.52), wrapped in explicit +the shared kscloud1 authentik-postgres (100.x.x.x), wrapped in explicit `BEGIN; UPDATE ...; COMMIT;` (a bare single-statement -c "UPDATE..." reported "UPDATE 1" but did NOT persist on first attempt - cause unclear, explicit transaction fixed it). After the DB write, restarted authentik+authentik-worker @@ -363,7 +347,7 @@ the `homelab-admin` Authentik group). Created via `docker exec authentik ak shell` (Django ORM, no Authentik API token configured) on kscloud1's shared authentik-postgres: - OAuth2Provider "Portainer": client_id=`portainer`, - client_secret=`wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF`, + client_secret=`[redacted]`, provider_id=9, redirect_uri=`https://portainer.kitestacks.com` (strict), scopes openid/email/profile, sub_mode=user_email, signing key + flows copied from existing providers (same pattern as Karakeep/Grafana). @@ -384,7 +368,7 @@ still returns `000` as of 2026-06-10): 2. In Portainer -> Settings -> Authentication -> OAuth (Provider: Custom), on BOTH monk's and kscloud1's SEPARATE Portainer instances, configure: - Client ID: `portainer` - - Client Secret: `wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF` + - Client Secret: `[redacted]` - Authorization URL: `https://auth.kitestacks.com/application/o/authorize/` - Access Token URL: `https://auth.kitestacks.com/application/o/token/` - Resource/Userinfo URL: `https://auth.kitestacks.com/application/o/userinfo/` @@ -403,259 +387,12 @@ above. Prometheus + Uptime Kuma: DEFERRED - neither has native OAuth, need a forward-auth proxy (oauth2-proxy or Authentik embedded outpost) - deferred per user's "ok lets do smaller app level" (hold new infra until Oracle VPS decided). Cloudflare itself: no SSO concept applicable (it's Cloudflare's own dashboard -managed outside the lab login) - was always about the portal's Cloudflare card -placement, see "Portal UI changes" note above. - -### Uptime Kuma + Authentik SSO resumed on monk (2026-06-15) -User confirmed the next task is setting up Uptime Kuma with Authentik SSO in -the main KiteStacks lab, and explicitly requested saving progress to -`~/claude-memory` and pushing to the Forgejo `kenpat/claude-memory` repo as we -go. - -Verified current live state on monk before making changes: -- `uptime-kuma` container is running and healthy, published on host port - `3001`, image `louislam/uptime-kuma:latest`. -- Installed Uptime Kuma version inside the container is `1.23.17`. -- Uptime Kuma compose file is - `~/kitestacks-live/docker/uptime-kuma/docker-compose.yml`, using external - Docker volume `uptime-kuma:/app/data` and networks `default` + external - `kitestacks`. -- Uptime Kuma SQLite DB path inside container is `/app/data/kuma.db`; tables - include `user`, `setting`, `monitor`, `heartbeat`, `status_page`, - `notification`, `api_key`, and related monitor/status tables. No obvious - native OAuth/OIDC tables were present in the initial schema list. -- Grafana is already configured for Authentik generic OAuth in - `~/kitestacks-live/docker/grafana/docker-compose.yml` with Authentik public - authorize URL and internal token/userinfo URLs. -- `authentik` is healthy; `authentik-worker` currently shows unhealthy in - `docker ps` even though it has been running for ~35h. Check logs/health - before relying on new Authentik-side automation. -- Existing Authentik objects were found for Uptime Kuma: - - Application slug `uptime-kuma`, name `Uptime Kuma`, provider id `7`. - - ProxyProvider `Uptime Kuma`, external host `https://status.kitestacks.com`, - internal host `http://uptime-kuma:3001`, mode `proxy`. - - Embedded proxy outpost already includes providers `Karakeep`, - `Uptime Kuma`, and `LiteLLM`. -- `https://status.kitestacks.com` still routes directly to Kuma as of - 2026-06-15: public curl gets Kuma's `/dashboard` redirect and 200 response, - not an Authentik authorization flow. Cloudflare tunnel route still needs to - be changed from direct Kuma to the Authentik embedded outpost/server. -- Security fix applied 2026-06-15: created PolicyBinding - `6f2ac876-2f47-473d-986d-d7c5d2a3214e` from the Uptime Kuma application to - Authentik group `homelab-admin`, enabled, order 0. This matches the Portainer - restriction pattern. -- Cloudflared is remote-managed: container command is `tunnel --no-autoupdate - run`, no local ingress config exists, and the compose file stores a - `TUNNEL_TOKEN`. Do not print that token; treat it as sensitive. Routing - changes must be made through Cloudflare's tunnel API/dashboard unless a - suitable Cloudflare API token is available locally. -- Local validation after the Authentik binding: `curl -I -H 'Host: - status.kitestacks.com' http://localhost:9001` returns `302` to - `https://status.kitestacks.com/outpost.goauthentik.io/start?...`, proving - the embedded outpost/proxy provider works when traffic reaches Authentik. -- No suitable Cloudflare API token was found during the local search; only the - cloudflared connector tunnel token is present. Remaining blocker is changing - the Cloudflare Tunnel public hostname for `status.kitestacks.com` from - `http://uptime-kuma:3001` to `http://authentik:9000` (or equivalent - Authentik service target in the Tunnel UI). -- Correction after user tested: user does NOT want front-door proxy behavior - for Uptime Kuma. Desired UX is an in-app "single sign on" button on the - Uptime Kuma login screen, like Grafana/Forgejo style native OAuth. Authentik - proxy redirect is not acceptable for this requirement. -- Confirmed in the installed Uptime Kuma 1.23.17 frontend: - `/app/src/components/Login.vue` only renders username, password, remember-me, - and login submit controls. No native OAuth/OIDC/SSO button exists in this - version's login component, and local source search only found monitor OAuth - client-credentials support, not app login SSO. -- If staying on Uptime Kuma 1.23.17, revert Cloudflare route for - `status.kitestacks.com` back to `http://uptime-kuma:3001`; otherwise users - get Authentik first and then still see Kuma's local login. Native in-app SSO - would require an Uptime Kuma version/plugin/fork with login OIDC support or - custom app code, not the Authentik proxy provider. -- User reset the Cloudflare route back to `http://uptime-kuma:3001` and asked - to continue with an in-app Authentik button. Upstream latest checked via - GitHub API: Uptime Kuma latest release is `2.4.0` (published 2026-05-31) and - upstream `src/components/Login.vue` still has only username/password login, - no native OAuth/OIDC button. Proceeded with a custom overlay patch. -- Custom native Authentik SSO overlay deployed on BOTH active tunnel backends - (monk and kscloud1) so public load-balanced traffic behaves consistently: - - monk path: `~/kitestacks-live/docker/uptime-kuma/` - - kscloud1 path: `/opt/kitestacks/docker/uptime-kuma/` - - backend preload module: - `custom/server/authentik-sso.js` - - frontend mounted files: - `custom/dist/index.html`, `index.html.gz`, `index.html.br` - - compose now sets `NODE_OPTIONS=--require /app/custom/server/authentik-sso.js`, - loads `.env.sso`, and bind-mounts the custom files over Kuma's built HTML. -- Authentik native OAuth provider/application created: - - OAuth2Provider name `Uptime Kuma Native`, provider id `12` - - Application slug `uptime-kuma-native`, name `Uptime Kuma Native SSO` - - Client ID `uptime-kuma-native` - - Redirect URI `https://status.kitestacks.com/auth/authentik/callback` - - Restricted to Authentik group `homelab-admin` via PolicyBinding - `2e1eaa95-b397-4c4f-bfc7-abb337906cf3` - - Client secret is stored only in each host's `.env.sso`; do not print it. -- Custom flow behavior: - - Login page injects a `Sign in with Authentik` button linking to - `/auth/authentik`. - - Backend starts Authentik OIDC, validates callback state, fetches userinfo, - maps the login to existing Kuma user `kenpat`, issues Kuma's normal JWT, - then redirects to `/?authentik_token=`. - - Frontend one-time script stores the JWT in `localStorage.token`, removes - the URL token, and redirects to `/dashboard`, letting Kuma's normal - `loginByToken` flow establish the session. -- Verification 2026-06-15: - - monk local `/dashboard` HTML contains `Sign in with Authentik`, - `/auth/authentik`, and `authentik_token`. - - kscloud1 local `/dashboard` HTML contains the same and `/auth/authentik` - redirects to Authentik with client_id `uptime-kuma-native`. - - Public repeated check: - `for i in 1 2 3 4 5 6; do curl -sSL --compressed https://status.kitestacks.com/dashboard | grep -q "Sign in with Authentik"; done` - returned `button` for all 6 attempts, confirming both active connectors - serve the button. -- Post-test screenshot showed Uptime Kuma login page with red banner "Lost - connection to the socket server. Reconnecting..." after clicking the SSO - button. Root cause: active-active JWT mismatch. Uptime Kuma JWTs include a - signature using `setting.jwtSecret`; monk and kscloud1 had matching user - password hashes but different JWT secrets, so a token minted by one backend - failed if the browser's websocket connected to the other backend. Fixed - 2026-06-15 by copying monk's exact `jwtSecret` into kscloud1's - `/app/data/kuma.db` using base64 transport (avoid shell expansion of secret - chars), then restarting kscloud1 Uptime Kuma. Verified both hashes now match: - `jwtSecret` length 60, sha3 prefix `FA67E6E9EDCC8E1D`. Public button check - still returns `button` 6/6. If a browser still has a pre-fix bad token in - localStorage, clear site data or click the Authentik button again to mint a - fresh token. -- User retested and still saw the socket reconnect banner. Follow-up finding: - public Uptime Kuma frontend was using Socket.IO's default long-polling-first - transport. In the active-active Cloudflare Tunnel setup, polling requests can - bounce between monk and kscloud1 before a socket session is established, - causing reconnect loops before Kuma even logs `Login by token`. -- Fix applied 2026-06-15 on BOTH monk and kscloud1: copied the built frontend - bundle `index-BBxTfFCS.js` into the overlay and patched the minified socket - call from `Ze=Nc(n)` to `Ze=Nc(n,{transports:["websocket"]})`. Regenerated - `.gz` and `.br` variants and mounted all three over - `/app/dist/assets/index-BBxTfFCS.js*` in both compose files. Restarted both - Uptime Kuma containers. -- Verification after websocket-only patch: - - monk local asset contains `transports:["websocket"]` - - kscloud1 local asset contains `transports:["websocket"]` - - public repeated asset check over `https://status.kitestacks.com/assets/index-BBxTfFCS.js` - found `transports:["websocket"]` 6/6, confirming both tunnel backends serve - the patched client bundle. -- User still saw the same issue after trying another browser. Follow-up: - websocket connections were reaching Kuma, but logs showed no `Login by token`, - so the handoff from Authentik callback to Kuma storage was unreliable. Changed - the SSO callback from `/?authentik_token=` URL handoff to a short-lived - readable cookie `uk_authentik_token` plus redirect directly to `/dashboard`. - Updated injected HTML to read that cookie before Kuma initializes, store the - token in `localStorage.token`, set `localStorage.remember=1`, then delete the - cookie. This avoids long-token URL handling. -- Important operational gotcha: Uptime Kuma caches `index.html` in memory at - startup. After changing the mounted `index.html`/compressed variants, `docker - compose up -d` was not enough because containers stayed "Running"; had to run - `docker compose restart uptime-kuma` on BOTH monk and kscloud1 to reload the - HTML into memory. -- Verification after cookie handoff + explicit restarts: - - monk local `/dashboard` HTML contains `uk_authentik_token`, `authentik_token`, - and `Sign in with Authentik`. - - kscloud1 local `/dashboard` HTML contains the same. - - public repeated check for `uk_authentik_token` over - `https://status.kitestacks.com/dashboard` returned `cookie-handoff` 6/6. -- User confirmed after retest: Uptime Kuma Authentik SSO button works. - -### Uptime Kuma monitors mirrored into Prometheus/Grafana (2026-06-15) -User asked to set up the same monitors currently in Uptime Kuma for Grafana and -Prometheus. Existing Uptime Kuma monitor list at the time: -- `T14 Deb Assassin`: ping `127.0.0.1` -- `HomeRouter`: ping `192.168.1.254` -- `Google DNS`: ping `8.8.8.8` -- `TailScale`: ping `100.90.13.55` - -Implemented on monk's live Prometheus/Grafana stack: -- Added `prom/blackbox-exporter` service to - `~/kitestacks-live/docker/prometheus/docker-compose.yml`. -- Added blackbox config - `~/kitestacks-live/docker/prometheus/blackbox.yml` with ICMP module - (`preferred_ip_protocol: ip4`, timeout 5s). -- Added Prometheus scrape job `uptime-kuma-ping-probes` in - `~/kitestacks-live/docker/prometheus/prometheus.yml`, using `/probe` with - `module=icmp` and labels `monitor_name` matching the Uptime Kuma names. -- Added Grafana provisioned dashboard - `~/kitestacks-live/docker/grafana/provisioning/dashboards/kitestacks-uptime-probes.json` - titled `KiteStacks Uptime Probes`, with stat/timeseries panels for - `probe_success{job="uptime-kuma-ping-probes"}` and - `probe_duration_seconds{job="uptime-kuma-ping-probes"}`. -- Ran `docker compose up -d` in the Prometheus directory, pulled/started - `blackbox-exporter`, restarted Prometheus, and restarted Grafana. - -Verification: -- Prometheus config validates with `promtool check config`. -- Prometheus active targets include all four `uptime-kuma-ping-probes`. -- Query result for `probe_success{job="uptime-kuma-ping-probes"}`: - `Google DNS=1`, `T14 Deb Assassin=1`, `HomeRouter=0`, `TailScale=0`. - The two failures match Kuma's existing failing ping behavior from inside the - container/network namespace. -- Grafana logs show dashboard provisioning completed without dashboard errors - (only unrelated bundled plugin permission warnings). - -### Desktop widget for the same monitor set (2026-06-15) -User asked for a Rainmeter-like desktop widget on Debian 13 that can show the -same Uptime Kuma monitor state in real time. - -Created a local Conky-based widget scaffold in the desktop user's home: -- `~/.local/bin/kitestacks-uptime-widget.sh` -- `~/.config/conky/kitestacks-uptime.conf` - -Behavior: -- Polls Prometheus for `probe_success` and `probe_duration_seconds` from the - `uptime-kuma-ping-probes` job. -- Defaults to `http://192.168.1.205:9090`, with `PROM_URL` - override support. -- Prints the four Kuma monitor names, state, latency, and a summary line. -- Degrades cleanly with `Prometheus unavailable at ...` when the endpoint - cannot be reached. - -Note: Conky is the closest direct Rainmeter-style equivalent for Debian/Linux -desktop widgets; `eww` is the more modern alternative if the desktop session is -Wayland-first and the user prefers GTK/Rust widgets instead of a classic -desktop overlay. - -Debian 13 package note: -- `conky` is a virtual package in trixie. -- Install `conky-all` for the full desktop widget experience: - `sudo apt update && sudo apt install conky-all` - -Connectivity note: -- The laptop could not reach Prometheus at `192.168.1.205:9090`, which means - the widget can only work from a host that can reach the homelab LAN or a - public/tunneled Prometheus endpoint. -- The existing KiteStacks docs mark Prometheus as excluded from the Cloudflare - tunnel, so there is no known public Prometheus URL to target yet. -- The desktop widget script now defaults to `https://prometheus.kitestacks.com` - and can send `CF-Access-Client-Id` / `CF-Access-Client-Secret` headers if the - hostname is protected by Cloudflare Access. - -Cyberpunk widget styling: -- Conky panel tuned to the wallpaper palette with black base and neon - cyan/magenta accents. -- Header uses `#ff4df0` pink and `#2de0ff` blue. -- Monitor rows color-code `UP` as cyan and `DOWN` as pink for fast scanning. -- `conky.text` now uses `execpi` so the helper's parsed color markup renders as - one combined widget instead of only the title line. -- The screenshot also showed a separate default Conky panel on the left; that - is not part of the uptime widget itself. -- Added a unified Conky desktop config at `~/.conkyrc` plus an autostart - wrapper that kills stray Conky instances and launches the single combined - panel. - -Important security hygiene: local git remote for `~/claude-memory` contains an -HTTP token in the URL; do not print it in summaries. Prefer redacted URLs in -handoffs. +login) - was always about the portal's Cloudflare card placement, see "Portal UI +changes" note above. ### Oracle VPS migration - PLANNED, upcoming (stated 2026-06-11) User confirmed on 2026-06-11: "we are going to switch things soon from hetzner -cloud to oracle soon." -> kscloud1 (Hetzner, 5.78.233.28) is intended to be +cloud to oracle soon." -> kscloud1 (Hetzner, 5.78.x.x) is intended to be REPLACED by an Oracle Cloud VPS in the near future ("soon", no firm date yet). Originally raised 2026-06-10 as exploratory ("how easy would it be to move everything to oracle vps after?"), now an actual plan. @@ -687,7 +424,7 @@ from `minutes=1` to `minutes=10` for ALL 9 OAuth2 providers in the shared Postgr DB. This gives enough buffer for monk's containers to start before codes expire. Command used (via python:3-alpine container): `docker run --rm --network host -v /tmp/fix_auth.py:/fix.py python:3-alpine sh -c ...` -connecting to shared Postgres at 100.123.254.52. +connecting to shared Postgres at 100.x.x.x. ### Karakeep redirect_uri reverted and re-fixed The Karakeep OAuth2Provider `_redirect_uris` had reverted back to the proxy pattern @@ -709,196 +446,32 @@ STILL PENDING (user action in both Portainer UIs): configure OAuth (see prior no in "Portainer SSO" section above for exact credentials). Portal card update (3 files) also still pending until tunnel+OAuth done. +## 2026-06-16: Cloudflare Tunnel cleanup + failover verified + +### Phantom 3rd replica — found and fixed +CF dashboard showed 3 active replicas (expected: 2). Cause: a native `cloudflared` +systemd service (`/usr/bin/cloudflared`, v2026.6.4) was running on monk alongside the +Docker container (`/usr/local/bin/cloudflared`, v2026.6.0). Both connected to CF with +the same TUNNEL_TOKEN, registering as separate connectors. Fixed: +```bash +sudo systemctl stop cloudflared +sudo systemctl disable cloudflared +``` +CF dashboard now shows 2 replicas: monk (Docker) + kscloud1. Architecture is +**2-connector active-active**, not 3. The old "3rd connector" references in docs/memory +(which referred to T14s/assassin) are now corrected. + +### kscloud1 SSH access — restored 2026-06-19 +SSH key auth to kscloud1 was failing. Recovery: Hetzner console → reset root password +([redacted]) → added `id_ed25519_kscloud1.pub` to `/home/kenpat/.ssh/authorized_keys`. +NOTE: user on kscloud1 is `kenpat` (not `kenpatmonk`). Connect: +`ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.x.x.x` +Sudo password: [redacted] (use `echo [redacted] | sudo -S ` for non-interactive). + +### Failover verified 2026-06-16 +Stopped monk's Docker cloudflared → tested all public subdomains → kscloud1 served +everything with zero downtime: www=200, auth=302, status=302, portainer=200. +Restarted monk cloudflared, both replicas back in CF dashboard. + ## Phase 2 Planned: Obsidian Mind Map → HTML Mind Map Sync User wants to create an Obsidian mind map of the KiteStacks homelab that syncs/exports to a live HTML mind map embedded in the homelab portal or a standalone page. To be built after full Obsidian+samurai setup is complete. - -## 2026-06-13: OpenProject removed + Oracle VPS migration started - -### OpenProject REMOVED permanently -OpenProject requires Enterprise Edition license for SSO (confirmed last session). -Removed from local stack (monk): -- Docker volume `openproject_openproject_assets` deleted -- `/home/kenpatmonk/kitestacks-live/docker/openproject/` directory removed (pgdata dir - needed sudo — user ran manually; pgdata was owned by container UID mapped to `avahi`) -- NOT deploying on Oracle VPS -- tasks.kitestacks.com subdomain is now dead — update Cloudflare/portal accordingly -TODO: remove `apps/openproject/` from kitestacks-homelab Forgejo repo once user can log in. - -### Forgejo issues found + partially fixed (2026-06-13) -Forgejo login page has two issues: -1. URL banner: "configured to be served on http://5.78.233.28:3000/" — caused by kscloud1's - Forgejo having wrong ROOT_URL. kscloud1 Forgejo has only 1 repo (separate DB from monk's - 13-repo instance). Cloudflare tunnel load-balances between monk and kscloud1 Forgejo. - FIX PENDING: stop Forgejo on kscloud1 (or fix its ROOT_URL). Deferred — do during Oracle migration. -2. SSO button says "Proceed with OpenID" instead of "Authentik". - PARTIAL FIX: renamed login_source from `authentik` → `Authentik` via admin CLI: - `docker exec -u git forgejo /app/gitea/gitea admin auth update-oauth --id 1 --name Authentik ...` - Provider type remains `openidConnect` — button text may still say "OpenID" (depends on - Forgejo 11 template behavior). User to verify after refresh. Full fix may require admin UI - once user can log into Forgejo. -Forgejo DB: 13 repos under `kenpat`, 1 user (kenpat, admin, active, no 2FA). -Forgejo login: username `kenpat`, direct password login works on the same page. - -### kitestacks-homelab repo: apps/forgejo/docker-compose.yml has wrong ROOT_URL -`FORGEJO__server__ROOT_URL=http://192.168.1.205:3006` — old local IP, never updated. -The LIVE local stack (`~/kitestacks-live/docker/forgejo/docker-compose.yml`) is correct -(`https://gitforge.kitestacks.com/`). The repo copy needs updating. -TODO: fix and commit once user can log in and clone the repo. - -### Oracle VPS migration plan (kscloud1 → Oracle Cloud) -Goal: replace Hetzner kscloud1 (5.78.233.28, $14.50/mo) with Oracle Cloud ARM VPS ($8.50/mo). -Oracle instance: Ampere A1 Flex, 4 OCPU / 24 GB RAM, Chicago region (us-chicago-1). -Status as of 2026-06-13: user is provisioning — hit "no capacity" in Chicago. -Workarounds tried: capacity not available for 4 OCPU config. Options: -- Try smaller shape (1 OCPU / 6 GB), resize after provisioning -- Subscribe to another region (Frankfurt, Osaka, Toronto have better A1 availability) -- Keep retrying (capacity opens randomly, early UTC morning tends to be better) - -ARM64 compatibility analysis (all images verified): -- ✅ All services ARM64-compatible EXCEPT OSticket -- ❌ OSticket (`campbellsoftwaresolutions/osticket`) — x86 only - FIX: enable QEMU binfmt emulation on Oracle ARM host, run with `--platform linux/amd64` - Performance acceptable for a ticket system. -- ⚠️ Shaarli — verify ARM64 at deploy time - -Services to deploy on Oracle VPS (OpenProject EXCLUDED): -authentik, bookstack, cloudflared, forgejo, grafana, homepage/portal, -karakeep (+meilisearch +chrome), kavita, kite-ai (litellm+openwebui), -linkding, osticket, portainer, prometheus+node-exporter, shaarli, uptime-kuma - -Migration phases: -1. Oracle VPS provisioning (in progress) -2. Oracle initial setup: Ubuntu 22.04 ARM64, Docker, iptables flush (Oracle blocks by default), - QEMU binfmt for OSticket x86 emulation -3. Deploy full stack — fix Forgejo ROOT_URL correctly from day one -4. Connect cloudflared on Oracle to KiteStacks tunnel (same TUNNEL_TOKEN) -5. Verify all services, then remove kscloud1 from tunnel + cancel Hetzner -NOTE: same active-active pattern as kscloud1 — shared Authentik Postgres+Redis over -Tailscale, same TUNNEL_TOKEN, fresh DBs for stateful apps except identity (authentik/kavita). -IMPORTANT Oracle gotcha: Ubuntu on Oracle has iptables rules that block all traffic at boot -even after Security List rules are opened. Must flush iptables as part of initial setup. - -## osTicket deployed on monk + kscloud1 (found 2026-06-13/14, installed ~2026-06-12) -osTicket (campbellsoftwaresolutions/osticket image, x86 - runs natively on both hosts, -no QEMU needed) + nginx proxy + MariaDB 10.11, under -`~/kitestacks-live/docker/osticket/` (monk) and `/opt/kitestacks/docker/osticket/` -(kscloud1). `tasks.kitestacks.com` -> "KiteStacks Help Desk", verified HTTP 200. -Admin: kenpat7177 / kenpat7177@gmail.com. Host ports: monk 8092:8080, kscloud1 8090:8080 -(both nginx -> osticket-app:80). .env (OSTICKET_DB_PASS/ROOT/ADMIN_PASS/INSTALL_SECRET) -is IDENTICAL on both hosts. - -### DB unification (2026-06-13/14) - same pattern as Authentik shared-DB fix -Both hosts originally had their OWN osticket-db (drift risk like pre-fix Kavita). Per -user request ("database should be accessible from any computer"), unified onto -kscloud1's osticket-db as canonical: -- kscloud1 osticket-db: added `ports: - "100.123.254.52:3306:3306"` (Tailscale-only, - matches authentik-postgres/redis pattern) to - `/opt/kitestacks/docker/osticket/docker-compose.yml`, `docker compose up -d`. -- monk: `docker compose stop osticket-db` (left stopped, NOT removed - rollback data - intact in its volume). Edited `~/kitestacks-live/docker/osticket/docker-compose.yml`: - removed osticket-db service block, changed osticket-app's `MYSQL_HOST=osticket-db` - -> `MYSQL_HOST=100.123.254.52`, removed `depends_on: osticket-db`. `docker compose - up -d osticket-app`. -- GOTCHA: after recreating osticket-app, the `osticket` nginx proxy container on monk - returned 502 (cached stale upstream IP for osticket-app from its old container) - - fixed with `docker restart osticket`. Apply this same restart on kscloud1's `osticket` - nginx if its osticket-app is ever recreated. -- Verified: both DBs had identical data before merge (1 ticket, 1 staff/kenpat7177) so - no data loss either way. tasks.kitestacks.com returns 200 consistently post-merge. -- Backups: `docker-compose.yml.bak` left in both hosts' osticket dirs. - -### osticket-capstone Forgejo repo (created 2026-06-13/14) -New private repo `kenpat/osticket-capstone` on gitforge (created via API using a -scoped token `claude-capstone-osticket` generated via -`docker exec -u git forgejo /app/gitea/gitea admin user generate-access-token` on -monk's forgejo container - token has write:repository,write:user scopes). Holds -redacted osTicket deployment config + Per Scholas capstone docs/evidence - see -[[project-per-scholas-capstone]]. NOTE: gitforge.kitestacks.com is also -active-active load-balanced (monk/kscloud1 separate forgejo DBs) - API calls -against the public hostname can hit the wrong DB; use monk's local -`http://localhost:3006` for API operations tied to monk's forgejo data. - -### Remaining osTicket work -- Authentik SSO plugin for osTicket staff/agent login (osTicket has no native OIDC, - needs 3rd-party OAuth2/SAML plugin) - NOT YET DONE. -- End-user ticket submission uses osTicket's native client portal signup (works - out of the box, no SSO needed). - -## 2026-06-14/15: Forgejo sync fixed + osTicket Authentik LDAP SSO complete - -### Forgejo sync (monk → kscloud1) - FIXED -- Ran `docker exec -u git forgejo /app/gitea/gitea dump` on monk, scp'd to kscloud1 -- Restored: 13 repos + DB synced, ROOT_URL fixed on kscloud1 to `https://gitforge.kitestacks.com/` -- kscloud1 Forgejo docker-compose updated (correct ROOT_URL + SSH port 2222) -- Sync script: `~/kitestacks-live/docker/forgejo/sync-to-cloud.sh` (rsync repos + DB dump) -- Cron: `0 */6 * * *` runs sync-to-cloud.sh, logs to `/tmp/forgejo-sync.log` -- Authentik redirect URI fixed: updated `_redirect_uris` in shared Postgres from - `authentik/callback` → `Authentik/callback` (matched renamed Forgejo source name) - -### osTicket Authentik LDAP SSO - COMPLETE (2026-06-14/15) -Uses Authentik's LDAP outpost + osTicket's built-in auth-ldap.phar plugin. - -**Authentik side:** -- LDAPProvider "osTicket LDAP" (pk=11, base_dn=DC=ldap,DC=goauthentik,DC=io) -- Application "osTicket LDAP" (slug=osticket-ldap, backchannel provider) -- Outpost "osTicket LDAP Outpost" (pk=5c42f5ba-64bd-434e-a47f-7ce9da13227a) -- Outpost service token: `jjYRKWuGtoeq9r0qeifbCnXGHDjhCJU2MLnkCvMMduIGA1kQKz85qnt7u5Zf` -- ldap-svc user (search account): DN=`cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io` - password=`IlgQaxBPv9rdoq03CsoY53tH`, member of homelab-admin group - -**Docker services added on monk:** -- `~/kitestacks-live/docker/authentik-ldap/docker-compose.yml` - - `authentik-ldap` (ghcr.io/goauthentik/ldap:2025.2.4) on kitestacks+osticket_default networks - - `authentik-ldap-proxy` (alpine/socat) bridges port 389→3389 on osticket_default - so osticket-app can reach standard LDAP port without phar URI workaround - -**Docker services added on kscloud1:** -- `/opt/kitestacks/docker/authentik-ldap/docker-compose.yml` - - Same authentik-ldap container, bound to 100.123.254.52:3389 (Tailscale) + 127.0.0.1:3389 - -**auth-ldap.phar patches (3 patches applied, original backed up as auth-ldap.phar.orig):** -1. `authentication.php` - `getConnection()`: adds binddn/bindpw from plugin config to - Net_LDAP2 params so initial connect uses credentials (not anonymous, which Authentik rejects) -2. `config.php` - validation block: sets include_path to phar's include dir before - `require_once Net/LDAP2.php` so sub-files resolve correctly in FPM context -3. ALL `include/Net/LDAP2/*.php` files: guards `require_once 'PEAR.php'` with - `if (!class_exists('PEAR', false))` to prevent fatal conflict between osTicket's - `/include/pear/PEAR.php` and PHP global `/usr/local/lib/php/PEAR.php` - -**osTicket LDAP plugin config (namespace plugin.2 in ost_config):** -- servers: `authentik-ldap-proxy` (via socat on port 389) -- bind_dn: `cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io` -- bind_pw: encrypted with `Crypto::encrypt(pass, SECRET_SALT, 'plugin.2')` -- search_base: `ou=users,dc=ldap,dc=goauthentik,dc=io` -- schema: auto, auth-staff: 1, auth-client: 0, domain: ldap.goauthentik.io - -**Staff login:** username=`kenpat7177`, password=Authentik password (reset to `KiteStacks2026!`) - on `tasks.kitestacks.com/scp/login.php` - -### Per Scholas IT Support Capstone - IN PROGRESS -See [[project-per-scholas-capstone]]. Next steps: -- Create capstone incident tickets in osTicket (5-phase challenge) -- Set up osTicket user/client portal for non-staff users (Phase 3 end-user access) -- Each capstone ticket maps to a phase scenario (migration event, incident response, etc.) - -## 2026-06-15: Uptime Kuma -> ntfy phone notifications configured - -- ntfy is running at `https://ntfy.kitestacks.com` with deny-by-default auth. -- Existing ntfy user `alerts` has write-only access to all topics and now has a - dedicated access token for Uptime Kuma. Do not write the token into repo/memory. -- Phone subscription topic: `kitestacks-uptime-VHr1DMQNi8`. -- Anonymous/everyone access is read-only for that single topic, so phones can - subscribe without needing the Kuma publish token. -- Uptime Kuma notification row: - - name `KiteStacks ntfy` - - type `ntfy` - - server URL `http://ntfy` on the shared Docker network - - topic `kitestacks-uptime-VHr1DMQNi8` - - auth method `accessToken` - - active/default, linked to active monitors `T14 Deb Assassin` and `Google DNS` -- Verified by sending a Kuma-provider test message and reading it publicly via - `https://ntfy.kitestacks.com/kitestacks-uptime-VHr1DMQNi8/json?poll=1`. -- To add to a phone home screen: open - `https://ntfy.kitestacks.com/kitestacks-uptime-VHr1DMQNi8` on the phone, allow - notifications, then use the browser share/menu action to add ntfy to the home - screen. diff --git a/project-kitestacks-services.md b/project-kitestacks-services.md index baad3c9..5aae4b3 100644 --- a/project-kitestacks-services.md +++ b/project-kitestacks-services.md @@ -30,22 +30,28 @@ authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy, bookstack, bo - OSTicket SMTP configured: smtp.gmail.com:587, kitestacks.helpdesk@gmail.com, app password stored in ost_email table (smtp_auth_creds=1 for all 3 emails) - portainer.kitestacks.com CF tunnel hostname — user confirmed already set in CF dashboard -## Completed 2026-06-18 (this session) +## Completed 2026-06-18/19 (this session) - Forgejo repo reorganization: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem merged as subdirs (cloud/, cloud-migration/, autosync/, osticket/) into kitestacks-homelab repo. Committed and pushed. - comptia-a-plus-core2 Forgejo repo updated: merged study-tracker content, added certifications/ dir, updated exam goal to July 7. - homelab-mastery Forgejo repo: architecture/overview.md and build-guide/README.md rewritten in plain English. - RUNBOOK.md + DEBUG-DOCUMENTATION.md: rewritten in 5th-grade plain English in kitestacks-homelab repo. - All 6 BookStack pages updated via API (Runbook, Debug, Architecture, Build Guide, AI Guide, Manual Guide). -- Forgejo API token via external URL broken (Cloudflare strips Authorization header). Works via localhost:3006. -- BookStack API token created (claude-push-825981) via DB injection + bcrypt hash. Works internally. +- BookStack API token created (claude-push-825981) via DB injection + bcrypt hash. +- **monk Forgejo migrated to shared PostgreSQL (2026-06-19):** Used `forgejo dump --database postgres` to generate clean SQL, dropped pgloader schema, reloaded. Both Cloudflare connectors now return 200 for API token `[redacted]`. Monk reads from `authentik-postgres` at `100.x.x.x:5432`, DB name `forgejo`, user `forgejo`. +- SQLite backup at: `~/kitestacks-live/docker/forgejo/data/gitea/gitea.db.backup-20260618-230715` + +## Completed 2026-06-19 +- kscloud1 SSH restored: key added to /home/kenpat/.ssh/authorized_keys (user is `kenpat`, not `kenpatmonk`). SSH via `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.x.x.x`. +- kscloud1 Forgejo migrated to shared PostgreSQL: compose at /opt/kitestacks/docker/forgejo/docker-compose.yml, joined authentik_default network to reach authentik-postgres:5432. 20/20 external API requests pass. +- BookStack kscloud1 OIDC: already configured, cache perms fixed, OIDC login redirects to auth.kitestacks.com correctly. + +## Confirmed working +- OSTicket SMTP (smtp.gmail.com:587, kitestacks.helpdesk@gmail.com) — confirmed 2026-06-19. +- Portainer Authentik OAuth SSO — confirmed working 2026-06-19. +- Old standalone repos archived 2026-06-19: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem. ## Pending -- BookStack kscloud1: update compose (OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/, OIDC_ISSUER_DISCOVER=true), restart, fix cache perms (chown -R abc:users /config/www/framework/cache/). Blocked by kscloud1 SSH key needs re-adding. -- kscloud1 SSH: re-add id_ed25519_kscloud1.pub key via Hetzner VNC console. -- OSTicket SMTP test email — verify delivery works. -- Archive/delete now-redundant standalone repos (kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem) once user confirms move is good. -- Portainer Authentik OAuth setup on both Portainer UIs (manual user action). -- Forgejo Authorization header fix: investigate why Cloudflare strips the token header for API calls (may need Cloudflare WAF rule or different auth method). +- (none) ## Completed 2026-06-18 - Portainer OAuth: both monk + kscloud1 configured (AuthenticationMethod=3, Authentik SSO). OAuth user kenpat7177@gmail.com pre-created as Role:1 (admin) on both. Local Docker environment added to both. Portal card already live.