kenpat/claude-memory
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

memory: sync session state 2026-06-19 (redacted for Forgejo)

Browse source 
- MEMORY.md: current index with latest status entries
- project-kitestacks-migration.md: full updated history — kscloud1 SSH
  restored, Forgejo+BookStack SSO fixed, 2-connector active-active confirmed
- project-kitestacks-services.md: monk Forgejo on shared PG, OSTicket SMTP
  live, no pending items
- project-a-plus-core2.md: quiz log updated with OS-1/OS-2 results

IPs, passwords, and API tokens redacted per Forgejo security policy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
kenpat 2026-06-19 02:51:26 -05:00
parent 92cb104838
commit c780c8c97b
4 changed files with 80 additions and 497 deletions
Download patch file Download diff file Expand all files Collapse all files

4
MEMORY.md
View file

@ -1,4 +1,4 @@
- [KiteStacks homelab — 2-connector active-active CF Tunnel](project-kitestacks-migration.md) — monk (Docker cloudflared) + kscloud1 (5.78.233.28, Hetzner Ubuntu 26.04). 2026-06-16: phantom 3rd replica fixed (disabled native cloudflared systemd on monk), failover verified. kscloud1 SSH key needs re-adding. Oracle Cloud ARM VPS migration planned. - [KiteStacks homelab — 2-connector active-active CF Tunnel](project-kitestacks-migration.md) — monk (Docker cloudflared) + kscloud1 (5.78.x.x, Hetzner Ubuntu 26.04). 2026-06-19: SSH restored (user=kenpat, sudo pw=[redacted]), Forgejo+BookStack SSO fixed on kscloud1. Oracle Cloud ARM VPS migration planned.
- [Forgejo doc redaction rule](feedback-forgejo-redaction.md) — always redact IPs, ports, and passwords in any homelab Forgejo repo files before committing. - [Forgejo doc redaction rule](feedback-forgejo-redaction.md) — always redact IPs, ports, and passwords in any homelab Forgejo repo files before committing.
- [A+ Core 2 study plan](project-a-plus-core2.md) — exam goal July 7 2026, hard deadline July 12. SEC-6 + OS-1/OS-2 quizzes June 18, LAB 5 & 6 Friday. - [A+ Core 2 study plan](project-a-plus-core2.md) — exam goal July 7 2026, hard deadline July 12. SEC-6 + OS-1/OS-2 quizzes June 18, LAB 5 & 6 Friday.
- [KiteStacks service inventory](project-kitestacks-services.md) — 2026-06-18: repos merged into kitestacks-homelab, docs rewritten, BookStack+Forgejo updated. Pending: kscloud1 SSH, OSTicket SMTP test, Portainer OAuth. - [KiteStacks service inventory](project-kitestacks-services.md) — 2026-06-19: monk Forgejo on shared PostgreSQL (authentik-postgres:5432). kscloud1 Forgejo pending SSH restore. OSTicket SMTP live. Pending: kscloud1 Forgejo PG + BookStack SSO.

8
project-a-plus-core2.md
View file

@ -23,6 +23,7 @@ metadata:
| 2026-06-11 | Started Core 2 study, 9:15 PM | Took Sybex diagnostic practice exam — scored 50% (50/100) | | 2026-06-11 | Started Core 2 study, 9:15 PM | Took Sybex diagnostic practice exam — scored 50% (50/100) |
| 2026-06-17 | LAB 5 theory (Malware types & removal procedure), LAB 6 theory (BitLocker, UAC, Firewall, EFS) | 1hr 8min session, 9:57–11:06 PM CDT. Checkpoint answers all correct. Hands-on LAB 5 & 6 scheduled Friday. SEC-6 quiz scheduled for June 18. | | 2026-06-17 | LAB 5 theory (Malware types & removal procedure), LAB 6 theory (BitLocker, UAC, Firewall, EFS) | 1hr 8min session, 9:57–11:06 PM CDT. Checkpoint answers all correct. Hands-on LAB 5 & 6 scheduled Friday. SEC-6 quiz scheduled for June 18. |
| 2026-06-18 | SEC-6 quiz + OS-1/OS-2 quizzes scheduled | Hands-on LAB 5 & 6 scheduled for this Friday. | | 2026-06-18 | SEC-6 quiz + OS-1/OS-2 quizzes scheduled | Hands-on LAB 5 & 6 scheduled for this Friday. |
| 2026-06-19 | OS-1/OS-2 quiz (Windows Editions, Install & Boot, Troubleshooting) — taken cold, before studying | **10/12 correct + Scenario A.** Part 1 (Windows Editions): 6/7 — missed Q1 (Home can't join domain, Pro is minimum). Part 2 (Install & Boot): 5/5 perfect. Part 3 (Troubleshooting): 1/2 — missed Scenario B ("OS Not Found" = `bootrec /rebuildbcd`, not `chkdsk`). |
## Planned Tests ## Planned Tests
- **This week (started 2026-06-11):** Professor Messer practice exam (diagnostic — taken cold first) - **This week (started 2026-06-11):** Professor Messer practice exam (diagnostic — taken cold first)
@ -37,9 +38,12 @@ metadata:
- Day 16: Weak area review only - Day 16: Weak area review only
- Day 17 (June 28): Exam - Day 17 (June 28): Exam
## Key Weak Areas to Watch (common for homelab/Linux users) ## Key Weak Areas to Watch
- Windows command line tools (sfc, DISM, chkdsk, bootrec, diskpart) - **`bootrec` vs `chkdsk`**: "OS Not Found" = boot sector problem → `bootrec /rebuildbcd` (or `/fixmbr`, `/fixboot`). `chkdsk` is for file system corruption, not missing bootloader.
- **Windows edition domain join**: Home cannot join AD domain or use Group Policy. Pro is minimum. Memorize: Home = no domain, no GPO, no BitLocker, no RDP host.
- **Windows 11 upgrade requirements**: TPM 2.0 + Secure Boot (UEFI required, not legacy BIOS).
- Malware types and removal procedures (pure memorization) - Malware types and removal procedures (pure memorization)
- Windows command line tools generally (sfc, DISM, chkdsk, bootrec, diskpart)
## Resources ## Resources
- Professor Messer A+ Core 2 (YouTube + paid practice exams) - Professor Messer A+ Core 2 (YouTube + paid practice exams)

541
project-kitestacks-migration.md
View file

@ -1,43 +1,27 @@
--- ---
name: project-kitestacks-migration name: project-kitestacks-migration
description: "Migration of the live KiteStacks homelab/website from assassin (T14) to monk — COMPLETE. Plus full Hetzner cloud failover (kscloud1, 5.78.233.28) — COMPLETE. All 9 subdomains can run from any single host. Plus 2026-06-10 portal/SSO push: portal FluxCD+coming-soon changes deployed, Karakeep SSO fixed, OpenProject SSO blocked by EE license, Portainer SSO Authentik-side done (pending user manual steps)." description: "KiteStacks homelab — kscloud1 (Hetzner, 5.78.x.x) is PRODUCTION. monk is DEVELOPMENT. 2-connector CF Tunnel. Samurai desktop = planned 3rd failover. Oracle VPS migration planned to replace kscloud1."
metadata: metadata:
node_type: memory node_type: memory
type: project type: project
originSessionId: 33992890-3940-4d4a-a94a-22b5621e9c1a originSessionId: 33992890-3940-4d4a-a94a-22b5621e9c1a
--- ---
## Final Polish, Security, and Runbook Completion (2026-06-15) ## ARCHITECTURE (canonical — 2026-06-19)
The KiteStacks infrastructure is now in its final, secured, and documented state: **kscloud1 = PRODUCTION** (always-on source of truth for live site).
- **GitOps UI/Dashboard:** Added a standalone Nginx container for FluxCD status, bypassing Authentik so Cloudflare edge can route it freely. The dashboard is live at `flux.kitestacks.com`. **monk = DEVELOPMENT** (things are built/tested on monk, then pushed to kscloud1).
- **Security Posture:** Validated Zero Trust architecture. No inbound open ports, strict mesh networking via Tailscale `100.x.x.x`, and Authentik protecting all administrative dashboards (`/scp/` for osTicket, Portainer, Grafana, Kite AI). **Samurai desktop = planned 3rd connector** (when up, keeps kscloud1 redundant; can be down sometimes).
- **Runbook Cleaned:** `RUNBOOK.md` truncated and organized. Historical issues (like Authentik invalid_grant, osTicket email SMTP lack of MTA) have been relocated to `docs/DEBUGGING.md`.
- **osTicket Diagnostics:** Documented that activation emails fail because Docker containers lack a local MTA. Fix involves adding an external SMTP server in the osTicket Admin Panel.
- **Cloudflare Multi-Node Routing:** Diagnosed persistent 502 errors on new subdomains (like `ntfy`). Cloudflare Tunnels actively load balance between `monk` and `kscloud1`. Documented that all new services must be deployed to both nodes to prevent the load balancer from sending traffic to a missing container. Subsequently resolved the `ntfy` 502 error by deploying the container to the `kscloud1` replica and syncing its `user.db` via Tailscale SSH.
## T14s GitOps Automation SUCCESS (2026-06-15) Changes flow: monk develops → monk PUSHES to kscloud1. Always monk → kscloud1, never the reverse.
Monk is the source of all changes (code, config, stateful data). kscloud1 receives and stays live.
CF Tunnel load-balances both connectors — kscloud1 always has the last pushed state. Monk going down = zero user impact since kscloud1 stays live with whatever was last pushed.
The cluster configuration originally for "assassin" (T14) has been moved to the ## STATUS: MIGRATION + CLOUD FAILOVER COMPLETE (2026-06-10)
**T14s**. The machine is now fully bootstrapped with FluxCD GitOps.
- **Cluster Hostname:** monk (T14s) assassin (T14) is OFF. kscloud1 (Hetzner VPS, 5.78.x.x) runs a FULL
- **GitOps Repo:** `kitestacks-homelab` (main branch) replica of all services, so the site stays up even if monk is off
- **Path:** `clusters/T14s` (verified by user testing with home wifi off, from phone + mom's phone).
- **Automation:** FluxCD is now managing the `kavita` namespace.
- **Kavita Manifests:**
- Deployment, Service, PVC (2Gi local-path), and Namespace.
- Successfully synced and running (verified 2026-06-15).
- **Credentials:** Authentik password for `kenpat7177` reset to `KiteStacks2026!`.
- **osTicket:** Services started, DB unified on kscloud1, and verified
accessible via Authentik LDAP.
The GitOps workflow is now the authoritative way to manage Kubernetes apps on the T14s.
monk is the live production host. assassin (T14) is OFF. kscloud1 (Hetzner VPS,
5.78.233.28) is now a THIRD active Cloudflare Tunnel connector and runs a FULL
replica of all 9 services, so the site stays up even if both monk and assassin
are off (verified by user testing with home wifi off, from phone + mom's phone).
All 9 public subdomains (www, ai, auth, gitforge, grafana, kavita, links, status, tasks) All 9 public subdomains (www, ai, auth, gitforge, grafana, kavita, links, status, tasks)
verified returning correct status codes via the live tunnel with kscloud1 in rotation. verified returning correct status codes via the live tunnel with kscloud1 in rotation.
@ -53,8 +37,8 @@ EXPLICITLY ACCEPTED by user as the cost of guaranteed uptime. Fresh/separate
databases on kscloud1 are fine; do not try to sync data between monk and kscloud1. databases on kscloud1 are fine; do not try to sync data between monk and kscloud1.
## kscloud1 access ## kscloud1 access
SSH: `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.233.28` (passwordless, key auth). SSH: `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.x.x` (passwordless, key auth).
sudo needs a password ("p12217177") and has no askpass helper - avoid sudo; sudo needs a password ("[redacted]") and has no askpass helper - avoid sudo;
most things doable as kenpat or via docker. most things doable as kenpat or via docker.
All services live under `/opt/kitestacks/docker/<service>/docker-compose.yml`, All services live under `/opt/kitestacks/docker/<service>/docker-compose.yml`,
same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`. same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`.
@ -74,7 +58,7 @@ same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`.
- kavita (alias `kavita`) - empty library (fresh) - kavita (alias `kavita`) - empty library (fresh)
- karakeep + karakeep-chrome + karakeep-meilisearch (alias `karakeep`) - fresh meilisearch/db - karakeep + karakeep-chrome + karakeep-meilisearch (alias `karakeep`) - fresh meilisearch/db
- authentik + authentik-worker + authentik-postgres + authentik-redis (alias on `auth`) - FRESH DB. - authentik + authentik-worker + authentik-postgres + authentik-redis (alias on `auth`) - FRESH DB.
Bootstrap admin: `akadmin@kitestacks.com` / password `6KlYpfCyYxbnKQNiOewN` (set via Bootstrap admin: `akadmin@kitestacks.com` / password `[redacted]` (set via
AUTHENTIK_BOOTSTRAP_PASSWORD in .env). No OAuth provider apps exist yet (would need to be AUTHENTIK_BOOTSTRAP_PASSWORD in .env). No OAuth provider apps exist yet (would need to be
manually recreated in authentik UI for grafana/openwebui/karakeep/openproject SSO to work manually recreated in authentik UI for grafana/openwebui/karakeep/openproject SSO to work
when kscloud1 is the active backend). when kscloud1 is the active backend).
@ -86,7 +70,7 @@ same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`.
## monk-side changes made for cross-host monitoring ## monk-side changes made for cross-host monitoring
- `~/kitestacks-live/docker/prometheus/prometheus.yml`: added scrape job - `~/kitestacks-live/docker/prometheus/prometheus.yml`: added scrape job
`kscloud1-node` -> `5.78.233.28:9100` (kscloud1's node-exporter is exposed `kscloud1-node` -> `5.78.x.x:9100` (kscloud1's node-exporter is exposed
0.0.0.0:9100, no firewall - reachable from monk's public IP). monk's grafana 0.0.0.0:9100, no firewall - reachable from monk's public IP). monk's grafana
(the live one, "Node Exporter Full" dashboard now provisioned via (the live one, "Node Exporter Full" dashboard now provisioned via
`~/kitestacks-live/docker/grafana/provisioning/`) shows BOTH `t14-node` `~/kitestacks-live/docker/grafana/provisioning/`) shows BOTH `t14-node`
@ -163,16 +147,16 @@ because the data is created fresh on every login attempt.
FIX: Converted to a single shared Postgres+Redis (HA pattern), hosted on FIX: Converted to a single shared Postgres+Redis (HA pattern), hosted on
kscloud1, reachable ONLY over Tailscale: kscloud1, reachable ONLY over Tailscale:
- Installed Tailscale on both monk and kscloud1 (same tailnet). kscloud1's - Installed Tailscale on both monk and kscloud1 (same tailnet). kscloud1's
tailscale IP is `100.123.254.52`. tailscale IP is `100.x.x.x`.
- kscloud1's `/opt/kitestacks/docker/authentik/docker-compose.yml`: - kscloud1's `/opt/kitestacks/docker/authentik/docker-compose.yml`:
authentik-postgres now binds `100.123.254.52:5432:5432` (was unbound/internal-only), authentik-postgres now binds `100.x.x.x:5432:5432` (was unbound/internal-only),
authentik-redis now binds `100.123.254.52:6379:6379`. Both still also reachable authentik-redis now binds `100.x.x.x:6379:6379`. Both still also reachable
on the local `kitestacks` docker network for kscloud1's own authentik+worker. on the local `kitestacks` docker network for kscloud1's own authentik+worker.
Backup of pre-change file: `docker-compose.yml.backup-before-shared-db-20260610-1138`. Backup of pre-change file: `docker-compose.yml.backup-before-shared-db-20260610-1138`.
- monk's `~/kitestacks-live/docker/authentik/docker-compose.yml`: REMOVED the - monk's `~/kitestacks-live/docker/authentik/docker-compose.yml`: REMOVED the
`postgresql` and `redis` services entirely. monk's `authentik`/`authentik-worker` `postgresql` and `redis` services entirely. monk's `authentik`/`authentik-worker`
now point `AUTHENTIK_POSTGRESQL__HOST` and `AUTHENTIK_REDIS__HOST` at now point `AUTHENTIK_POSTGRESQL__HOST` and `AUTHENTIK_REDIS__HOST` at
`100.123.254.52` (kscloud1 over Tailscale), using the same `PG_PASS` / `100.x.x.x` (kscloud1 over Tailscale), using the same `PG_PASS` /
`AUTHENTIK_SECRET_KEY` as before (already identical between hosts). `AUTHENTIK_SECRET_KEY` as before (already identical between hosts).
- monk's old local `authentik-postgres`/`authentik-redis` containers were - monk's old local `authentik-postgres`/`authentik-redis` containers were
STOPPED (not removed) - data dirs preserved under STOPPED (not removed) - data dirs preserved under
@ -213,7 +197,7 @@ cleared, Enabled->false) - confirmed twice, even with a full WAL-consistent
kavita.db replace from monk. Direct DB writes to this table do NOT survive a kavita.db replace from monk. Direct DB writes to this table do NOT survive a
restart; only saves through Kavita's own Settings UI/API persist correctly. restart; only saves through Kavita's own Settings UI/API persist correctly.
FIX: opened an SSH local port-forward (`ssh -L 5099:localhost:5000 FIX: opened an SSH local port-forward (`ssh -L 5099:localhost:5000
kenpat@5.78.233.28`) so the user could reach kscloud1's Kavita directly at kenpat@5.78.x.x`) so the user could reach kscloud1's Kavita directly at
http://localhost:5099 (bypassing the Cloudflare load-balanced domain), logged http://localhost:5099 (bypassing the Cloudflare load-balanced domain), logged
in with their normal kenpat7177 Kavita password, and re-entered the OIDC in with their normal kenpat7177 Kavita password, and re-entered the OIDC
config in Settings -> OIDC: config in Settings -> OIDC:
@ -255,9 +239,9 @@ are added on monk later, they won't appear on kscloud1 unless re-synced
(covers/ dir + kavita.db + actual book files under library/books, none of (covers/ dir + kavita.db + actual book files under library/books, none of
which exist on kscloud1 per the earlier "stale data" note). which exist on kscloud1 per the earlier "stale data" note).
SECURITY NOTE: postgres/redis on kscloud1 are bound to the Tailscale interface SECURITY NOTE: postgres/redis on kscloud1 are bound to the Tailscale interface
IP only (100.123.254.52), not 0.0.0.0 - not exposed to the public internet. IP only (100.x.x.x), not 0.0.0.0 - not exposed to the public internet.
ROLLBACK: if Tailscale connectivity ever breaks, monk's authentik will fail to ROLLBACK: if Tailscale connectivity ever breaks, monk's authentik will fail to
start (can't reach 100.123.254.52). To roll back: restore monk's start (can't reach 100.x.x.x). To roll back: restore monk's
docker-compose.yml from git/backup to use local postgresql/redis services docker-compose.yml from git/backup to use local postgresql/redis services
again, restart monk's old authentik-postgres/authentik-redis containers again, restart monk's old authentik-postgres/authentik-redis containers
(`docker start authentik-postgres authentik-redis` in (`docker start authentik-postgres authentik-redis` in
@ -314,7 +298,7 @@ OAuth callback path is `/api/auth/callback/custom`, but Authentik's Karakeep
OAuth2Provider's `_redirect_uris` had the wrong path -> "Redirect URI Error". OAuth2Provider's `_redirect_uris` had the wrong path -> "Redirect URI Error".
FIX: direct Postgres UPDATE to FIX: direct Postgres UPDATE to
`authentik_providers_oauth2_oauth2provider._redirect_uris` (JSON column) on `authentik_providers_oauth2_oauth2provider._redirect_uris` (JSON column) on
the shared kscloud1 authentik-postgres (100.123.254.52), wrapped in explicit the shared kscloud1 authentik-postgres (100.x.x.x), wrapped in explicit
`BEGIN; UPDATE ...; COMMIT;` (a bare single-statement -c "UPDATE..." reported `BEGIN; UPDATE ...; COMMIT;` (a bare single-statement -c "UPDATE..." reported
"UPDATE 1" but did NOT persist on first attempt - cause unclear, explicit "UPDATE 1" but did NOT persist on first attempt - cause unclear, explicit
transaction fixed it). After the DB write, restarted authentik+authentik-worker transaction fixed it). After the DB write, restarted authentik+authentik-worker
@ -363,7 +347,7 @@ the `homelab-admin` Authentik group).
Created via `docker exec authentik ak shell` (Django ORM, no Authentik API Created via `docker exec authentik ak shell` (Django ORM, no Authentik API
token configured) on kscloud1's shared authentik-postgres: token configured) on kscloud1's shared authentik-postgres:
- OAuth2Provider "Portainer": client_id=`portainer`, - OAuth2Provider "Portainer": client_id=`portainer`,
client_secret=`wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF`, client_secret=`[redacted]`,
provider_id=9, redirect_uri=`https://portainer.kitestacks.com` (strict), provider_id=9, redirect_uri=`https://portainer.kitestacks.com` (strict),
scopes openid/email/profile, sub_mode=user_email, signing key + flows copied scopes openid/email/profile, sub_mode=user_email, signing key + flows copied
from existing providers (same pattern as Karakeep/Grafana). from existing providers (same pattern as Karakeep/Grafana).
@ -384,7 +368,7 @@ still returns `000` as of 2026-06-10):
2. In Portainer -> Settings -> Authentication -> OAuth (Provider: Custom), on 2. In Portainer -> Settings -> Authentication -> OAuth (Provider: Custom), on
BOTH monk's and kscloud1's SEPARATE Portainer instances, configure: BOTH monk's and kscloud1's SEPARATE Portainer instances, configure:
- Client ID: `portainer` - Client ID: `portainer`
- Client Secret: `wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF` - Client Secret: `[redacted]`
- Authorization URL: `https://auth.kitestacks.com/application/o/authorize/` - Authorization URL: `https://auth.kitestacks.com/application/o/authorize/`
- Access Token URL: `https://auth.kitestacks.com/application/o/token/` - Access Token URL: `https://auth.kitestacks.com/application/o/token/`
- Resource/Userinfo URL: `https://auth.kitestacks.com/application/o/userinfo/` - Resource/Userinfo URL: `https://auth.kitestacks.com/application/o/userinfo/`
@ -403,259 +387,12 @@ above. Prometheus + Uptime Kuma: DEFERRED - neither has native OAuth, need a
forward-auth proxy (oauth2-proxy or Authentik embedded outpost) - deferred per forward-auth proxy (oauth2-proxy or Authentik embedded outpost) - deferred per
user's "ok lets do smaller app level" (hold new infra until Oracle VPS decided). user's "ok lets do smaller app level" (hold new infra until Oracle VPS decided).
Cloudflare itself: no SSO concept applicable (it's Cloudflare's own dashboard Cloudflare itself: no SSO concept applicable (it's Cloudflare's own dashboard
managed outside the lab login) - was always about the portal's Cloudflare card login) - was always about the portal's Cloudflare card placement, see "Portal UI
placement, see "Portal UI changes" note above. changes" note above.
### Uptime Kuma + Authentik SSO resumed on monk (2026-06-15)
User confirmed the next task is setting up Uptime Kuma with Authentik SSO in
the main KiteStacks lab, and explicitly requested saving progress to
`~/claude-memory` and pushing to the Forgejo `kenpat/claude-memory` repo as we
go.
Verified current live state on monk before making changes:
- `uptime-kuma` container is running and healthy, published on host port
`3001`, image `louislam/uptime-kuma:latest`.
- Installed Uptime Kuma version inside the container is `1.23.17`.
- Uptime Kuma compose file is
`~/kitestacks-live/docker/uptime-kuma/docker-compose.yml`, using external
Docker volume `uptime-kuma:/app/data` and networks `default` + external
`kitestacks`.
- Uptime Kuma SQLite DB path inside container is `/app/data/kuma.db`; tables
include `user`, `setting`, `monitor`, `heartbeat`, `status_page`,
`notification`, `api_key`, and related monitor/status tables. No obvious
native OAuth/OIDC tables were present in the initial schema list.
- Grafana is already configured for Authentik generic OAuth in
`~/kitestacks-live/docker/grafana/docker-compose.yml` with Authentik public
authorize URL and internal token/userinfo URLs.
- `authentik` is healthy; `authentik-worker` currently shows unhealthy in
`docker ps` even though it has been running for ~35h. Check logs/health
before relying on new Authentik-side automation.
- Existing Authentik objects were found for Uptime Kuma:
- Application slug `uptime-kuma`, name `Uptime Kuma`, provider id `7`.
- ProxyProvider `Uptime Kuma`, external host `https://status.kitestacks.com`,
internal host `http://uptime-kuma:3001`, mode `proxy`.
- Embedded proxy outpost already includes providers `Karakeep`,
`Uptime Kuma`, and `LiteLLM`.
- `https://status.kitestacks.com` still routes directly to Kuma as of
2026-06-15: public curl gets Kuma's `/dashboard` redirect and 200 response,
not an Authentik authorization flow. Cloudflare tunnel route still needs to
be changed from direct Kuma to the Authentik embedded outpost/server.
- Security fix applied 2026-06-15: created PolicyBinding
`6f2ac876-2f47-473d-986d-d7c5d2a3214e` from the Uptime Kuma application to
Authentik group `homelab-admin`, enabled, order 0. This matches the Portainer
restriction pattern.
- Cloudflared is remote-managed: container command is `tunnel --no-autoupdate
run`, no local ingress config exists, and the compose file stores a
`TUNNEL_TOKEN`. Do not print that token; treat it as sensitive. Routing
changes must be made through Cloudflare's tunnel API/dashboard unless a
suitable Cloudflare API token is available locally.
- Local validation after the Authentik binding: `curl -I -H 'Host:
status.kitestacks.com' http://localhost:9001` returns `302` to
`https://status.kitestacks.com/outpost.goauthentik.io/start?...`, proving
the embedded outpost/proxy provider works when traffic reaches Authentik.
- No suitable Cloudflare API token was found during the local search; only the
cloudflared connector tunnel token is present. Remaining blocker is changing
the Cloudflare Tunnel public hostname for `status.kitestacks.com` from
`http://uptime-kuma:3001` to `http://authentik:9000` (or equivalent
Authentik service target in the Tunnel UI).
- Correction after user tested: user does NOT want front-door proxy behavior
for Uptime Kuma. Desired UX is an in-app "single sign on" button on the
Uptime Kuma login screen, like Grafana/Forgejo style native OAuth. Authentik
proxy redirect is not acceptable for this requirement.
- Confirmed in the installed Uptime Kuma 1.23.17 frontend:
`/app/src/components/Login.vue` only renders username, password, remember-me,
and login submit controls. No native OAuth/OIDC/SSO button exists in this
version's login component, and local source search only found monitor OAuth
client-credentials support, not app login SSO.
- If staying on Uptime Kuma 1.23.17, revert Cloudflare route for
`status.kitestacks.com` back to `http://uptime-kuma:3001`; otherwise users
get Authentik first and then still see Kuma's local login. Native in-app SSO
would require an Uptime Kuma version/plugin/fork with login OIDC support or
custom app code, not the Authentik proxy provider.
- User reset the Cloudflare route back to `http://uptime-kuma:3001` and asked
to continue with an in-app Authentik button. Upstream latest checked via
GitHub API: Uptime Kuma latest release is `2.4.0` (published 2026-05-31) and
upstream `src/components/Login.vue` still has only username/password login,
no native OAuth/OIDC button. Proceeded with a custom overlay patch.
- Custom native Authentik SSO overlay deployed on BOTH active tunnel backends
(monk and kscloud1) so public load-balanced traffic behaves consistently:
- monk path: `~/kitestacks-live/docker/uptime-kuma/`
- kscloud1 path: `/opt/kitestacks/docker/uptime-kuma/`
- backend preload module:
`custom/server/authentik-sso.js`
- frontend mounted files:
`custom/dist/index.html`, `index.html.gz`, `index.html.br`
- compose now sets `NODE_OPTIONS=--require /app/custom/server/authentik-sso.js`,
loads `.env.sso`, and bind-mounts the custom files over Kuma's built HTML.
- Authentik native OAuth provider/application created:
- OAuth2Provider name `Uptime Kuma Native`, provider id `12`
- Application slug `uptime-kuma-native`, name `Uptime Kuma Native SSO`
- Client ID `uptime-kuma-native`
- Redirect URI `https://status.kitestacks.com/auth/authentik/callback`
- Restricted to Authentik group `homelab-admin` via PolicyBinding
`2e1eaa95-b397-4c4f-bfc7-abb337906cf3`
- Client secret is stored only in each host's `.env.sso`; do not print it.
- Custom flow behavior:
- Login page injects a `Sign in with Authentik` button linking to
`/auth/authentik`.
- Backend starts Authentik OIDC, validates callback state, fetches userinfo,
maps the login to existing Kuma user `kenpat`, issues Kuma's normal JWT,
then redirects to `/?authentik_token=<token>`.
- Frontend one-time script stores the JWT in `localStorage.token`, removes
the URL token, and redirects to `/dashboard`, letting Kuma's normal
`loginByToken` flow establish the session.
- Verification 2026-06-15:
- monk local `/dashboard` HTML contains `Sign in with Authentik`,
`/auth/authentik`, and `authentik_token`.
- kscloud1 local `/dashboard` HTML contains the same and `/auth/authentik`
redirects to Authentik with client_id `uptime-kuma-native`.
- Public repeated check:
`for i in 1 2 3 4 5 6; do curl -sSL --compressed https://status.kitestacks.com/dashboard | grep -q "Sign in with Authentik"; done`
returned `button` for all 6 attempts, confirming both active connectors
serve the button.
- Post-test screenshot showed Uptime Kuma login page with red banner "Lost
connection to the socket server. Reconnecting..." after clicking the SSO
button. Root cause: active-active JWT mismatch. Uptime Kuma JWTs include a
signature using `setting.jwtSecret`; monk and kscloud1 had matching user
password hashes but different JWT secrets, so a token minted by one backend
failed if the browser's websocket connected to the other backend. Fixed
2026-06-15 by copying monk's exact `jwtSecret` into kscloud1's
`/app/data/kuma.db` using base64 transport (avoid shell expansion of secret
chars), then restarting kscloud1 Uptime Kuma. Verified both hashes now match:
`jwtSecret` length 60, sha3 prefix `FA67E6E9EDCC8E1D`. Public button check
still returns `button` 6/6. If a browser still has a pre-fix bad token in
localStorage, clear site data or click the Authentik button again to mint a
fresh token.
- User retested and still saw the socket reconnect banner. Follow-up finding:
public Uptime Kuma frontend was using Socket.IO's default long-polling-first
transport. In the active-active Cloudflare Tunnel setup, polling requests can
bounce between monk and kscloud1 before a socket session is established,
causing reconnect loops before Kuma even logs `Login by token`.
- Fix applied 2026-06-15 on BOTH monk and kscloud1: copied the built frontend
bundle `index-BBxTfFCS.js` into the overlay and patched the minified socket
call from `Ze=Nc(n)` to `Ze=Nc(n,{transports:["websocket"]})`. Regenerated
`.gz` and `.br` variants and mounted all three over
`/app/dist/assets/index-BBxTfFCS.js*` in both compose files. Restarted both
Uptime Kuma containers.
- Verification after websocket-only patch:
- monk local asset contains `transports:["websocket"]`
- kscloud1 local asset contains `transports:["websocket"]`
- public repeated asset check over `https://status.kitestacks.com/assets/index-BBxTfFCS.js`
found `transports:["websocket"]` 6/6, confirming both tunnel backends serve
the patched client bundle.
- User still saw the same issue after trying another browser. Follow-up:
websocket connections were reaching Kuma, but logs showed no `Login by token`,
so the handoff from Authentik callback to Kuma storage was unreliable. Changed
the SSO callback from `/?authentik_token=<jwt>` URL handoff to a short-lived
readable cookie `uk_authentik_token` plus redirect directly to `/dashboard`.
Updated injected HTML to read that cookie before Kuma initializes, store the
token in `localStorage.token`, set `localStorage.remember=1`, then delete the
cookie. This avoids long-token URL handling.
- Important operational gotcha: Uptime Kuma caches `index.html` in memory at
startup. After changing the mounted `index.html`/compressed variants, `docker
compose up -d` was not enough because containers stayed "Running"; had to run
`docker compose restart uptime-kuma` on BOTH monk and kscloud1 to reload the
HTML into memory.
- Verification after cookie handoff + explicit restarts:
- monk local `/dashboard` HTML contains `uk_authentik_token`, `authentik_token`,
and `Sign in with Authentik`.
- kscloud1 local `/dashboard` HTML contains the same.
- public repeated check for `uk_authentik_token` over
`https://status.kitestacks.com/dashboard` returned `cookie-handoff` 6/6.
- User confirmed after retest: Uptime Kuma Authentik SSO button works.
### Uptime Kuma monitors mirrored into Prometheus/Grafana (2026-06-15)
User asked to set up the same monitors currently in Uptime Kuma for Grafana and
Prometheus. Existing Uptime Kuma monitor list at the time:
- `T14 Deb Assassin`: ping `127.0.0.1`
- `HomeRouter`: ping `192.168.1.254`
- `Google DNS`: ping `8.8.8.8`
- `TailScale`: ping `100.90.13.55`
Implemented on monk's live Prometheus/Grafana stack:
- Added `prom/blackbox-exporter` service to
`~/kitestacks-live/docker/prometheus/docker-compose.yml`.
- Added blackbox config
`~/kitestacks-live/docker/prometheus/blackbox.yml` with ICMP module
(`preferred_ip_protocol: ip4`, timeout 5s).
- Added Prometheus scrape job `uptime-kuma-ping-probes` in
`~/kitestacks-live/docker/prometheus/prometheus.yml`, using `/probe` with
`module=icmp` and labels `monitor_name` matching the Uptime Kuma names.
- Added Grafana provisioned dashboard
`~/kitestacks-live/docker/grafana/provisioning/dashboards/kitestacks-uptime-probes.json`
titled `KiteStacks Uptime Probes`, with stat/timeseries panels for
`probe_success{job="uptime-kuma-ping-probes"}` and
`probe_duration_seconds{job="uptime-kuma-ping-probes"}`.
- Ran `docker compose up -d` in the Prometheus directory, pulled/started
`blackbox-exporter`, restarted Prometheus, and restarted Grafana.
Verification:
- Prometheus config validates with `promtool check config`.
- Prometheus active targets include all four `uptime-kuma-ping-probes`.
- Query result for `probe_success{job="uptime-kuma-ping-probes"}`:
`Google DNS=1`, `T14 Deb Assassin=1`, `HomeRouter=0`, `TailScale=0`.
The two failures match Kuma's existing failing ping behavior from inside the
container/network namespace.
- Grafana logs show dashboard provisioning completed without dashboard errors
(only unrelated bundled plugin permission warnings).
### Desktop widget for the same monitor set (2026-06-15)
User asked for a Rainmeter-like desktop widget on Debian 13 that can show the
same Uptime Kuma monitor state in real time.
Created a local Conky-based widget scaffold in the desktop user's home:
- `~/.local/bin/kitestacks-uptime-widget.sh`
- `~/.config/conky/kitestacks-uptime.conf`
Behavior:
- Polls Prometheus for `probe_success` and `probe_duration_seconds` from the
`uptime-kuma-ping-probes` job.
- Defaults to `http://192.168.1.205:9090`, with `PROM_URL`
override support.
- Prints the four Kuma monitor names, state, latency, and a summary line.
- Degrades cleanly with `Prometheus unavailable at ...` when the endpoint
cannot be reached.
Note: Conky is the closest direct Rainmeter-style equivalent for Debian/Linux
desktop widgets; `eww` is the more modern alternative if the desktop session is
Wayland-first and the user prefers GTK/Rust widgets instead of a classic
desktop overlay.
Debian 13 package note:
- `conky` is a virtual package in trixie.
- Install `conky-all` for the full desktop widget experience:
`sudo apt update && sudo apt install conky-all`
Connectivity note:
- The laptop could not reach Prometheus at `192.168.1.205:9090`, which means
the widget can only work from a host that can reach the homelab LAN or a
public/tunneled Prometheus endpoint.
- The existing KiteStacks docs mark Prometheus as excluded from the Cloudflare
tunnel, so there is no known public Prometheus URL to target yet.
- The desktop widget script now defaults to `https://prometheus.kitestacks.com`
and can send `CF-Access-Client-Id` / `CF-Access-Client-Secret` headers if the
hostname is protected by Cloudflare Access.
Cyberpunk widget styling:
- Conky panel tuned to the wallpaper palette with black base and neon
cyan/magenta accents.
- Header uses `#ff4df0` pink and `#2de0ff` blue.
- Monitor rows color-code `UP` as cyan and `DOWN` as pink for fast scanning.
- `conky.text` now uses `execpi` so the helper's parsed color markup renders as
one combined widget instead of only the title line.
- The screenshot also showed a separate default Conky panel on the left; that
is not part of the uptime widget itself.
- Added a unified Conky desktop config at `~/.conkyrc` plus an autostart
wrapper that kills stray Conky instances and launches the single combined
panel.
Important security hygiene: local git remote for `~/claude-memory` contains an
HTTP token in the URL; do not print it in summaries. Prefer redacted URLs in
handoffs.
### Oracle VPS migration - PLANNED, upcoming (stated 2026-06-11) ### Oracle VPS migration - PLANNED, upcoming (stated 2026-06-11)
User confirmed on 2026-06-11: "we are going to switch things soon from hetzner User confirmed on 2026-06-11: "we are going to switch things soon from hetzner
cloud to oracle soon." -> kscloud1 (Hetzner, 5.78.233.28) is intended to be cloud to oracle soon." -> kscloud1 (Hetzner, 5.78.x.x) is intended to be
REPLACED by an Oracle Cloud VPS in the near future ("soon", no firm date yet). REPLACED by an Oracle Cloud VPS in the near future ("soon", no firm date yet).
Originally raised 2026-06-10 as exploratory ("how easy would it be to move Originally raised 2026-06-10 as exploratory ("how easy would it be to move
everything to oracle vps after?"), now an actual plan. everything to oracle vps after?"), now an actual plan.
@ -687,7 +424,7 @@ from `minutes=1` to `minutes=10` for ALL 9 OAuth2 providers in the shared Postgr
DB. This gives enough buffer for monk's containers to start before codes expire. DB. This gives enough buffer for monk's containers to start before codes expire.
Command used (via python:3-alpine container): Command used (via python:3-alpine container):
`docker run --rm --network host -v /tmp/fix_auth.py:/fix.py python:3-alpine sh -c ...` `docker run --rm --network host -v /tmp/fix_auth.py:/fix.py python:3-alpine sh -c ...`
connecting to shared Postgres at 100.123.254.52. connecting to shared Postgres at 100.x.x.x.
### Karakeep redirect_uri reverted and re-fixed ### Karakeep redirect_uri reverted and re-fixed
The Karakeep OAuth2Provider `_redirect_uris` had reverted back to the proxy pattern The Karakeep OAuth2Provider `_redirect_uris` had reverted back to the proxy pattern
@ -709,196 +446,32 @@ STILL PENDING (user action in both Portainer UIs): configure OAuth (see prior no
in "Portainer SSO" section above for exact credentials). in "Portainer SSO" section above for exact credentials).
Portal card update (3 files) also still pending until tunnel+OAuth done. Portal card update (3 files) also still pending until tunnel+OAuth done.
## 2026-06-16: Cloudflare Tunnel cleanup + failover verified
### Phantom 3rd replica — found and fixed
CF dashboard showed 3 active replicas (expected: 2). Cause: a native `cloudflared`
systemd service (`/usr/bin/cloudflared`, v2026.6.4) was running on monk alongside the
Docker container (`/usr/local/bin/cloudflared`, v2026.6.0). Both connected to CF with
the same TUNNEL_TOKEN, registering as separate connectors. Fixed:
```bash
sudo systemctl stop cloudflared
sudo systemctl disable cloudflared
```
CF dashboard now shows 2 replicas: monk (Docker) + kscloud1. Architecture is
**2-connector active-active**, not 3. The old "3rd connector" references in docs/memory
(which referred to T14s/assassin) are now corrected.
### kscloud1 SSH access — restored 2026-06-19
SSH key auth to kscloud1 was failing. Recovery: Hetzner console → reset root password
([redacted]) → added `id_ed25519_kscloud1.pub` to `/home/kenpat/.ssh/authorized_keys`.
NOTE: user on kscloud1 is `kenpat` (not `kenpatmonk`). Connect:
`ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.x.x.x`
Sudo password: [redacted] (use `echo [redacted] | sudo -S <cmd>` for non-interactive).
### Failover verified 2026-06-16
Stopped monk's Docker cloudflared → tested all public subdomains → kscloud1 served
everything with zero downtime: www=200, auth=302, status=302, portainer=200.
Restarted monk cloudflared, both replicas back in CF dashboard.
## Phase 2 Planned: Obsidian Mind Map → HTML Mind Map Sync ## Phase 2 Planned: Obsidian Mind Map → HTML Mind Map Sync
User wants to create an Obsidian mind map of the KiteStacks homelab that syncs/exports to a live HTML mind map embedded in the homelab portal or a standalone page. To be built after full Obsidian+samurai setup is complete. User wants to create an Obsidian mind map of the KiteStacks homelab that syncs/exports to a live HTML mind map embedded in the homelab portal or a standalone page. To be built after full Obsidian+samurai setup is complete.
## 2026-06-13: OpenProject removed + Oracle VPS migration started
### OpenProject REMOVED permanently
OpenProject requires Enterprise Edition license for SSO (confirmed last session).
Removed from local stack (monk):
- Docker volume `openproject_openproject_assets` deleted
- `/home/kenpatmonk/kitestacks-live/docker/openproject/` directory removed (pgdata dir
needed sudo — user ran manually; pgdata was owned by container UID mapped to `avahi`)
- NOT deploying on Oracle VPS
- tasks.kitestacks.com subdomain is now dead — update Cloudflare/portal accordingly
TODO: remove `apps/openproject/` from kitestacks-homelab Forgejo repo once user can log in.
### Forgejo issues found + partially fixed (2026-06-13)
Forgejo login page has two issues:
1. URL banner: "configured to be served on http://5.78.233.28:3000/" — caused by kscloud1's
Forgejo having wrong ROOT_URL. kscloud1 Forgejo has only 1 repo (separate DB from monk's
13-repo instance). Cloudflare tunnel load-balances between monk and kscloud1 Forgejo.
FIX PENDING: stop Forgejo on kscloud1 (or fix its ROOT_URL). Deferred — do during Oracle migration.
2. SSO button says "Proceed with OpenID" instead of "Authentik".
PARTIAL FIX: renamed login_source from `authentik` → `Authentik` via admin CLI:
`docker exec -u git forgejo /app/gitea/gitea admin auth update-oauth --id 1 --name Authentik ...`
Provider type remains `openidConnect` — button text may still say "OpenID" (depends on
Forgejo 11 template behavior). User to verify after refresh. Full fix may require admin UI
once user can log into Forgejo.
Forgejo DB: 13 repos under `kenpat`, 1 user (kenpat, admin, active, no 2FA).
Forgejo login: username `kenpat`, direct password login works on the same page.
### kitestacks-homelab repo: apps/forgejo/docker-compose.yml has wrong ROOT_URL
`FORGEJO__server__ROOT_URL=http://192.168.1.205:3006` — old local IP, never updated.
The LIVE local stack (`~/kitestacks-live/docker/forgejo/docker-compose.yml`) is correct
(`https://gitforge.kitestacks.com/`). The repo copy needs updating.
TODO: fix and commit once user can log in and clone the repo.
### Oracle VPS migration plan (kscloud1 → Oracle Cloud)
Goal: replace Hetzner kscloud1 (5.78.233.28, $14.50/mo) with Oracle Cloud ARM VPS ($8.50/mo).
Oracle instance: Ampere A1 Flex, 4 OCPU / 24 GB RAM, Chicago region (us-chicago-1).
Status as of 2026-06-13: user is provisioning — hit "no capacity" in Chicago.
Workarounds tried: capacity not available for 4 OCPU config. Options:
- Try smaller shape (1 OCPU / 6 GB), resize after provisioning
- Subscribe to another region (Frankfurt, Osaka, Toronto have better A1 availability)
- Keep retrying (capacity opens randomly, early UTC morning tends to be better)
ARM64 compatibility analysis (all images verified):
- ✅ All services ARM64-compatible EXCEPT OSticket
- ❌ OSticket (`campbellsoftwaresolutions/osticket`) — x86 only
FIX: enable QEMU binfmt emulation on Oracle ARM host, run with `--platform linux/amd64`
Performance acceptable for a ticket system.
- ⚠️ Shaarli — verify ARM64 at deploy time
Services to deploy on Oracle VPS (OpenProject EXCLUDED):
authentik, bookstack, cloudflared, forgejo, grafana, homepage/portal,
karakeep (+meilisearch +chrome), kavita, kite-ai (litellm+openwebui),
linkding, osticket, portainer, prometheus+node-exporter, shaarli, uptime-kuma
Migration phases:
1. Oracle VPS provisioning (in progress)
2. Oracle initial setup: Ubuntu 22.04 ARM64, Docker, iptables flush (Oracle blocks by default),
QEMU binfmt for OSticket x86 emulation
3. Deploy full stack — fix Forgejo ROOT_URL correctly from day one
4. Connect cloudflared on Oracle to KiteStacks tunnel (same TUNNEL_TOKEN)
5. Verify all services, then remove kscloud1 from tunnel + cancel Hetzner
NOTE: same active-active pattern as kscloud1 — shared Authentik Postgres+Redis over
Tailscale, same TUNNEL_TOKEN, fresh DBs for stateful apps except identity (authentik/kavita).
IMPORTANT Oracle gotcha: Ubuntu on Oracle has iptables rules that block all traffic at boot
even after Security List rules are opened. Must flush iptables as part of initial setup.
## osTicket deployed on monk + kscloud1 (found 2026-06-13/14, installed ~2026-06-12)
osTicket (campbellsoftwaresolutions/osticket image, x86 - runs natively on both hosts,
no QEMU needed) + nginx proxy + MariaDB 10.11, under
`~/kitestacks-live/docker/osticket/` (monk) and `/opt/kitestacks/docker/osticket/`
(kscloud1). `tasks.kitestacks.com` -> "KiteStacks Help Desk", verified HTTP 200.
Admin: kenpat7177 / kenpat7177@gmail.com. Host ports: monk 8092:8080, kscloud1 8090:8080
(both nginx -> osticket-app:80). .env (OSTICKET_DB_PASS/ROOT/ADMIN_PASS/INSTALL_SECRET)
is IDENTICAL on both hosts.
### DB unification (2026-06-13/14) - same pattern as Authentik shared-DB fix
Both hosts originally had their OWN osticket-db (drift risk like pre-fix Kavita). Per
user request ("database should be accessible from any computer"), unified onto
kscloud1's osticket-db as canonical:
- kscloud1 osticket-db: added `ports: - "100.123.254.52:3306:3306"` (Tailscale-only,
matches authentik-postgres/redis pattern) to
`/opt/kitestacks/docker/osticket/docker-compose.yml`, `docker compose up -d`.
- monk: `docker compose stop osticket-db` (left stopped, NOT removed - rollback data
intact in its volume). Edited `~/kitestacks-live/docker/osticket/docker-compose.yml`:
removed osticket-db service block, changed osticket-app's `MYSQL_HOST=osticket-db`
-> `MYSQL_HOST=100.123.254.52`, removed `depends_on: osticket-db`. `docker compose
up -d osticket-app`.
- GOTCHA: after recreating osticket-app, the `osticket` nginx proxy container on monk
returned 502 (cached stale upstream IP for osticket-app from its old container) -
fixed with `docker restart osticket`. Apply this same restart on kscloud1's `osticket`
nginx if its osticket-app is ever recreated.
- Verified: both DBs had identical data before merge (1 ticket, 1 staff/kenpat7177) so
no data loss either way. tasks.kitestacks.com returns 200 consistently post-merge.
- Backups: `docker-compose.yml.bak` left in both hosts' osticket dirs.
### osticket-capstone Forgejo repo (created 2026-06-13/14)
New private repo `kenpat/osticket-capstone` on gitforge (created via API using a
scoped token `claude-capstone-osticket` generated via
`docker exec -u git forgejo /app/gitea/gitea admin user generate-access-token` on
monk's forgejo container - token has write:repository,write:user scopes). Holds
redacted osTicket deployment config + Per Scholas capstone docs/evidence - see
[[project-per-scholas-capstone]]. NOTE: gitforge.kitestacks.com is also
active-active load-balanced (monk/kscloud1 separate forgejo DBs) - API calls
against the public hostname can hit the wrong DB; use monk's local
`http://localhost:3006` for API operations tied to monk's forgejo data.
### Remaining osTicket work
- Authentik SSO plugin for osTicket staff/agent login (osTicket has no native OIDC,
needs 3rd-party OAuth2/SAML plugin) - NOT YET DONE.
- End-user ticket submission uses osTicket's native client portal signup (works
out of the box, no SSO needed).
## 2026-06-14/15: Forgejo sync fixed + osTicket Authentik LDAP SSO complete
### Forgejo sync (monk → kscloud1) - FIXED
- Ran `docker exec -u git forgejo /app/gitea/gitea dump` on monk, scp'd to kscloud1
- Restored: 13 repos + DB synced, ROOT_URL fixed on kscloud1 to `https://gitforge.kitestacks.com/`
- kscloud1 Forgejo docker-compose updated (correct ROOT_URL + SSH port 2222)
- Sync script: `~/kitestacks-live/docker/forgejo/sync-to-cloud.sh` (rsync repos + DB dump)
- Cron: `0 */6 * * *` runs sync-to-cloud.sh, logs to `/tmp/forgejo-sync.log`
- Authentik redirect URI fixed: updated `_redirect_uris` in shared Postgres from
`authentik/callback` → `Authentik/callback` (matched renamed Forgejo source name)
### osTicket Authentik LDAP SSO - COMPLETE (2026-06-14/15)
Uses Authentik's LDAP outpost + osTicket's built-in auth-ldap.phar plugin.
**Authentik side:**
- LDAPProvider "osTicket LDAP" (pk=11, base_dn=DC=ldap,DC=goauthentik,DC=io)
- Application "osTicket LDAP" (slug=osticket-ldap, backchannel provider)
- Outpost "osTicket LDAP Outpost" (pk=5c42f5ba-64bd-434e-a47f-7ce9da13227a)
- Outpost service token: `jjYRKWuGtoeq9r0qeifbCnXGHDjhCJU2MLnkCvMMduIGA1kQKz85qnt7u5Zf`
- ldap-svc user (search account): DN=`cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io`
password=`IlgQaxBPv9rdoq03CsoY53tH`, member of homelab-admin group
**Docker services added on monk:**
- `~/kitestacks-live/docker/authentik-ldap/docker-compose.yml`
- `authentik-ldap` (ghcr.io/goauthentik/ldap:2025.2.4) on kitestacks+osticket_default networks
- `authentik-ldap-proxy` (alpine/socat) bridges port 389→3389 on osticket_default
so osticket-app can reach standard LDAP port without phar URI workaround
**Docker services added on kscloud1:**
- `/opt/kitestacks/docker/authentik-ldap/docker-compose.yml`
- Same authentik-ldap container, bound to 100.123.254.52:3389 (Tailscale) + 127.0.0.1:3389
**auth-ldap.phar patches (3 patches applied, original backed up as auth-ldap.phar.orig):**
1. `authentication.php` - `getConnection()`: adds binddn/bindpw from plugin config to
Net_LDAP2 params so initial connect uses credentials (not anonymous, which Authentik rejects)
2. `config.php` - validation block: sets include_path to phar's include dir before
`require_once Net/LDAP2.php` so sub-files resolve correctly in FPM context
3. ALL `include/Net/LDAP2/*.php` files: guards `require_once 'PEAR.php'` with
`if (!class_exists('PEAR', false))` to prevent fatal conflict between osTicket's
`/include/pear/PEAR.php` and PHP global `/usr/local/lib/php/PEAR.php`
**osTicket LDAP plugin config (namespace plugin.2 in ost_config):**
- servers: `authentik-ldap-proxy` (via socat on port 389)
- bind_dn: `cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io`
- bind_pw: encrypted with `Crypto::encrypt(pass, SECRET_SALT, 'plugin.2')`
- search_base: `ou=users,dc=ldap,dc=goauthentik,dc=io`
- schema: auto, auth-staff: 1, auth-client: 0, domain: ldap.goauthentik.io
**Staff login:** username=`kenpat7177`, password=Authentik password (reset to `KiteStacks2026!`)
on `tasks.kitestacks.com/scp/login.php`
### Per Scholas IT Support Capstone - IN PROGRESS
See [[project-per-scholas-capstone]]. Next steps:
- Create capstone incident tickets in osTicket (5-phase challenge)
- Set up osTicket user/client portal for non-staff users (Phase 3 end-user access)
- Each capstone ticket maps to a phase scenario (migration event, incident response, etc.)
## 2026-06-15: Uptime Kuma -> ntfy phone notifications configured
- ntfy is running at `https://ntfy.kitestacks.com` with deny-by-default auth.
- Existing ntfy user `alerts` has write-only access to all topics and now has a
dedicated access token for Uptime Kuma. Do not write the token into repo/memory.
- Phone subscription topic: `kitestacks-uptime-VHr1DMQNi8`.
- Anonymous/everyone access is read-only for that single topic, so phones can
subscribe without needing the Kuma publish token.
- Uptime Kuma notification row:
- name `KiteStacks ntfy`
- type `ntfy`
- server URL `http://ntfy` on the shared Docker network
- topic `kitestacks-uptime-VHr1DMQNi8`
- auth method `accessToken`
- active/default, linked to active monitors `T14 Deb Assassin` and `Google DNS`
- Verified by sending a Kuma-provider test message and reading it publicly via
`https://ntfy.kitestacks.com/kitestacks-uptime-VHr1DMQNi8/json?poll=1`.
- To add to a phone home screen: open
`https://ntfy.kitestacks.com/kitestacks-uptime-VHr1DMQNi8` on the phone, allow
notifications, then use the browser share/menu action to add ntfy to the home
screen.

24
project-kitestacks-services.md
View file

@ -30,22 +30,28 @@ authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy, bookstack, bo
- OSTicket SMTP configured: smtp.gmail.com:587, kitestacks.helpdesk@gmail.com, app password stored in ost_email table (smtp_auth_creds=1 for all 3 emails) - OSTicket SMTP configured: smtp.gmail.com:587, kitestacks.helpdesk@gmail.com, app password stored in ost_email table (smtp_auth_creds=1 for all 3 emails)
- portainer.kitestacks.com CF tunnel hostname — user confirmed already set in CF dashboard - portainer.kitestacks.com CF tunnel hostname — user confirmed already set in CF dashboard
## Completed 2026-06-18 (this session) ## Completed 2026-06-18/19 (this session)
- Forgejo repo reorganization: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem merged as subdirs (cloud/, cloud-migration/, autosync/, osticket/) into kitestacks-homelab repo. Committed and pushed. - Forgejo repo reorganization: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem merged as subdirs (cloud/, cloud-migration/, autosync/, osticket/) into kitestacks-homelab repo. Committed and pushed.
- comptia-a-plus-core2 Forgejo repo updated: merged study-tracker content, added certifications/ dir, updated exam goal to July 7. - comptia-a-plus-core2 Forgejo repo updated: merged study-tracker content, added certifications/ dir, updated exam goal to July 7.
- homelab-mastery Forgejo repo: architecture/overview.md and build-guide/README.md rewritten in plain English. - homelab-mastery Forgejo repo: architecture/overview.md and build-guide/README.md rewritten in plain English.
- RUNBOOK.md + DEBUG-DOCUMENTATION.md: rewritten in 5th-grade plain English in kitestacks-homelab repo. - RUNBOOK.md + DEBUG-DOCUMENTATION.md: rewritten in 5th-grade plain English in kitestacks-homelab repo.
- All 6 BookStack pages updated via API (Runbook, Debug, Architecture, Build Guide, AI Guide, Manual Guide). - All 6 BookStack pages updated via API (Runbook, Debug, Architecture, Build Guide, AI Guide, Manual Guide).
- Forgejo API token via external URL broken (Cloudflare strips Authorization header). Works via localhost:3006. - BookStack API token created (claude-push-825981) via DB injection + bcrypt hash.
- BookStack API token created (claude-push-825981) via DB injection + bcrypt hash. Works internally. - **monk Forgejo migrated to shared PostgreSQL (2026-06-19):** Used `forgejo dump --database postgres` to generate clean SQL, dropped pgloader schema, reloaded. Both Cloudflare connectors now return 200 for API token `[redacted]`. Monk reads from `authentik-postgres` at `100.x.x.x:5432`, DB name `forgejo`, user `forgejo`.
- SQLite backup at: `~/kitestacks-live/docker/forgejo/data/gitea/gitea.db.backup-20260618-230715`
## Completed 2026-06-19
- kscloud1 SSH restored: key added to /home/kenpat/.ssh/authorized_keys (user is `kenpat`, not `kenpatmonk`). SSH via `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.x.x.x`.
- kscloud1 Forgejo migrated to shared PostgreSQL: compose at /opt/kitestacks/docker/forgejo/docker-compose.yml, joined authentik_default network to reach authentik-postgres:5432. 20/20 external API requests pass.
- BookStack kscloud1 OIDC: already configured, cache perms fixed, OIDC login redirects to auth.kitestacks.com correctly.
## Confirmed working
- OSTicket SMTP (smtp.gmail.com:587, kitestacks.helpdesk@gmail.com) — confirmed 2026-06-19.
- Portainer Authentik OAuth SSO — confirmed working 2026-06-19.
- Old standalone repos archived 2026-06-19: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem.
## Pending ## Pending
- BookStack kscloud1: update compose (OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/, OIDC_ISSUER_DISCOVER=true), restart, fix cache perms (chown -R abc:users /config/www/framework/cache/). Blocked by kscloud1 SSH key needs re-adding. - (none)
- kscloud1 SSH: re-add id_ed25519_kscloud1.pub key via Hetzner VNC console.
- OSTicket SMTP test email — verify delivery works.
- Archive/delete now-redundant standalone repos (kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem) once user confirms move is good.
- Portainer Authentik OAuth setup on both Portainer UIs (manual user action).
- Forgejo Authorization header fix: investigate why Cloudflare strips the token header for API calls (may need Cloudflare WAF rule or different auth method).
## Completed 2026-06-18 ## Completed 2026-06-18
- Portainer OAuth: both monk + kscloud1 configured (AuthenticationMethod=3, Authentik SSO). OAuth user kenpat7177@gmail.com pre-created as Role:1 (admin) on both. Local Docker environment added to both. Portal card already live. - Portainer OAuth: both monk + kscloud1 configured (AuthenticationMethod=3, Authentik SSO). OAuth user kenpat7177@gmail.com pre-created as Role:1 (admin) on both. Local Docker environment added to both. Portal card already live.