memory: sync session state 2026-06-19 (redacted for Forgejo)
- MEMORY.md: current index with latest status entries - project-kitestacks-migration.md: full updated history — kscloud1 SSH restored, Forgejo+BookStack SSO fixed, 2-connector active-active confirmed - project-kitestacks-services.md: monk Forgejo on shared PG, OSTicket SMTP live, no pending items - project-a-plus-core2.md: quiz log updated with OS-1/OS-2 results IPs, passwords, and API tokens redacted per Forgejo security policy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
92cb104838
commit
c780c8c97b
4 changed files with 80 additions and 497 deletions
|
|
@ -1,4 +1,4 @@
|
|||
- [KiteStacks homelab — 2-connector active-active CF Tunnel](project-kitestacks-migration.md) — monk (Docker cloudflared) + kscloud1 (5.78.233.28, Hetzner Ubuntu 26.04). 2026-06-16: phantom 3rd replica fixed (disabled native cloudflared systemd on monk), failover verified. kscloud1 SSH key needs re-adding. Oracle Cloud ARM VPS migration planned.
|
||||
- [KiteStacks homelab — 2-connector active-active CF Tunnel](project-kitestacks-migration.md) — monk (Docker cloudflared) + kscloud1 (5.78.x.x, Hetzner Ubuntu 26.04). 2026-06-19: SSH restored (user=kenpat, sudo pw=[redacted]), Forgejo+BookStack SSO fixed on kscloud1. Oracle Cloud ARM VPS migration planned.
|
||||
- [Forgejo doc redaction rule](feedback-forgejo-redaction.md) — always redact IPs, ports, and passwords in any homelab Forgejo repo files before committing.
|
||||
- [A+ Core 2 study plan](project-a-plus-core2.md) — exam goal July 7 2026, hard deadline July 12. SEC-6 + OS-1/OS-2 quizzes June 18, LAB 5 & 6 Friday.
|
||||
- [KiteStacks service inventory](project-kitestacks-services.md) — 2026-06-18: repos merged into kitestacks-homelab, docs rewritten, BookStack+Forgejo updated. Pending: kscloud1 SSH, OSTicket SMTP test, Portainer OAuth.
|
||||
- [KiteStacks service inventory](project-kitestacks-services.md) — 2026-06-19: monk Forgejo on shared PostgreSQL (authentik-postgres:5432). kscloud1 Forgejo pending SSH restore. OSTicket SMTP live. Pending: kscloud1 Forgejo PG + BookStack SSO.
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ metadata:
|
|||
| 2026-06-11 | Started Core 2 study, 9:15 PM | Took Sybex diagnostic practice exam — scored 50% (50/100) |
|
||||
| 2026-06-17 | LAB 5 theory (Malware types & removal procedure), LAB 6 theory (BitLocker, UAC, Firewall, EFS) | 1hr 8min session, 9:57–11:06 PM CDT. Checkpoint answers all correct. Hands-on LAB 5 & 6 scheduled Friday. SEC-6 quiz scheduled for June 18. |
|
||||
| 2026-06-18 | SEC-6 quiz + OS-1/OS-2 quizzes scheduled | Hands-on LAB 5 & 6 scheduled for this Friday. |
|
||||
| 2026-06-19 | OS-1/OS-2 quiz (Windows Editions, Install & Boot, Troubleshooting) — taken cold, before studying | **10/12 correct + Scenario A.** Part 1 (Windows Editions): 6/7 — missed Q1 (Home can't join domain, Pro is minimum). Part 2 (Install & Boot): 5/5 perfect. Part 3 (Troubleshooting): 1/2 — missed Scenario B ("OS Not Found" = `bootrec /rebuildbcd`, not `chkdsk`). |
|
||||
|
||||
## Planned Tests
|
||||
- **This week (started 2026-06-11):** Professor Messer practice exam (diagnostic — taken cold first)
|
||||
|
|
@ -37,9 +38,12 @@ metadata:
|
|||
- Day 16: Weak area review only
|
||||
- Day 17 (June 28): Exam
|
||||
|
||||
## Key Weak Areas to Watch (common for homelab/Linux users)
|
||||
- Windows command line tools (sfc, DISM, chkdsk, bootrec, diskpart)
|
||||
## Key Weak Areas to Watch
|
||||
- **`bootrec` vs `chkdsk`**: "OS Not Found" = boot sector problem → `bootrec /rebuildbcd` (or `/fixmbr`, `/fixboot`). `chkdsk` is for file system corruption, not missing bootloader.
|
||||
- **Windows edition domain join**: Home cannot join AD domain or use Group Policy. Pro is minimum. Memorize: Home = no domain, no GPO, no BitLocker, no RDP host.
|
||||
- **Windows 11 upgrade requirements**: TPM 2.0 + Secure Boot (UEFI required, not legacy BIOS).
|
||||
- Malware types and removal procedures (pure memorization)
|
||||
- Windows command line tools generally (sfc, DISM, chkdsk, bootrec, diskpart)
|
||||
|
||||
## Resources
|
||||
- Professor Messer A+ Core 2 (YouTube + paid practice exams)
|
||||
|
|
|
|||
|
|
@ -1,43 +1,27 @@
|
|||
---
|
||||
name: project-kitestacks-migration
|
||||
description: "Migration of the live KiteStacks homelab/website from assassin (T14) to monk — COMPLETE. Plus full Hetzner cloud failover (kscloud1, 5.78.233.28) — COMPLETE. All 9 subdomains can run from any single host. Plus 2026-06-10 portal/SSO push: portal FluxCD+coming-soon changes deployed, Karakeep SSO fixed, OpenProject SSO blocked by EE license, Portainer SSO Authentik-side done (pending user manual steps)."
|
||||
description: "KiteStacks homelab — kscloud1 (Hetzner, 5.78.x.x) is PRODUCTION. monk is DEVELOPMENT. 2-connector CF Tunnel. Samurai desktop = planned 3rd failover. Oracle VPS migration planned to replace kscloud1."
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 33992890-3940-4d4a-a94a-22b5621e9c1a
|
||||
---
|
||||
|
||||
## Final Polish, Security, and Runbook Completion (2026-06-15)
|
||||
## ARCHITECTURE (canonical — 2026-06-19)
|
||||
|
||||
The KiteStacks infrastructure is now in its final, secured, and documented state:
|
||||
- **GitOps UI/Dashboard:** Added a standalone Nginx container for FluxCD status, bypassing Authentik so Cloudflare edge can route it freely. The dashboard is live at `flux.kitestacks.com`.
|
||||
- **Security Posture:** Validated Zero Trust architecture. No inbound open ports, strict mesh networking via Tailscale `100.x.x.x`, and Authentik protecting all administrative dashboards (`/scp/` for osTicket, Portainer, Grafana, Kite AI).
|
||||
- **Runbook Cleaned:** `RUNBOOK.md` truncated and organized. Historical issues (like Authentik invalid_grant, osTicket email SMTP lack of MTA) have been relocated to `docs/DEBUGGING.md`.
|
||||
- **osTicket Diagnostics:** Documented that activation emails fail because Docker containers lack a local MTA. Fix involves adding an external SMTP server in the osTicket Admin Panel.
|
||||
- **Cloudflare Multi-Node Routing:** Diagnosed persistent 502 errors on new subdomains (like `ntfy`). Cloudflare Tunnels actively load balance between `monk` and `kscloud1`. Documented that all new services must be deployed to both nodes to prevent the load balancer from sending traffic to a missing container. Subsequently resolved the `ntfy` 502 error by deploying the container to the `kscloud1` replica and syncing its `user.db` via Tailscale SSH.
|
||||
**kscloud1 = PRODUCTION** (always-on source of truth for live site).
|
||||
**monk = DEVELOPMENT** (things are built/tested on monk, then pushed to kscloud1).
|
||||
**Samurai desktop = planned 3rd connector** (when up, keeps kscloud1 redundant; can be down sometimes).
|
||||
|
||||
## T14s GitOps Automation SUCCESS (2026-06-15)
|
||||
Changes flow: monk develops → monk PUSHES to kscloud1. Always monk → kscloud1, never the reverse.
|
||||
Monk is the source of all changes (code, config, stateful data). kscloud1 receives and stays live.
|
||||
CF Tunnel load-balances both connectors — kscloud1 always has the last pushed state. Monk going down = zero user impact since kscloud1 stays live with whatever was last pushed.
|
||||
|
||||
The cluster configuration originally for "assassin" (T14) has been moved to the
|
||||
**T14s**. The machine is now fully bootstrapped with FluxCD GitOps.
|
||||
## STATUS: MIGRATION + CLOUD FAILOVER COMPLETE (2026-06-10)
|
||||
|
||||
- **Cluster Hostname:** monk (T14s)
|
||||
- **GitOps Repo:** `kitestacks-homelab` (main branch)
|
||||
- **Path:** `clusters/T14s`
|
||||
- **Automation:** FluxCD is now managing the `kavita` namespace.
|
||||
- **Kavita Manifests:**
|
||||
- Deployment, Service, PVC (2Gi local-path), and Namespace.
|
||||
- Successfully synced and running (verified 2026-06-15).
|
||||
- **Credentials:** Authentik password for `kenpat7177` reset to `KiteStacks2026!`.
|
||||
- **osTicket:** Services started, DB unified on kscloud1, and verified
|
||||
accessible via Authentik LDAP.
|
||||
|
||||
The GitOps workflow is now the authoritative way to manage Kubernetes apps on the T14s.
|
||||
|
||||
monk is the live production host. assassin (T14) is OFF. kscloud1 (Hetzner VPS,
|
||||
5.78.233.28) is now a THIRD active Cloudflare Tunnel connector and runs a FULL
|
||||
replica of all 9 services, so the site stays up even if both monk and assassin
|
||||
are off (verified by user testing with home wifi off, from phone + mom's phone).
|
||||
assassin (T14) is OFF. kscloud1 (Hetzner VPS, 5.78.x.x) runs a FULL
|
||||
replica of all services, so the site stays up even if monk is off
|
||||
(verified by user testing with home wifi off, from phone + mom's phone).
|
||||
|
||||
All 9 public subdomains (www, ai, auth, gitforge, grafana, kavita, links, status, tasks)
|
||||
verified returning correct status codes via the live tunnel with kscloud1 in rotation.
|
||||
|
|
@ -53,8 +37,8 @@ EXPLICITLY ACCEPTED by user as the cost of guaranteed uptime. Fresh/separate
|
|||
databases on kscloud1 are fine; do not try to sync data between monk and kscloud1.
|
||||
|
||||
## kscloud1 access
|
||||
SSH: `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.233.28` (passwordless, key auth).
|
||||
sudo needs a password ("p12217177") and has no askpass helper - avoid sudo;
|
||||
SSH: `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.x.x` (passwordless, key auth).
|
||||
sudo needs a password ("[redacted]") and has no askpass helper - avoid sudo;
|
||||
most things doable as kenpat or via docker.
|
||||
All services live under `/opt/kitestacks/docker/<service>/docker-compose.yml`,
|
||||
same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`.
|
||||
|
|
@ -74,7 +58,7 @@ same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`.
|
|||
- kavita (alias `kavita`) - empty library (fresh)
|
||||
- karakeep + karakeep-chrome + karakeep-meilisearch (alias `karakeep`) - fresh meilisearch/db
|
||||
- authentik + authentik-worker + authentik-postgres + authentik-redis (alias on `auth`) - FRESH DB.
|
||||
Bootstrap admin: `akadmin@kitestacks.com` / password `6KlYpfCyYxbnKQNiOewN` (set via
|
||||
Bootstrap admin: `akadmin@kitestacks.com` / password `[redacted]` (set via
|
||||
AUTHENTIK_BOOTSTRAP_PASSWORD in .env). No OAuth provider apps exist yet (would need to be
|
||||
manually recreated in authentik UI for grafana/openwebui/karakeep/openproject SSO to work
|
||||
when kscloud1 is the active backend).
|
||||
|
|
@ -86,7 +70,7 @@ same one-dir-per-app pattern as monk's `~/kitestacks-live/docker/`.
|
|||
|
||||
## monk-side changes made for cross-host monitoring
|
||||
- `~/kitestacks-live/docker/prometheus/prometheus.yml`: added scrape job
|
||||
`kscloud1-node` -> `5.78.233.28:9100` (kscloud1's node-exporter is exposed
|
||||
`kscloud1-node` -> `5.78.x.x:9100` (kscloud1's node-exporter is exposed
|
||||
0.0.0.0:9100, no firewall - reachable from monk's public IP). monk's grafana
|
||||
(the live one, "Node Exporter Full" dashboard now provisioned via
|
||||
`~/kitestacks-live/docker/grafana/provisioning/`) shows BOTH `t14-node`
|
||||
|
|
@ -163,16 +147,16 @@ because the data is created fresh on every login attempt.
|
|||
FIX: Converted to a single shared Postgres+Redis (HA pattern), hosted on
|
||||
kscloud1, reachable ONLY over Tailscale:
|
||||
- Installed Tailscale on both monk and kscloud1 (same tailnet). kscloud1's
|
||||
tailscale IP is `100.123.254.52`.
|
||||
tailscale IP is `100.x.x.x`.
|
||||
- kscloud1's `/opt/kitestacks/docker/authentik/docker-compose.yml`:
|
||||
authentik-postgres now binds `100.123.254.52:5432:5432` (was unbound/internal-only),
|
||||
authentik-redis now binds `100.123.254.52:6379:6379`. Both still also reachable
|
||||
authentik-postgres now binds `100.x.x.x:5432:5432` (was unbound/internal-only),
|
||||
authentik-redis now binds `100.x.x.x:6379:6379`. Both still also reachable
|
||||
on the local `kitestacks` docker network for kscloud1's own authentik+worker.
|
||||
Backup of pre-change file: `docker-compose.yml.backup-before-shared-db-20260610-1138`.
|
||||
- monk's `~/kitestacks-live/docker/authentik/docker-compose.yml`: REMOVED the
|
||||
`postgresql` and `redis` services entirely. monk's `authentik`/`authentik-worker`
|
||||
now point `AUTHENTIK_POSTGRESQL__HOST` and `AUTHENTIK_REDIS__HOST` at
|
||||
`100.123.254.52` (kscloud1 over Tailscale), using the same `PG_PASS` /
|
||||
`100.x.x.x` (kscloud1 over Tailscale), using the same `PG_PASS` /
|
||||
`AUTHENTIK_SECRET_KEY` as before (already identical between hosts).
|
||||
- monk's old local `authentik-postgres`/`authentik-redis` containers were
|
||||
STOPPED (not removed) - data dirs preserved under
|
||||
|
|
@ -213,7 +197,7 @@ cleared, Enabled->false) - confirmed twice, even with a full WAL-consistent
|
|||
kavita.db replace from monk. Direct DB writes to this table do NOT survive a
|
||||
restart; only saves through Kavita's own Settings UI/API persist correctly.
|
||||
FIX: opened an SSH local port-forward (`ssh -L 5099:localhost:5000
|
||||
kenpat@5.78.233.28`) so the user could reach kscloud1's Kavita directly at
|
||||
kenpat@5.78.x.x`) so the user could reach kscloud1's Kavita directly at
|
||||
http://localhost:5099 (bypassing the Cloudflare load-balanced domain), logged
|
||||
in with their normal kenpat7177 Kavita password, and re-entered the OIDC
|
||||
config in Settings -> OIDC:
|
||||
|
|
@ -255,9 +239,9 @@ are added on monk later, they won't appear on kscloud1 unless re-synced
|
|||
(covers/ dir + kavita.db + actual book files under library/books, none of
|
||||
which exist on kscloud1 per the earlier "stale data" note).
|
||||
SECURITY NOTE: postgres/redis on kscloud1 are bound to the Tailscale interface
|
||||
IP only (100.123.254.52), not 0.0.0.0 - not exposed to the public internet.
|
||||
IP only (100.x.x.x), not 0.0.0.0 - not exposed to the public internet.
|
||||
ROLLBACK: if Tailscale connectivity ever breaks, monk's authentik will fail to
|
||||
start (can't reach 100.123.254.52). To roll back: restore monk's
|
||||
start (can't reach 100.x.x.x). To roll back: restore monk's
|
||||
docker-compose.yml from git/backup to use local postgresql/redis services
|
||||
again, restart monk's old authentik-postgres/authentik-redis containers
|
||||
(`docker start authentik-postgres authentik-redis` in
|
||||
|
|
@ -314,7 +298,7 @@ OAuth callback path is `/api/auth/callback/custom`, but Authentik's Karakeep
|
|||
OAuth2Provider's `_redirect_uris` had the wrong path -> "Redirect URI Error".
|
||||
FIX: direct Postgres UPDATE to
|
||||
`authentik_providers_oauth2_oauth2provider._redirect_uris` (JSON column) on
|
||||
the shared kscloud1 authentik-postgres (100.123.254.52), wrapped in explicit
|
||||
the shared kscloud1 authentik-postgres (100.x.x.x), wrapped in explicit
|
||||
`BEGIN; UPDATE ...; COMMIT;` (a bare single-statement -c "UPDATE..." reported
|
||||
"UPDATE 1" but did NOT persist on first attempt - cause unclear, explicit
|
||||
transaction fixed it). After the DB write, restarted authentik+authentik-worker
|
||||
|
|
@ -363,7 +347,7 @@ the `homelab-admin` Authentik group).
|
|||
Created via `docker exec authentik ak shell` (Django ORM, no Authentik API
|
||||
token configured) on kscloud1's shared authentik-postgres:
|
||||
- OAuth2Provider "Portainer": client_id=`portainer`,
|
||||
client_secret=`wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF`,
|
||||
client_secret=`[redacted]`,
|
||||
provider_id=9, redirect_uri=`https://portainer.kitestacks.com` (strict),
|
||||
scopes openid/email/profile, sub_mode=user_email, signing key + flows copied
|
||||
from existing providers (same pattern as Karakeep/Grafana).
|
||||
|
|
@ -384,7 +368,7 @@ still returns `000` as of 2026-06-10):
|
|||
2. In Portainer -> Settings -> Authentication -> OAuth (Provider: Custom), on
|
||||
BOTH monk's and kscloud1's SEPARATE Portainer instances, configure:
|
||||
- Client ID: `portainer`
|
||||
- Client Secret: `wTim3mrMwt34ko1RYMvK1RNnjwWOMi_d4r4cS6exr7DjozCrL5zKthHl-5KjargF`
|
||||
- Client Secret: `[redacted]`
|
||||
- Authorization URL: `https://auth.kitestacks.com/application/o/authorize/`
|
||||
- Access Token URL: `https://auth.kitestacks.com/application/o/token/`
|
||||
- Resource/Userinfo URL: `https://auth.kitestacks.com/application/o/userinfo/`
|
||||
|
|
@ -403,259 +387,12 @@ above. Prometheus + Uptime Kuma: DEFERRED - neither has native OAuth, need a
|
|||
forward-auth proxy (oauth2-proxy or Authentik embedded outpost) - deferred per
|
||||
user's "ok lets do smaller app level" (hold new infra until Oracle VPS decided).
|
||||
Cloudflare itself: no SSO concept applicable (it's Cloudflare's own dashboard
|
||||
managed outside the lab login) - was always about the portal's Cloudflare card
|
||||
placement, see "Portal UI changes" note above.
|
||||
|
||||
### Uptime Kuma + Authentik SSO resumed on monk (2026-06-15)
|
||||
User confirmed the next task is setting up Uptime Kuma with Authentik SSO in
|
||||
the main KiteStacks lab, and explicitly requested saving progress to
|
||||
`~/claude-memory` and pushing to the Forgejo `kenpat/claude-memory` repo as we
|
||||
go.
|
||||
|
||||
Verified current live state on monk before making changes:
|
||||
- `uptime-kuma` container is running and healthy, published on host port
|
||||
`3001`, image `louislam/uptime-kuma:latest`.
|
||||
- Installed Uptime Kuma version inside the container is `1.23.17`.
|
||||
- Uptime Kuma compose file is
|
||||
`~/kitestacks-live/docker/uptime-kuma/docker-compose.yml`, using external
|
||||
Docker volume `uptime-kuma:/app/data` and networks `default` + external
|
||||
`kitestacks`.
|
||||
- Uptime Kuma SQLite DB path inside container is `/app/data/kuma.db`; tables
|
||||
include `user`, `setting`, `monitor`, `heartbeat`, `status_page`,
|
||||
`notification`, `api_key`, and related monitor/status tables. No obvious
|
||||
native OAuth/OIDC tables were present in the initial schema list.
|
||||
- Grafana is already configured for Authentik generic OAuth in
|
||||
`~/kitestacks-live/docker/grafana/docker-compose.yml` with Authentik public
|
||||
authorize URL and internal token/userinfo URLs.
|
||||
- `authentik` is healthy; `authentik-worker` currently shows unhealthy in
|
||||
`docker ps` even though it has been running for ~35h. Check logs/health
|
||||
before relying on new Authentik-side automation.
|
||||
- Existing Authentik objects were found for Uptime Kuma:
|
||||
- Application slug `uptime-kuma`, name `Uptime Kuma`, provider id `7`.
|
||||
- ProxyProvider `Uptime Kuma`, external host `https://status.kitestacks.com`,
|
||||
internal host `http://uptime-kuma:3001`, mode `proxy`.
|
||||
- Embedded proxy outpost already includes providers `Karakeep`,
|
||||
`Uptime Kuma`, and `LiteLLM`.
|
||||
- `https://status.kitestacks.com` still routes directly to Kuma as of
|
||||
2026-06-15: public curl gets Kuma's `/dashboard` redirect and 200 response,
|
||||
not an Authentik authorization flow. Cloudflare tunnel route still needs to
|
||||
be changed from direct Kuma to the Authentik embedded outpost/server.
|
||||
- Security fix applied 2026-06-15: created PolicyBinding
|
||||
`6f2ac876-2f47-473d-986d-d7c5d2a3214e` from the Uptime Kuma application to
|
||||
Authentik group `homelab-admin`, enabled, order 0. This matches the Portainer
|
||||
restriction pattern.
|
||||
- Cloudflared is remote-managed: container command is `tunnel --no-autoupdate
|
||||
run`, no local ingress config exists, and the compose file stores a
|
||||
`TUNNEL_TOKEN`. Do not print that token; treat it as sensitive. Routing
|
||||
changes must be made through Cloudflare's tunnel API/dashboard unless a
|
||||
suitable Cloudflare API token is available locally.
|
||||
- Local validation after the Authentik binding: `curl -I -H 'Host:
|
||||
status.kitestacks.com' http://localhost:9001` returns `302` to
|
||||
`https://status.kitestacks.com/outpost.goauthentik.io/start?...`, proving
|
||||
the embedded outpost/proxy provider works when traffic reaches Authentik.
|
||||
- No suitable Cloudflare API token was found during the local search; only the
|
||||
cloudflared connector tunnel token is present. Remaining blocker is changing
|
||||
the Cloudflare Tunnel public hostname for `status.kitestacks.com` from
|
||||
`http://uptime-kuma:3001` to `http://authentik:9000` (or equivalent
|
||||
Authentik service target in the Tunnel UI).
|
||||
- Correction after user tested: user does NOT want front-door proxy behavior
|
||||
for Uptime Kuma. Desired UX is an in-app "single sign on" button on the
|
||||
Uptime Kuma login screen, like Grafana/Forgejo style native OAuth. Authentik
|
||||
proxy redirect is not acceptable for this requirement.
|
||||
- Confirmed in the installed Uptime Kuma 1.23.17 frontend:
|
||||
`/app/src/components/Login.vue` only renders username, password, remember-me,
|
||||
and login submit controls. No native OAuth/OIDC/SSO button exists in this
|
||||
version's login component, and local source search only found monitor OAuth
|
||||
client-credentials support, not app login SSO.
|
||||
- If staying on Uptime Kuma 1.23.17, revert Cloudflare route for
|
||||
`status.kitestacks.com` back to `http://uptime-kuma:3001`; otherwise users
|
||||
get Authentik first and then still see Kuma's local login. Native in-app SSO
|
||||
would require an Uptime Kuma version/plugin/fork with login OIDC support or
|
||||
custom app code, not the Authentik proxy provider.
|
||||
- User reset the Cloudflare route back to `http://uptime-kuma:3001` and asked
|
||||
to continue with an in-app Authentik button. Upstream latest checked via
|
||||
GitHub API: Uptime Kuma latest release is `2.4.0` (published 2026-05-31) and
|
||||
upstream `src/components/Login.vue` still has only username/password login,
|
||||
no native OAuth/OIDC button. Proceeded with a custom overlay patch.
|
||||
- Custom native Authentik SSO overlay deployed on BOTH active tunnel backends
|
||||
(monk and kscloud1) so public load-balanced traffic behaves consistently:
|
||||
- monk path: `~/kitestacks-live/docker/uptime-kuma/`
|
||||
- kscloud1 path: `/opt/kitestacks/docker/uptime-kuma/`
|
||||
- backend preload module:
|
||||
`custom/server/authentik-sso.js`
|
||||
- frontend mounted files:
|
||||
`custom/dist/index.html`, `index.html.gz`, `index.html.br`
|
||||
- compose now sets `NODE_OPTIONS=--require /app/custom/server/authentik-sso.js`,
|
||||
loads `.env.sso`, and bind-mounts the custom files over Kuma's built HTML.
|
||||
- Authentik native OAuth provider/application created:
|
||||
- OAuth2Provider name `Uptime Kuma Native`, provider id `12`
|
||||
- Application slug `uptime-kuma-native`, name `Uptime Kuma Native SSO`
|
||||
- Client ID `uptime-kuma-native`
|
||||
- Redirect URI `https://status.kitestacks.com/auth/authentik/callback`
|
||||
- Restricted to Authentik group `homelab-admin` via PolicyBinding
|
||||
`2e1eaa95-b397-4c4f-bfc7-abb337906cf3`
|
||||
- Client secret is stored only in each host's `.env.sso`; do not print it.
|
||||
- Custom flow behavior:
|
||||
- Login page injects a `Sign in with Authentik` button linking to
|
||||
`/auth/authentik`.
|
||||
- Backend starts Authentik OIDC, validates callback state, fetches userinfo,
|
||||
maps the login to existing Kuma user `kenpat`, issues Kuma's normal JWT,
|
||||
then redirects to `/?authentik_token=<token>`.
|
||||
- Frontend one-time script stores the JWT in `localStorage.token`, removes
|
||||
the URL token, and redirects to `/dashboard`, letting Kuma's normal
|
||||
`loginByToken` flow establish the session.
|
||||
- Verification 2026-06-15:
|
||||
- monk local `/dashboard` HTML contains `Sign in with Authentik`,
|
||||
`/auth/authentik`, and `authentik_token`.
|
||||
- kscloud1 local `/dashboard` HTML contains the same and `/auth/authentik`
|
||||
redirects to Authentik with client_id `uptime-kuma-native`.
|
||||
- Public repeated check:
|
||||
`for i in 1 2 3 4 5 6; do curl -sSL --compressed https://status.kitestacks.com/dashboard | grep -q "Sign in with Authentik"; done`
|
||||
returned `button` for all 6 attempts, confirming both active connectors
|
||||
serve the button.
|
||||
- Post-test screenshot showed Uptime Kuma login page with red banner "Lost
|
||||
connection to the socket server. Reconnecting..." after clicking the SSO
|
||||
button. Root cause: active-active JWT mismatch. Uptime Kuma JWTs include a
|
||||
signature using `setting.jwtSecret`; monk and kscloud1 had matching user
|
||||
password hashes but different JWT secrets, so a token minted by one backend
|
||||
failed if the browser's websocket connected to the other backend. Fixed
|
||||
2026-06-15 by copying monk's exact `jwtSecret` into kscloud1's
|
||||
`/app/data/kuma.db` using base64 transport (avoid shell expansion of secret
|
||||
chars), then restarting kscloud1 Uptime Kuma. Verified both hashes now match:
|
||||
`jwtSecret` length 60, sha3 prefix `FA67E6E9EDCC8E1D`. Public button check
|
||||
still returns `button` 6/6. If a browser still has a pre-fix bad token in
|
||||
localStorage, clear site data or click the Authentik button again to mint a
|
||||
fresh token.
|
||||
- User retested and still saw the socket reconnect banner. Follow-up finding:
|
||||
public Uptime Kuma frontend was using Socket.IO's default long-polling-first
|
||||
transport. In the active-active Cloudflare Tunnel setup, polling requests can
|
||||
bounce between monk and kscloud1 before a socket session is established,
|
||||
causing reconnect loops before Kuma even logs `Login by token`.
|
||||
- Fix applied 2026-06-15 on BOTH monk and kscloud1: copied the built frontend
|
||||
bundle `index-BBxTfFCS.js` into the overlay and patched the minified socket
|
||||
call from `Ze=Nc(n)` to `Ze=Nc(n,{transports:["websocket"]})`. Regenerated
|
||||
`.gz` and `.br` variants and mounted all three over
|
||||
`/app/dist/assets/index-BBxTfFCS.js*` in both compose files. Restarted both
|
||||
Uptime Kuma containers.
|
||||
- Verification after websocket-only patch:
|
||||
- monk local asset contains `transports:["websocket"]`
|
||||
- kscloud1 local asset contains `transports:["websocket"]`
|
||||
- public repeated asset check over `https://status.kitestacks.com/assets/index-BBxTfFCS.js`
|
||||
found `transports:["websocket"]` 6/6, confirming both tunnel backends serve
|
||||
the patched client bundle.
|
||||
- User still saw the same issue after trying another browser. Follow-up:
|
||||
websocket connections were reaching Kuma, but logs showed no `Login by token`,
|
||||
so the handoff from Authentik callback to Kuma storage was unreliable. Changed
|
||||
the SSO callback from `/?authentik_token=<jwt>` URL handoff to a short-lived
|
||||
readable cookie `uk_authentik_token` plus redirect directly to `/dashboard`.
|
||||
Updated injected HTML to read that cookie before Kuma initializes, store the
|
||||
token in `localStorage.token`, set `localStorage.remember=1`, then delete the
|
||||
cookie. This avoids long-token URL handling.
|
||||
- Important operational gotcha: Uptime Kuma caches `index.html` in memory at
|
||||
startup. After changing the mounted `index.html`/compressed variants, `docker
|
||||
compose up -d` was not enough because containers stayed "Running"; had to run
|
||||
`docker compose restart uptime-kuma` on BOTH monk and kscloud1 to reload the
|
||||
HTML into memory.
|
||||
- Verification after cookie handoff + explicit restarts:
|
||||
- monk local `/dashboard` HTML contains `uk_authentik_token`, `authentik_token`,
|
||||
and `Sign in with Authentik`.
|
||||
- kscloud1 local `/dashboard` HTML contains the same.
|
||||
- public repeated check for `uk_authentik_token` over
|
||||
`https://status.kitestacks.com/dashboard` returned `cookie-handoff` 6/6.
|
||||
- User confirmed after retest: Uptime Kuma Authentik SSO button works.
|
||||
|
||||
### Uptime Kuma monitors mirrored into Prometheus/Grafana (2026-06-15)
|
||||
User asked to set up the same monitors currently in Uptime Kuma for Grafana and
|
||||
Prometheus. Existing Uptime Kuma monitor list at the time:
|
||||
- `T14 Deb Assassin`: ping `127.0.0.1`
|
||||
- `HomeRouter`: ping `192.168.1.254`
|
||||
- `Google DNS`: ping `8.8.8.8`
|
||||
- `TailScale`: ping `100.90.13.55`
|
||||
|
||||
Implemented on monk's live Prometheus/Grafana stack:
|
||||
- Added `prom/blackbox-exporter` service to
|
||||
`~/kitestacks-live/docker/prometheus/docker-compose.yml`.
|
||||
- Added blackbox config
|
||||
`~/kitestacks-live/docker/prometheus/blackbox.yml` with ICMP module
|
||||
(`preferred_ip_protocol: ip4`, timeout 5s).
|
||||
- Added Prometheus scrape job `uptime-kuma-ping-probes` in
|
||||
`~/kitestacks-live/docker/prometheus/prometheus.yml`, using `/probe` with
|
||||
`module=icmp` and labels `monitor_name` matching the Uptime Kuma names.
|
||||
- Added Grafana provisioned dashboard
|
||||
`~/kitestacks-live/docker/grafana/provisioning/dashboards/kitestacks-uptime-probes.json`
|
||||
titled `KiteStacks Uptime Probes`, with stat/timeseries panels for
|
||||
`probe_success{job="uptime-kuma-ping-probes"}` and
|
||||
`probe_duration_seconds{job="uptime-kuma-ping-probes"}`.
|
||||
- Ran `docker compose up -d` in the Prometheus directory, pulled/started
|
||||
`blackbox-exporter`, restarted Prometheus, and restarted Grafana.
|
||||
|
||||
Verification:
|
||||
- Prometheus config validates with `promtool check config`.
|
||||
- Prometheus active targets include all four `uptime-kuma-ping-probes`.
|
||||
- Query result for `probe_success{job="uptime-kuma-ping-probes"}`:
|
||||
`Google DNS=1`, `T14 Deb Assassin=1`, `HomeRouter=0`, `TailScale=0`.
|
||||
The two failures match Kuma's existing failing ping behavior from inside the
|
||||
container/network namespace.
|
||||
- Grafana logs show dashboard provisioning completed without dashboard errors
|
||||
(only unrelated bundled plugin permission warnings).
|
||||
|
||||
### Desktop widget for the same monitor set (2026-06-15)
|
||||
User asked for a Rainmeter-like desktop widget on Debian 13 that can show the
|
||||
same Uptime Kuma monitor state in real time.
|
||||
|
||||
Created a local Conky-based widget scaffold in the desktop user's home:
|
||||
- `~/.local/bin/kitestacks-uptime-widget.sh`
|
||||
- `~/.config/conky/kitestacks-uptime.conf`
|
||||
|
||||
Behavior:
|
||||
- Polls Prometheus for `probe_success` and `probe_duration_seconds` from the
|
||||
`uptime-kuma-ping-probes` job.
|
||||
- Defaults to `http://192.168.1.205:9090`, with `PROM_URL`
|
||||
override support.
|
||||
- Prints the four Kuma monitor names, state, latency, and a summary line.
|
||||
- Degrades cleanly with `Prometheus unavailable at ...` when the endpoint
|
||||
cannot be reached.
|
||||
|
||||
Note: Conky is the closest direct Rainmeter-style equivalent for Debian/Linux
|
||||
desktop widgets; `eww` is the more modern alternative if the desktop session is
|
||||
Wayland-first and the user prefers GTK/Rust widgets instead of a classic
|
||||
desktop overlay.
|
||||
|
||||
Debian 13 package note:
|
||||
- `conky` is a virtual package in trixie.
|
||||
- Install `conky-all` for the full desktop widget experience:
|
||||
`sudo apt update && sudo apt install conky-all`
|
||||
|
||||
Connectivity note:
|
||||
- The laptop could not reach Prometheus at `192.168.1.205:9090`, which means
|
||||
the widget can only work from a host that can reach the homelab LAN or a
|
||||
public/tunneled Prometheus endpoint.
|
||||
- The existing KiteStacks docs mark Prometheus as excluded from the Cloudflare
|
||||
tunnel, so there is no known public Prometheus URL to target yet.
|
||||
- The desktop widget script now defaults to `https://prometheus.kitestacks.com`
|
||||
and can send `CF-Access-Client-Id` / `CF-Access-Client-Secret` headers if the
|
||||
hostname is protected by Cloudflare Access.
|
||||
|
||||
Cyberpunk widget styling:
|
||||
- Conky panel tuned to the wallpaper palette with black base and neon
|
||||
cyan/magenta accents.
|
||||
- Header uses `#ff4df0` pink and `#2de0ff` blue.
|
||||
- Monitor rows color-code `UP` as cyan and `DOWN` as pink for fast scanning.
|
||||
- `conky.text` now uses `execpi` so the helper's parsed color markup renders as
|
||||
one combined widget instead of only the title line.
|
||||
- The screenshot also showed a separate default Conky panel on the left; that
|
||||
is not part of the uptime widget itself.
|
||||
- Added a unified Conky desktop config at `~/.conkyrc` plus an autostart
|
||||
wrapper that kills stray Conky instances and launches the single combined
|
||||
panel.
|
||||
|
||||
Important security hygiene: local git remote for `~/claude-memory` contains an
|
||||
HTTP token in the URL; do not print it in summaries. Prefer redacted URLs in
|
||||
handoffs.
|
||||
login) - was always about the portal's Cloudflare card placement, see "Portal UI
|
||||
changes" note above.
|
||||
|
||||
### Oracle VPS migration - PLANNED, upcoming (stated 2026-06-11)
|
||||
User confirmed on 2026-06-11: "we are going to switch things soon from hetzner
|
||||
cloud to oracle soon." -> kscloud1 (Hetzner, 5.78.233.28) is intended to be
|
||||
cloud to oracle soon." -> kscloud1 (Hetzner, 5.78.x.x) is intended to be
|
||||
REPLACED by an Oracle Cloud VPS in the near future ("soon", no firm date yet).
|
||||
Originally raised 2026-06-10 as exploratory ("how easy would it be to move
|
||||
everything to oracle vps after?"), now an actual plan.
|
||||
|
|
@ -687,7 +424,7 @@ from `minutes=1` to `minutes=10` for ALL 9 OAuth2 providers in the shared Postgr
|
|||
DB. This gives enough buffer for monk's containers to start before codes expire.
|
||||
Command used (via python:3-alpine container):
|
||||
`docker run --rm --network host -v /tmp/fix_auth.py:/fix.py python:3-alpine sh -c ...`
|
||||
connecting to shared Postgres at 100.123.254.52.
|
||||
connecting to shared Postgres at 100.x.x.x.
|
||||
|
||||
### Karakeep redirect_uri reverted and re-fixed
|
||||
The Karakeep OAuth2Provider `_redirect_uris` had reverted back to the proxy pattern
|
||||
|
|
@ -709,196 +446,32 @@ STILL PENDING (user action in both Portainer UIs): configure OAuth (see prior no
|
|||
in "Portainer SSO" section above for exact credentials).
|
||||
Portal card update (3 files) also still pending until tunnel+OAuth done.
|
||||
|
||||
## 2026-06-16: Cloudflare Tunnel cleanup + failover verified
|
||||
|
||||
### Phantom 3rd replica — found and fixed
|
||||
CF dashboard showed 3 active replicas (expected: 2). Cause: a native `cloudflared`
|
||||
systemd service (`/usr/bin/cloudflared`, v2026.6.4) was running on monk alongside the
|
||||
Docker container (`/usr/local/bin/cloudflared`, v2026.6.0). Both connected to CF with
|
||||
the same TUNNEL_TOKEN, registering as separate connectors. Fixed:
|
||||
```bash
|
||||
sudo systemctl stop cloudflared
|
||||
sudo systemctl disable cloudflared
|
||||
```
|
||||
CF dashboard now shows 2 replicas: monk (Docker) + kscloud1. Architecture is
|
||||
**2-connector active-active**, not 3. The old "3rd connector" references in docs/memory
|
||||
(which referred to T14s/assassin) are now corrected.
|
||||
|
||||
### kscloud1 SSH access — restored 2026-06-19
|
||||
SSH key auth to kscloud1 was failing. Recovery: Hetzner console → reset root password
|
||||
([redacted]) → added `id_ed25519_kscloud1.pub` to `/home/kenpat/.ssh/authorized_keys`.
|
||||
NOTE: user on kscloud1 is `kenpat` (not `kenpatmonk`). Connect:
|
||||
`ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.x.x.x`
|
||||
Sudo password: [redacted] (use `echo [redacted] | sudo -S <cmd>` for non-interactive).
|
||||
|
||||
### Failover verified 2026-06-16
|
||||
Stopped monk's Docker cloudflared → tested all public subdomains → kscloud1 served
|
||||
everything with zero downtime: www=200, auth=302, status=302, portainer=200.
|
||||
Restarted monk cloudflared, both replicas back in CF dashboard.
|
||||
|
||||
## Phase 2 Planned: Obsidian Mind Map → HTML Mind Map Sync
|
||||
User wants to create an Obsidian mind map of the KiteStacks homelab that syncs/exports to a live HTML mind map embedded in the homelab portal or a standalone page. To be built after full Obsidian+samurai setup is complete.
|
||||
|
||||
## 2026-06-13: OpenProject removed + Oracle VPS migration started
|
||||
|
||||
### OpenProject REMOVED permanently
|
||||
OpenProject requires Enterprise Edition license for SSO (confirmed last session).
|
||||
Removed from local stack (monk):
|
||||
- Docker volume `openproject_openproject_assets` deleted
|
||||
- `/home/kenpatmonk/kitestacks-live/docker/openproject/` directory removed (pgdata dir
|
||||
needed sudo — user ran manually; pgdata was owned by container UID mapped to `avahi`)
|
||||
- NOT deploying on Oracle VPS
|
||||
- tasks.kitestacks.com subdomain is now dead — update Cloudflare/portal accordingly
|
||||
TODO: remove `apps/openproject/` from kitestacks-homelab Forgejo repo once user can log in.
|
||||
|
||||
### Forgejo issues found + partially fixed (2026-06-13)
|
||||
Forgejo login page has two issues:
|
||||
1. URL banner: "configured to be served on http://5.78.233.28:3000/" — caused by kscloud1's
|
||||
Forgejo having wrong ROOT_URL. kscloud1 Forgejo has only 1 repo (separate DB from monk's
|
||||
13-repo instance). Cloudflare tunnel load-balances between monk and kscloud1 Forgejo.
|
||||
FIX PENDING: stop Forgejo on kscloud1 (or fix its ROOT_URL). Deferred — do during Oracle migration.
|
||||
2. SSO button says "Proceed with OpenID" instead of "Authentik".
|
||||
PARTIAL FIX: renamed login_source from `authentik` → `Authentik` via admin CLI:
|
||||
`docker exec -u git forgejo /app/gitea/gitea admin auth update-oauth --id 1 --name Authentik ...`
|
||||
Provider type remains `openidConnect` — button text may still say "OpenID" (depends on
|
||||
Forgejo 11 template behavior). User to verify after refresh. Full fix may require admin UI
|
||||
once user can log into Forgejo.
|
||||
Forgejo DB: 13 repos under `kenpat`, 1 user (kenpat, admin, active, no 2FA).
|
||||
Forgejo login: username `kenpat`, direct password login works on the same page.
|
||||
|
||||
### kitestacks-homelab repo: apps/forgejo/docker-compose.yml has wrong ROOT_URL
|
||||
`FORGEJO__server__ROOT_URL=http://192.168.1.205:3006` — old local IP, never updated.
|
||||
The LIVE local stack (`~/kitestacks-live/docker/forgejo/docker-compose.yml`) is correct
|
||||
(`https://gitforge.kitestacks.com/`). The repo copy needs updating.
|
||||
TODO: fix and commit once user can log in and clone the repo.
|
||||
|
||||
### Oracle VPS migration plan (kscloud1 → Oracle Cloud)
|
||||
Goal: replace Hetzner kscloud1 (5.78.233.28, $14.50/mo) with Oracle Cloud ARM VPS ($8.50/mo).
|
||||
Oracle instance: Ampere A1 Flex, 4 OCPU / 24 GB RAM, Chicago region (us-chicago-1).
|
||||
Status as of 2026-06-13: user is provisioning — hit "no capacity" in Chicago.
|
||||
Workarounds tried: capacity not available for 4 OCPU config. Options:
|
||||
- Try smaller shape (1 OCPU / 6 GB), resize after provisioning
|
||||
- Subscribe to another region (Frankfurt, Osaka, Toronto have better A1 availability)
|
||||
- Keep retrying (capacity opens randomly, early UTC morning tends to be better)
|
||||
|
||||
ARM64 compatibility analysis (all images verified):
|
||||
- ✅ All services ARM64-compatible EXCEPT OSticket
|
||||
- ❌ OSticket (`campbellsoftwaresolutions/osticket`) — x86 only
|
||||
FIX: enable QEMU binfmt emulation on Oracle ARM host, run with `--platform linux/amd64`
|
||||
Performance acceptable for a ticket system.
|
||||
- ⚠️ Shaarli — verify ARM64 at deploy time
|
||||
|
||||
Services to deploy on Oracle VPS (OpenProject EXCLUDED):
|
||||
authentik, bookstack, cloudflared, forgejo, grafana, homepage/portal,
|
||||
karakeep (+meilisearch +chrome), kavita, kite-ai (litellm+openwebui),
|
||||
linkding, osticket, portainer, prometheus+node-exporter, shaarli, uptime-kuma
|
||||
|
||||
Migration phases:
|
||||
1. Oracle VPS provisioning (in progress)
|
||||
2. Oracle initial setup: Ubuntu 22.04 ARM64, Docker, iptables flush (Oracle blocks by default),
|
||||
QEMU binfmt for OSticket x86 emulation
|
||||
3. Deploy full stack — fix Forgejo ROOT_URL correctly from day one
|
||||
4. Connect cloudflared on Oracle to KiteStacks tunnel (same TUNNEL_TOKEN)
|
||||
5. Verify all services, then remove kscloud1 from tunnel + cancel Hetzner
|
||||
NOTE: same active-active pattern as kscloud1 — shared Authentik Postgres+Redis over
|
||||
Tailscale, same TUNNEL_TOKEN, fresh DBs for stateful apps except identity (authentik/kavita).
|
||||
IMPORTANT Oracle gotcha: Ubuntu on Oracle has iptables rules that block all traffic at boot
|
||||
even after Security List rules are opened. Must flush iptables as part of initial setup.
|
||||
|
||||
## osTicket deployed on monk + kscloud1 (found 2026-06-13/14, installed ~2026-06-12)
|
||||
osTicket (campbellsoftwaresolutions/osticket image, x86 - runs natively on both hosts,
|
||||
no QEMU needed) + nginx proxy + MariaDB 10.11, under
|
||||
`~/kitestacks-live/docker/osticket/` (monk) and `/opt/kitestacks/docker/osticket/`
|
||||
(kscloud1). `tasks.kitestacks.com` -> "KiteStacks Help Desk", verified HTTP 200.
|
||||
Admin: kenpat7177 / kenpat7177@gmail.com. Host ports: monk 8092:8080, kscloud1 8090:8080
|
||||
(both nginx -> osticket-app:80). .env (OSTICKET_DB_PASS/ROOT/ADMIN_PASS/INSTALL_SECRET)
|
||||
is IDENTICAL on both hosts.
|
||||
|
||||
### DB unification (2026-06-13/14) - same pattern as Authentik shared-DB fix
|
||||
Both hosts originally had their OWN osticket-db (drift risk like pre-fix Kavita). Per
|
||||
user request ("database should be accessible from any computer"), unified onto
|
||||
kscloud1's osticket-db as canonical:
|
||||
- kscloud1 osticket-db: added `ports: - "100.123.254.52:3306:3306"` (Tailscale-only,
|
||||
matches authentik-postgres/redis pattern) to
|
||||
`/opt/kitestacks/docker/osticket/docker-compose.yml`, `docker compose up -d`.
|
||||
- monk: `docker compose stop osticket-db` (left stopped, NOT removed - rollback data
|
||||
intact in its volume). Edited `~/kitestacks-live/docker/osticket/docker-compose.yml`:
|
||||
removed osticket-db service block, changed osticket-app's `MYSQL_HOST=osticket-db`
|
||||
-> `MYSQL_HOST=100.123.254.52`, removed `depends_on: osticket-db`. `docker compose
|
||||
up -d osticket-app`.
|
||||
- GOTCHA: after recreating osticket-app, the `osticket` nginx proxy container on monk
|
||||
returned 502 (cached stale upstream IP for osticket-app from its old container) -
|
||||
fixed with `docker restart osticket`. Apply this same restart on kscloud1's `osticket`
|
||||
nginx if its osticket-app is ever recreated.
|
||||
- Verified: both DBs had identical data before merge (1 ticket, 1 staff/kenpat7177) so
|
||||
no data loss either way. tasks.kitestacks.com returns 200 consistently post-merge.
|
||||
- Backups: `docker-compose.yml.bak` left in both hosts' osticket dirs.
|
||||
|
||||
### osticket-capstone Forgejo repo (created 2026-06-13/14)
|
||||
New private repo `kenpat/osticket-capstone` on gitforge (created via API using a
|
||||
scoped token `claude-capstone-osticket` generated via
|
||||
`docker exec -u git forgejo /app/gitea/gitea admin user generate-access-token` on
|
||||
monk's forgejo container - token has write:repository,write:user scopes). Holds
|
||||
redacted osTicket deployment config + Per Scholas capstone docs/evidence - see
|
||||
[[project-per-scholas-capstone]]. NOTE: gitforge.kitestacks.com is also
|
||||
active-active load-balanced (monk/kscloud1 separate forgejo DBs) - API calls
|
||||
against the public hostname can hit the wrong DB; use monk's local
|
||||
`http://localhost:3006` for API operations tied to monk's forgejo data.
|
||||
|
||||
### Remaining osTicket work
|
||||
- Authentik SSO plugin for osTicket staff/agent login (osTicket has no native OIDC,
|
||||
needs 3rd-party OAuth2/SAML plugin) - NOT YET DONE.
|
||||
- End-user ticket submission uses osTicket's native client portal signup (works
|
||||
out of the box, no SSO needed).
|
||||
|
||||
## 2026-06-14/15: Forgejo sync fixed + osTicket Authentik LDAP SSO complete
|
||||
|
||||
### Forgejo sync (monk → kscloud1) - FIXED
|
||||
- Ran `docker exec -u git forgejo /app/gitea/gitea dump` on monk, scp'd to kscloud1
|
||||
- Restored: 13 repos + DB synced, ROOT_URL fixed on kscloud1 to `https://gitforge.kitestacks.com/`
|
||||
- kscloud1 Forgejo docker-compose updated (correct ROOT_URL + SSH port 2222)
|
||||
- Sync script: `~/kitestacks-live/docker/forgejo/sync-to-cloud.sh` (rsync repos + DB dump)
|
||||
- Cron: `0 */6 * * *` runs sync-to-cloud.sh, logs to `/tmp/forgejo-sync.log`
|
||||
- Authentik redirect URI fixed: updated `_redirect_uris` in shared Postgres from
|
||||
`authentik/callback` → `Authentik/callback` (matched renamed Forgejo source name)
|
||||
|
||||
### osTicket Authentik LDAP SSO - COMPLETE (2026-06-14/15)
|
||||
Uses Authentik's LDAP outpost + osTicket's built-in auth-ldap.phar plugin.
|
||||
|
||||
**Authentik side:**
|
||||
- LDAPProvider "osTicket LDAP" (pk=11, base_dn=DC=ldap,DC=goauthentik,DC=io)
|
||||
- Application "osTicket LDAP" (slug=osticket-ldap, backchannel provider)
|
||||
- Outpost "osTicket LDAP Outpost" (pk=5c42f5ba-64bd-434e-a47f-7ce9da13227a)
|
||||
- Outpost service token: `jjYRKWuGtoeq9r0qeifbCnXGHDjhCJU2MLnkCvMMduIGA1kQKz85qnt7u5Zf`
|
||||
- ldap-svc user (search account): DN=`cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io`
|
||||
password=`IlgQaxBPv9rdoq03CsoY53tH`, member of homelab-admin group
|
||||
|
||||
**Docker services added on monk:**
|
||||
- `~/kitestacks-live/docker/authentik-ldap/docker-compose.yml`
|
||||
- `authentik-ldap` (ghcr.io/goauthentik/ldap:2025.2.4) on kitestacks+osticket_default networks
|
||||
- `authentik-ldap-proxy` (alpine/socat) bridges port 389→3389 on osticket_default
|
||||
so osticket-app can reach standard LDAP port without phar URI workaround
|
||||
|
||||
**Docker services added on kscloud1:**
|
||||
- `/opt/kitestacks/docker/authentik-ldap/docker-compose.yml`
|
||||
- Same authentik-ldap container, bound to 100.123.254.52:3389 (Tailscale) + 127.0.0.1:3389
|
||||
|
||||
**auth-ldap.phar patches (3 patches applied, original backed up as auth-ldap.phar.orig):**
|
||||
1. `authentication.php` - `getConnection()`: adds binddn/bindpw from plugin config to
|
||||
Net_LDAP2 params so initial connect uses credentials (not anonymous, which Authentik rejects)
|
||||
2. `config.php` - validation block: sets include_path to phar's include dir before
|
||||
`require_once Net/LDAP2.php` so sub-files resolve correctly in FPM context
|
||||
3. ALL `include/Net/LDAP2/*.php` files: guards `require_once 'PEAR.php'` with
|
||||
`if (!class_exists('PEAR', false))` to prevent fatal conflict between osTicket's
|
||||
`/include/pear/PEAR.php` and PHP global `/usr/local/lib/php/PEAR.php`
|
||||
|
||||
**osTicket LDAP plugin config (namespace plugin.2 in ost_config):**
|
||||
- servers: `authentik-ldap-proxy` (via socat on port 389)
|
||||
- bind_dn: `cn=ldap-svc,ou=users,dc=ldap,dc=goauthentik,dc=io`
|
||||
- bind_pw: encrypted with `Crypto::encrypt(pass, SECRET_SALT, 'plugin.2')`
|
||||
- search_base: `ou=users,dc=ldap,dc=goauthentik,dc=io`
|
||||
- schema: auto, auth-staff: 1, auth-client: 0, domain: ldap.goauthentik.io
|
||||
|
||||
**Staff login:** username=`kenpat7177`, password=Authentik password (reset to `KiteStacks2026!`)
|
||||
on `tasks.kitestacks.com/scp/login.php`
|
||||
|
||||
### Per Scholas IT Support Capstone - IN PROGRESS
|
||||
See [[project-per-scholas-capstone]]. Next steps:
|
||||
- Create capstone incident tickets in osTicket (5-phase challenge)
|
||||
- Set up osTicket user/client portal for non-staff users (Phase 3 end-user access)
|
||||
- Each capstone ticket maps to a phase scenario (migration event, incident response, etc.)
|
||||
|
||||
## 2026-06-15: Uptime Kuma -> ntfy phone notifications configured
|
||||
|
||||
- ntfy is running at `https://ntfy.kitestacks.com` with deny-by-default auth.
|
||||
- Existing ntfy user `alerts` has write-only access to all topics and now has a
|
||||
dedicated access token for Uptime Kuma. Do not write the token into repo/memory.
|
||||
- Phone subscription topic: `kitestacks-uptime-VHr1DMQNi8`.
|
||||
- Anonymous/everyone access is read-only for that single topic, so phones can
|
||||
subscribe without needing the Kuma publish token.
|
||||
- Uptime Kuma notification row:
|
||||
- name `KiteStacks ntfy`
|
||||
- type `ntfy`
|
||||
- server URL `http://ntfy` on the shared Docker network
|
||||
- topic `kitestacks-uptime-VHr1DMQNi8`
|
||||
- auth method `accessToken`
|
||||
- active/default, linked to active monitors `T14 Deb Assassin` and `Google DNS`
|
||||
- Verified by sending a Kuma-provider test message and reading it publicly via
|
||||
`https://ntfy.kitestacks.com/kitestacks-uptime-VHr1DMQNi8/json?poll=1`.
|
||||
- To add to a phone home screen: open
|
||||
`https://ntfy.kitestacks.com/kitestacks-uptime-VHr1DMQNi8` on the phone, allow
|
||||
notifications, then use the browser share/menu action to add ntfy to the home
|
||||
screen.
|
||||
|
|
|
|||
|
|
@ -30,22 +30,28 @@ authentik, authentik-worker, authentik-ldap, authentik-ldap-proxy, bookstack, bo
|
|||
- OSTicket SMTP configured: smtp.gmail.com:587, kitestacks.helpdesk@gmail.com, app password stored in ost_email table (smtp_auth_creds=1 for all 3 emails)
|
||||
- portainer.kitestacks.com CF tunnel hostname — user confirmed already set in CF dashboard
|
||||
|
||||
## Completed 2026-06-18 (this session)
|
||||
## Completed 2026-06-18/19 (this session)
|
||||
- Forgejo repo reorganization: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem merged as subdirs (cloud/, cloud-migration/, autosync/, osticket/) into kitestacks-homelab repo. Committed and pushed.
|
||||
- comptia-a-plus-core2 Forgejo repo updated: merged study-tracker content, added certifications/ dir, updated exam goal to July 7.
|
||||
- homelab-mastery Forgejo repo: architecture/overview.md and build-guide/README.md rewritten in plain English.
|
||||
- RUNBOOK.md + DEBUG-DOCUMENTATION.md: rewritten in 5th-grade plain English in kitestacks-homelab repo.
|
||||
- All 6 BookStack pages updated via API (Runbook, Debug, Architecture, Build Guide, AI Guide, Manual Guide).
|
||||
- Forgejo API token via external URL broken (Cloudflare strips Authorization header). Works via localhost:3006.
|
||||
- BookStack API token created (claude-push-825981) via DB injection + bcrypt hash. Works internally.
|
||||
- BookStack API token created (claude-push-825981) via DB injection + bcrypt hash.
|
||||
- **monk Forgejo migrated to shared PostgreSQL (2026-06-19):** Used `forgejo dump --database postgres` to generate clean SQL, dropped pgloader schema, reloaded. Both Cloudflare connectors now return 200 for API token `[redacted]`. Monk reads from `authentik-postgres` at `100.x.x.x:5432`, DB name `forgejo`, user `forgejo`.
|
||||
- SQLite backup at: `~/kitestacks-live/docker/forgejo/data/gitea/gitea.db.backup-20260618-230715`
|
||||
|
||||
## Completed 2026-06-19
|
||||
- kscloud1 SSH restored: key added to /home/kenpat/.ssh/authorized_keys (user is `kenpat`, not `kenpatmonk`). SSH via `ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.x.x.x`.
|
||||
- kscloud1 Forgejo migrated to shared PostgreSQL: compose at /opt/kitestacks/docker/forgejo/docker-compose.yml, joined authentik_default network to reach authentik-postgres:5432. 20/20 external API requests pass.
|
||||
- BookStack kscloud1 OIDC: already configured, cache perms fixed, OIDC login redirects to auth.kitestacks.com correctly.
|
||||
|
||||
## Confirmed working
|
||||
- OSTicket SMTP (smtp.gmail.com:587, kitestacks.helpdesk@gmail.com) — confirmed 2026-06-19.
|
||||
- Portainer Authentik OAuth SSO — confirmed working 2026-06-19.
|
||||
- Old standalone repos archived 2026-06-19: kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem.
|
||||
|
||||
## Pending
|
||||
- BookStack kscloud1: update compose (OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/, OIDC_ISSUER_DISCOVER=true), restart, fix cache perms (chown -R abc:users /config/www/framework/cache/). Blocked by kscloud1 SSH key needs re-adding.
|
||||
- kscloud1 SSH: re-add id_ed25519_kscloud1.pub key via Hetzner VNC console.
|
||||
- OSTicket SMTP test email — verify delivery works.
|
||||
- Archive/delete now-redundant standalone repos (kitestacks-cloud, kitestacks-cloud-migration, kitestacks-homelab-autosync-test, OSTicketSystem) once user confirms move is good.
|
||||
- Portainer Authentik OAuth setup on both Portainer UIs (manual user action).
|
||||
- Forgejo Authorization header fix: investigate why Cloudflare strips the token header for API calls (may need Cloudflare WAF rule or different auth method).
|
||||
- (none)
|
||||
|
||||
## Completed 2026-06-18
|
||||
- Portainer OAuth: both monk + kscloud1 configured (AuthenticationMethod=3, Authentik SSO). OAuth user kenpat7177@gmail.com pre-created as Role:1 (admin) on both. Local Docker environment added to both. Portal card already live.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue