kitestacks-homelab/homelab-mastery/architecture/services.md

# KiteStacks — Complete Service Reference

Every service that runs in KiteStacks: what it does, where it lives, how to manage it,
and what commands to use. This is the day-to-day operations reference.

**Last Updated:** 2026-06-19

---

## Quick Reference — All Containers on monk

```
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```

| Container | Purpose | Public URL |
|-----------|---------|-----------|
| `homepage` | Portal / main website | www.kitestacks.com |
| `authentik` | SSO identity provider | auth.kitestacks.com |
| `authentik-worker` | Authentik background jobs | — |
| `authentik-ldap` | LDAP interface for Authentik | — |
| `authentik-ldap-proxy` | LDAP proxy | — |
| `forgejo` | Git platform | gitforge.kitestacks.com |
| `kite-openwebui` | AI chat | ai.kitestacks.com |
| `kite-litellm` | LLM proxy gateway | — |
| `karakeep` | Bookmarks | links.kitestacks.com |
| `karakeep-chrome` | Headless browser for Karakeep | — |
| `karakeep-meilisearch` | Search engine for Karakeep | — |
| `kavita` | eBook reader | kavita.kitestacks.com |
| `grafana` | Monitoring dashboards | grafana.kitestacks.com |
| `uptime-kuma` | Status page | status.kitestacks.com |
| `bookstack` | Wiki / docs | wiki.kitestacks.com |
| `bookstack-db` | MariaDB for BookStack | — |
| `osticket-app` | Help desk | tasks.kitestacks.com |
| `osticket-db` | MySQL for OSTicket | — |
| `portainer` | Docker management UI | portainer.kitestacks.com |
| `cloudflared` | Tunnel connector | — |
| `prometheus` | Metrics collector | — |
| `node-exporter` | Host metrics exporter | — |
| `blackbox-exporter` | HTTP probe monitor | — |
| `kitestacks-metrics-api` | System stats API for portal | — |
| `ntfy` | Push notifications | — |
| `flux` | GitOps controller | — |

---

## Service Deep Dives

### homepage — Portal

**What it is:** Custom-built static website served by nginx.
**Directory:** `~/kitestacks-live/docker/kitestacks-portal/`
**Public files:** `./public/index.html` — edit this to change what the portal shows
**Config:** `./nginx.conf` — nginx routing rules

```bash
# Restart portal
cd ~/kitestacks-live/docker/kitestacks-portal
docker compose restart homepage

# Edit the portal
nano public/index.html

# View nginx logs
docker logs homepage -f
```

**Ports:** 3005:3000 (host:container). Cloudflare Tunnel uses container port 3000 directly.

---

### authentik — SSO Identity Provider

**What it is:** Self-hosted OAuth2/OIDC identity provider. Handles all logins for every service.
**Directory:** `~/kitestacks-live/docker/authentik/`
**Database:** Shared PostgreSQL on kscloud1 at `100.123.x.x:5432`, database `authentik`
**Redis:** Shared Redis on kscloud1 at `100.123.x.x:6379`

```bash
cd ~/kitestacks-live/docker/authentik

# Start all Authentik services
docker compose up -d

# Check health (wait for "healthy" before testing SSO)
docker inspect --format '{{.State.Health.Status}}' authentik
docker inspect --format '{{.State.Health.Status}}' authentik-worker

# Run a Django management command (admin tasks, user management)
docker exec authentik ak shell

# View logs
docker logs authentik -f
docker logs authentik-worker -f
```

**SSO apps configured in Authentik:**
- Grafana, Forgejo, Kavita, Karakeep, Open WebUI, Portainer, BookStack

**Key Authentik admin panel:** https://auth.kitestacks.com/if/admin/

**Important:** OAuth2 code TTL is set to 10 minutes (increased from default 1 minute)
to allow monk's Authentik to finish starting up after a reconnect before codes expire.

---

### forgejo — Git Platform

**What it is:** Self-hosted Git. Stores all homelab code, configs, and documentation.
**Directory:** `~/kitestacks-live/docker/forgejo/`
**Database:** Shared PostgreSQL on kscloud1, database `forgejo`, user `forgejo`
**Data volume:** `./data/` (repositories, avatars, attachments)

```bash
cd ~/kitestacks-live/docker/forgejo

# Start
docker compose up -d

# Admin commands
docker exec -u git forgejo forgejo admin user list
docker exec -u git forgejo forgejo admin user create --username newuser --password pass --email e@mail.com --admin

# View logs
docker logs forgejo -f

# API token for automation
# Token: stored in .env — used by kitestacks-metrics-api for activity feed
```

**API base URL:** `https://gitforge.kitestacks.com/api/v1/`
**Local access (via Cloudflare):** gitforge.kitestacks.com

---

### kite-openwebui — AI Chat

**What it is:** Self-hosted ChatGPT-like interface connected to LiteLLM proxy.
**Directory:** `~/kitestacks-live/docker/kite-openwebui/`
**Backend:** `kite-litellm` — routes to OpenRouter (many models, free tier available)

```bash
cd ~/kitestacks-live/docker/kite-openwebui
docker compose up -d
docker logs kite-openwebui -f
docker logs kite-litellm -f
```

**SSO:** Authentik OIDC — "Sign in with Authentik" on login page.

---

### karakeep — Bookmarks

**What it is:** Bookmark manager and read-it-later tool. Saves full page content.
**Directory:** `~/kitestacks-live/docker/karakeep/`
**Depends on:** `karakeep-chrome` (headless Chromium for page capture) + `karakeep-meilisearch` (search)

```bash
cd ~/kitestacks-live/docker/karakeep
docker compose up -d

# SSO callback URL: https://links.kitestacks.com/api/auth/callback/custom
# (NextAuth.js uses "custom" as the provider ID, not "authentik")
```

**SSO:** Authentik OAuth2 — redirect URI must be `/api/auth/callback/custom` (not `/callback/authentik`)

---

### kavita — eBook Reader

**What it is:** eBook, manga, and comic library.
**Directory:** `~/kitestacks-live/docker/kavita/`
**Book files:** `./library/books/` — add books here, then scan library in Kavita UI
**Config/DB:** `./config/kavita.db` (SQLite)

```bash
cd ~/kitestacks-live/docker/kavita
docker compose up -d
docker logs kavita -f

# If you change OIDC settings, use the Kavita UI at kavita.kitestacks.com/settings
# Do NOT edit kavita.db directly for OIDC config — Kavita overwrites it on restart
# Use SSH port-forward to access kscloud1's Kavita directly if needed:
# ssh -L 5099:localhost:5000 kenpat@kscloud1-tailscale-ip
# Then visit http://localhost:5099
```

**SSO:** Authentik OIDC — Authority URL must end with trailing slash:
`https://auth.kitestacks.com/application/o/kavita/`

---

### grafana — Monitoring Dashboards

**What it is:** Visualizes metrics collected by Prometheus.
**Directory:** `~/kitestacks-live/docker/grafana/`
**Provisioning:** `./provisioning/` — auto-loads datasource (Prometheus) and dashboard (Node Exporter Full)
**Data:** Named Docker volume `grafana-data`

```bash
cd ~/kitestacks-live/docker/grafana
docker compose up -d
docker logs grafana -f
```

**Dashboards auto-loaded:**
- Node Exporter Full (id 1860) — CPU, RAM, disk, network for both monk and kscloud1
- Switch between hosts using the "instance" variable at top of dashboard

**SSO:** Authentik OAuth2. Local admin login also works.

---

### uptime-kuma — Status Page

**What it is:** Uptime monitoring with a public status page.
**Directory:** `~/kitestacks-live/docker/uptime-kuma/`
**Database:** Named Docker volume `uptime-kuma` (SQLite kuma.db)
**Status page slug:** `homelab` → https://status.kitestacks.com/status/homelab

```bash
cd ~/kitestacks-live/docker/uptime-kuma
docker compose up -d
docker logs uptime-kuma -f

# To push kuma.db to kscloud1 after changes (monk → kscloud1):
# See scripts/sync-kuma.sh (or follow the sqlite backup pattern)
```

**Monitors configured:** All 11 public services + kscloud1 ping + Monk ping + Samurai ping.

**Conky widget:** Reads kscloud1's Uptime Kuma directly via Tailscale IP at
`http://100.123.x.x:3001/api/status-page/homelab`. This means the widget shows
kscloud1's health, not monk's — which is what matters for production status.

---

### bookstack — Wiki

**What it is:** Self-hosted documentation wiki with a clean UI.
**Directory:** `~/kitestacks-live/docker/bookstack/`
**Database:** MariaDB container `bookstack-db`
**Config:** `.env` file (APP_URL, DB settings, OIDC config)

```bash
cd ~/kitestacks-live/docker/bookstack
docker compose up -d
docker logs bookstack -f

# BookStack API (used to push docs from Forgejo):
# Token created via: DB injection + bcrypt hash for API key
# Token ID/secret stored in .env
```

**SSO:** Authentik OIDC. Key config:
- `OIDC_ISSUER=https://auth.kitestacks.com/application/o/bookstack/`
- `OIDC_ISSUER_DISCOVER=true`
- Cache dir must be writable: `chown -R abc:users /config/www/framework/cache/`

---

### osticket-app — Help Desk

**What it is:** OSTicket help desk and ticketing system.
**Directory:** `~/kitestacks-live/docker/osticket/`
**Database:** MySQL container `osticket-db`
**URL:** tasks.kitestacks.com (took over from OpenProject)

```bash
cd ~/kitestacks-live/docker/osticket
docker compose up -d
docker logs osticket-app -f
```

**SMTP:** Configured for smtp.gmail.com:587 using kitestacks.helpdesk@gmail.com.
App password stored in `ost_email` table (smtp_auth_creds=1 for all email entries).
**Confirmed working:** Email delivery verified 2026-06-19.

---

### portainer — Docker Management

**What it is:** Web UI for managing Docker containers on both monk and kscloud1.
**Directory:** `~/kitestacks-live/docker/portainer/`
**URL:** portainer.kitestacks.com

```bash
cd ~/kitestacks-live/docker/portainer
docker compose up -d
```

**SSO:** Authentik OAuth2 (AuthenticationMethod=3). User kenpat7177@gmail.com pre-created as admin.
**Security:** Authentik PolicyBinding restricts Portainer app to `homelab-admin` group only.

---

### cloudflared — Tunnel Connector

**What it is:** Creates the outbound tunnel to Cloudflare. This is what makes all
public services reachable without opening ports on the router.
**Directory:** `~/kitestacks-live/docker/cloudflared/`
**Token:** Read from `.env` file as `TUNNEL_TOKEN` (never hardcoded in docker-compose.yml)

```bash
cd ~/kitestacks-live/docker/cloudflared
docker compose up -d
docker logs cloudflared -f

# To rotate the token (runs on both monk and kscloud1):
# ~/kitestacks-homelab/scripts/rollout-cloudflared-token.sh '<new-token>'
```

**Tunnel ID:** 5e60ea8e-a543-49b6-bab5-325f39441e00
**Account:** Cloudflare dashboard → Zero Trust → Networks → Tunnels

---

### prometheus + node-exporter — Metrics

**What it is:** Prometheus collects time-series metrics. node-exporter exposes host stats.
**Directory:** `~/kitestacks-live/docker/prometheus/`
**Config:** `./prometheus.yml` — defines scrape targets

```bash
cd ~/kitestacks-live/docker/prometheus
docker compose up -d
docker logs prometheus -f

# Scrape targets configured:
# - node-exporter:9100 (monk, via Docker DNS)
# - 5.78.x.x:9100     (kscloud1, via public IP — node-exporter exposed on 0.0.0.0)
```

---

## Common Operations

### Restart a single service
```bash
cd ~/kitestacks-live/docker/<service-name>
docker compose restart <container-name>
```

### View live logs
```bash
docker logs <container-name> -f
# -f = follow (live tail). Ctrl+C to stop.
```

### Update a service to latest image
```bash
cd ~/kitestacks-live/docker/<service-name>
docker compose pull
docker compose up -d
```

### Check all container health at once
```bash
docker ps --format "table {{.Names}}\t{{.Status}}"
```

### Enter a container's shell
```bash
docker exec -it <container-name> bash
# or sh if bash isn't available:
docker exec -it <container-name> sh
```

### Check disk and memory usage
```bash
docker system df        # Docker disk usage
free -h                 # RAM usage
df -h                   # Disk usage
```

### Push a kuma.db update to kscloud1
```bash
# 1. Make changes to monk's Uptime Kuma (add monitors, etc.)
# 2. Backup monk's db:
docker run --rm -v uptime-kuma:/src:ro -v /tmp:/out python:3-alpine \
  python3 -c "import sqlite3; s=sqlite3.connect('/src/kuma.db'); b=sqlite3.connect('/out/kuma.db.push'); s.backup(b); b.close(); s.close()"
# 3. Transfer and restore on kscloud1:
gzip -c /tmp/kuma.db.push | ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@100.123.x.x \
  "gunzip > /home/kenpat/kuma.db.push"
# Then on kscloud1: stop uptime-kuma, restore via same sqlite.backup() pattern, restart
```