init: complete homelab mastery guide

Architecture overview, design decisions, Docker/networking/OAuth2/Linux concept deep-dives, cert roadmap for cloud engineering track, interview prep with model answers, and structured learning path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 20:08:27 -05:00 · 2026-06-11 20:08:27 -05:00 · ca9e8a7959
commit ca9e8a7959
10 changed files with 1534 additions and 0 deletions
--- a/concepts/docker.md
+++ b/concepts/docker.md
@ -0,0 +1,126 @@
+# Docker — What It Actually Is
+
+## The Wrong Mental Model
+
+Most people think containers are "mini virtual machines." They're not. Understanding the real model is what makes Docker make sense.
+
+## What a Container Actually Is
+
+A container is a **process** (or group of processes) running on the host Linux kernel, with two things applied:
+
+1. **Namespaces** — isolation. The process gets its own view of the filesystem, network, processes, users. It can't see other containers' processes.
+2. **cgroups (control groups)** — resource limits. The process is limited to a certain amount of CPU, RAM, etc.
+
+That's it. There's no second kernel. No hypervisor. No hardware emulation. The nginx running in your `homepage` container is a regular Linux process on your laptop — it just *thinks* it's alone.
+
+This is why containers start in milliseconds (no boot) and use almost no overhead.
+
+## Images vs Containers
+
+| Concept | Analogy | What it is |
+|---------|---------|-----------|
+| **Image** | A recipe | A read-only template — filesystem layers, default command, environment |
+| **Container** | A meal cooked from that recipe | A running instance of an image — has its own writable layer on top |
+
+You can run 10 containers from the same image. They all share the read-only image layers and each gets their own writable layer on top. If a container is deleted, its writable layer is gone. The image remains.
+
+When you run `docker compose up -d`, Docker:
+1. Pulls the image if not already local
+2. Creates a container (adds writable layer)
+3. Attaches it to the specified networks
+4. Mounts the volumes
+5. Starts the process defined in the image's CMD or your compose override
+
+## Docker Networks — Why the `kitestacks` Network Exists
+
+Docker creates several default networks. Containers on the **same network** can reach each other by **container name** (Docker has its own DNS built in).
+
+In this homelab:
+```
+docker network create kitestacks
+```
+
+Every container joins this network. So when cloudflared routes traffic for `www.kitestacks.com`, it resolves `homepage` via Docker DNS to the container's IP on the `kitestacks` network. Without this shared network, cloudflared can't find the other containers.
+
+```
+cloudflared container → DNS lookup "homepage" → 172.x.x.x (homepage container)
+```
+
+**`network_mode: host`** is different — the container shares the HOST's network namespace entirely. No isolation. Used for the metrics API so it can read actual host network stats.
+
+## Volumes — Keeping Data When Containers Are Deleted
+
+Containers are ephemeral — their writable layer is deleted when the container is removed. To persist data:
+
+**Bind mount:** Links a host directory to a container path.
+```yaml
+volumes:
+  - ./data:/forgejo-data
+```
+`./data` on your laptop → `/forgejo-data` inside container. Data lives on your laptop. You can browse it with `ls`.
+
+**Named volume:** Docker manages the storage location.
+```yaml
+volumes:
+  - prometheus-data:/prometheus
+```
+Docker stores it in `/var/lib/docker/volumes/prometheus-data/`. You don't specify where.
+
+**In this homelab:** Databases, config files, and user data use bind mounts (`./data`, `./config`, etc.) so you know exactly where everything is. Named volumes are used where location doesn't matter (Prometheus metrics, Portainer settings).
+
+## Docker Compose — What It's Doing
+
+`docker compose up -d` reads `docker-compose.yml` and for each service:
+1. Ensures the image exists (pull if needed)
+2. Creates the network if it doesn't exist
+3. Creates the container with all specified config (env vars, volumes, ports, networks)
+4. Starts the container
+
+`-d` means detached — run in background.
+
+`restart: unless-stopped` means Docker will restart the container if it crashes or if the host reboots — unless you explicitly stop it with `docker compose stop`.
+
+## Port Mappings
+
+```yaml
+ports:
+  - "3006:3000"
+```
+
+`HOST_PORT:CONTAINER_PORT`
+
+Port 3000 inside the container is mapped to port 3006 on the host. From the host, `http://localhost:3006` reaches the service. From within the `kitestacks` Docker network, other containers use `http://forgejo:3000` (the container port, via Docker DNS).
+
+Cloudflare Tunnel doesn't use host ports — it goes through the Docker network directly using the container name and container port.
+
+## Commands to Know Cold
+
+```bash
+# See all running containers
+docker ps
+
+# See logs for a container
+docker logs forgejo
+docker logs -f forgejo          # follow (live tail)
+
+# Execute a command inside a running container
+docker exec -it forgejo bash   # open a shell
+docker exec forgejo forgejo admin user list  # run a specific command
+
+# Inspect a container's config
+docker inspect authentik
+
+# See all networks
+docker network ls
+docker network inspect kitestacks
+
+# See disk usage
+docker system df
+
+# Remove unused images/containers/networks
+docker system prune
+```
+
+## What to Say About Docker
+
+> *"I containerized every service using Docker and Docker Compose. Each service is isolated in its own container with its own dependencies, connected through a shared Docker bridge network named 'kitestacks' so they can communicate by container name. Data is persisted via bind-mounted host directories. The entire stack is defined in version-controlled YAML files, making it reproducible on any Linux host."*
--- a/concepts/linux.md
+++ b/concepts/linux.md
@ -0,0 +1,185 @@
+# Linux — Commands and Concepts to Own
+
+You've been running Linux commands without fully owning them. This fixes that.
+
+---
+
+## The Filesystem
+
+Everything in Linux is a file. The filesystem tree starts at `/` (root):
+
+```
+/
+├── etc/         Configuration files (system-wide)
+├── home/        User home directories (/home/kenpat)
+├── opt/         Optional/third-party software (kscloud1 services live here)
+├── proc/        Virtual filesystem — running processes, kernel info
+│   ├── uptime       System uptime in seconds
+│   └── net/route    Routing table (used by metrics API to find active interface)
+├── sys/         Virtual filesystem — hardware/kernel info
+├── var/         Variable data — logs, databases, cache
+└── usr/         User programs and libraries
+```
+
+---
+
+## File Permissions
+
+Every file has three permission sets: **owner**, **group**, **others**.
+
+```bash
+ls -la ~/kitestacks-live/docker/kavita/config/kavita.db
+-rw-r--r-- 1 kenpat kenpat 2.4M Jun 11 kavita.db
+```
+
+Breaking it down:
+- `-` — it's a file (not a directory `d` or symlink `l`)
+- `rw-` — owner (kenpat): read + write
+- `r--` — group (kenpat): read only
+- `r--` — others: read only
+
+**chmod** changes permissions. **chown** changes owner.
+
+Why this mattered: When syncing kavita.db to kscloud1, you ran `chown 1000:1000` because the Kavita container runs as user ID 1000. If the file is owned by the wrong user ID, the container can't write to it.
+
+```bash
+chmod 644 kavita.db      # rw-r--r--
+chmod 755 script.sh      # rwxr-xr-x (executable)
+chown 1000:1000 kavita.db  # set owner to UID 1000, GID 1000
+chown -R kenpat:kenpat ./  # recursive (-R) on a directory
+```
+
+---
+
+## Processes
+
+```bash
+ps aux                   # all running processes
+ps aux | grep forgejo    # find forgejo processes
+kill 1234               # send SIGTERM to PID 1234 (polite stop)
+kill -9 1234            # send SIGKILL (force kill, no cleanup)
+```
+
+**systemctl** manages systemd services (services that start on boot):
+```bash
+systemctl status tailscaled     # is tailscale running?
+systemctl restart tailscaled    # restart it
+systemctl enable tailscaled     # start on boot
+journalctl -u tailscaled        # logs for tailscale service
+```
+
+Your containers don't use systemd — Docker manages them with `restart: unless-stopped`.
+
+---
+
+## Networking Commands
+
+```bash
+# What ports is this machine listening on?
+ss -tlnp                 # TCP listening, numeric, with process
+ss -tlnp | grep :3006   # is Forgejo's port bound?
+
+# Test connectivity
+ping 8.8.8.8             # can I reach Google DNS?
+curl -I https://auth.kitestacks.com  # HTTP headers from Authentik
+curl -s http://localhost:8000/api/health  # test metrics API
+
+# DNS lookup
+dig www.kitestacks.com   # full DNS query details
+nslookup gitforge.kitestacks.com  # simpler DNS lookup
+
+# Firewall
+sudo ufw status          # what rules are active?
+sudo ufw allow 22/tcp    # allow SSH
+sudo ufw deny 3306/tcp   # block MySQL from outside
+```
+
+---
+
+## Piping and Redirection
+
+The `|` (pipe) sends output of one command as input to another:
+
+```bash
+docker ps | grep forgejo           # filter docker ps output
+cat prometheus.yml | grep job      # find job lines in config
+docker logs authentik 2>&1 | grep ERROR  # show only errors
+```
+
+`2>&1` redirects stderr (error output, stream 2) to stdout (stream 1) — so errors appear in the same stream as normal output and can be piped.
+
+`>` redirects output to a file (overwrites):
+```bash
+pg_dump authentik > authentik-backup.sql
+```
+
+`>>` appends to a file:
+```bash
+echo "new line" >> ~/.ssh/config
+```
+
+---
+
+## SSH
+
+SSH (Secure Shell) gives you a terminal session on a remote machine.
+
+```bash
+ssh kenpat@5.78.x.x                    # basic SSH
+ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.x.x  # specify key
+ssh -L 5099:localhost:5000 kenpat@5.78.x.x          # local port forward
+```
+
+**Local port forward** (`-L local:remote_host:remote_port`):
+`ssh -L 5099:localhost:5000 kenpat@kscloud1` means:
+- Traffic to YOUR localhost:5099
+- Gets tunneled through the SSH connection
+- And hits kscloud1's localhost:5000
+
+You used this to access kscloud1's Kavita instance (running on port 5000) from your browser at http://localhost:5099 — without opening that port to the internet.
+
+---
+
+## sudo and Non-Interactive Usage
+
+`sudo` runs a command as root. It normally prompts for your password.
+
+**The kscloud1 problem:** In automated scripts, there's no terminal to enter a password. Solution:
+```bash
+echo YOUR_PASSWORD | sudo -S command
+# -S reads password from stdin instead of terminal
+```
+
+You used this to run ufw commands non-interactively. In real production environments, this is handled differently (sudoers file with NOPASSWD for specific commands, or SSH key-based service accounts).
+
+---
+
+## Grep, Sed, Awk
+
+**grep** finds lines matching a pattern:
+```bash
+grep "error" /var/log/syslog          # lines containing "error"
+grep -i "error" logfile               # case-insensitive
+grep -n "AUTHENTIK" docker-compose.yml # show line numbers
+grep -r "p12217177" /opt/kitestacks/  # recursive search in directory
+```
+
+**sed** (stream editor) modifies text:
+```bash
+sed 's/old_text/new_text/g' file.txt  # replace all occurrences
+sed -i 's/old/new/g' file.txt         # -i edits file in place
+```
+
+**awk** processes structured text (columns):
+```bash
+grep PG_PASS .env | cut -d= -f2-      # get value after = (including trailing =)
+# cut -d= splits on =, -f2- means "field 2 and everything after"
+# This is why: PG_PASS=abc= → if you use -f2, you get "abc" (loses trailing =)
+# With -f2-, you get "abc=" (correct)
+```
+
+---
+
+## What to Say About Linux
+
+> *"All services run on Linux hosts. I'm comfortable with file permissions, process management, SSH configuration (including local port forwarding for secure access to non-exposed services), firewall rules with ufw, and command-line tools like grep, curl, and docker CLI. I diagnosed and fixed a network configuration issue on the cloud VPS where ufw's default-deny policy was blocking Docker container traffic to a host-network-mode service."*
--- a/concepts/networking.md
+++ b/concepts/networking.md
@ -0,0 +1,187 @@
+# Networking — The Foundation of Everything
+
+This is the most important concept to master. Every other technology in this homelab is built on networking fundamentals. CCNA will teach this deeply — this is the overview.
+
+---
+
+## IP Addresses
+
+Every device on a network has an IP address — a unique identifier.
+
+**IPv4:** Four numbers 0–255 separated by dots: `192.168.1.205`
+
+**Private ranges** (not routable on the internet, only on local networks):
+- `192.168.x.x` — home networks (your router assigns these)
+- `172.16.x.x – 172.31.x.x` — Docker bridge networks use this range
+- `10.x.x.x` — corporate networks often use this
+
+**Public IPs:** Routable on the internet. Your home has one (assigned by ISP). kscloud1 has one (assigned by Hetzner).
+
+**Tailscale IPs:** `100.x.x.x` — a special private range used by Tailscale for its overlay network.
+
+---
+
+## Subnets and CIDR Notation
+
+A subnet is a range of IP addresses. CIDR notation describes the range:
+- `192.168.1.0/24` — all addresses from `192.168.1.0` to `192.168.1.255` (256 addresses)
+- `172.16.0.0/12` — a large range covering all Docker bridge networks
+- `/32` — a single IP address
+
+The number after `/` is the prefix length — how many bits are fixed. The remaining bits define the host range.
+
+In this homelab: `ufw allow from 172.16.0.0/12` allows traffic from any Docker container to the host. That `/12` covers all possible Docker bridge subnet addresses.
+
+---
+
+## Ports
+
+A port is a number (0–65535) that identifies a specific service on a host. Think of the IP address as the building, and the port as the apartment number.
+
+**Well-known ports:**
+- 22 — SSH
+- 80 — HTTP
+- 443 — HTTPS
+- 3306 — MySQL/MariaDB
+- 5432 — PostgreSQL
+- 6379 — Redis
+
+**Your homelab ports** (just examples — you know yours):
+- Each service binds to a port inside the container
+- Docker maps host ports to container ports: `3006:3000`
+
+When a service "listens on a port," it's waiting for TCP/UDP connections on that port. When cloudflared connects to `http://grafana:3000`, it's connecting to IP of the `grafana` container on port 3000.
+
+---
+
+## DNS — How Names Become IPs
+
+DNS (Domain Name System) translates human-readable names to IP addresses.
+
+```
+www.kitestacks.com  →  DNS lookup  →  Cloudflare's IP address
+grafana             →  Docker DNS  →  172.x.x.x (container IP)
+100.123.x.x         →  Tailscale DNS  →  kscloud1
+```
+
+**Cloudflare DNS:** You configured NS records to point `kitestacks.com` to Cloudflare's nameservers. Cloudflare then controls all DNS for that domain. The A record for `www.kitestacks.com` points to Cloudflare's anycast IP, not your home IP.
+
+**Docker DNS:** Inside the `kitestacks` Docker network, Docker runs an internal DNS server at `127.0.0.11`. When cloudflared looks up `homepage`, Docker DNS returns the container's IP on the bridge network.
+
+**How to check DNS:**
+```bash
+dig www.kitestacks.com          # what does the public DNS say?
+nslookup grafana                # from inside a container
+```
+
+---
+
+## HTTP vs HTTPS
+
+**HTTP (HyperText Transfer Protocol):** Data is sent in plain text. Anyone who can see the network traffic can read it.
+
+**HTTPS:** HTTP + TLS encryption. Data is encrypted in transit.
+
+**TLS (Transport Layer Security):** A cryptographic protocol. Requires a certificate proving the server is who it claims to be.
+
+In this homelab:
+- All internal Docker network traffic is HTTP — it never leaves the host, so encryption isn't needed
+- All public traffic goes through Cloudflare, which handles TLS — Cloudflare terminates HTTPS at the edge
+- Between Cloudflare and cloudflared (the tunnel itself), traffic is encrypted by the tunnel protocol
+
+**Certificates:** Cloudflare manages TLS certificates for `*.kitestacks.com` automatically — you don't need to configure Let's Encrypt or buy a certificate.
+
+---
+
+## Reverse Proxy
+
+A reverse proxy sits in front of services and routes requests to them.
+
+```
+Client → Reverse Proxy → Service A
+                      ↘ Service B
+                      ↘ Service C
+```
+
+In this homelab, Cloudflare + cloudflared acts as the reverse proxy:
+- Receives all inbound HTTPS traffic
+- Decrypts TLS
+- Reads the `Host` header (`www.kitestacks.com`, `grafana.kitestacks.com`, etc.)
+- Routes to the correct container based on the hostname rules you configured
+
+nginx (the portal container) is also a reverse proxy — it forwards `/api/*` requests to the metrics API running on the host.
+
+---
+
+## Cloudflare Tunnel — Deep Dive
+
+The tunnel replaces the need for port forwarding. Here's exactly what happens:
+
+**Setup (happens once when you start cloudflared):**
+1. cloudflared reads the TUNNEL_TOKEN
+2. It makes an outbound HTTPS connection to Cloudflare's edge servers (`region1.argotunnel.com`)
+3. It authenticates and registers as a connector
+4. Cloudflare keeps this connection open (persistent, long-lived)
+
+**When a request comes in:**
+1. User's browser connects to Cloudflare's edge (the public IP in DNS)
+2. Cloudflare sees the Host header: `grafana.kitestacks.com`
+3. Cloudflare looks up the tunnel configuration — `grafana.kitestacks.com` → `http://grafana:3000`
+4. Cloudflare sends the request over the existing tunnel connection to cloudflared
+5. cloudflared resolves `grafana` via Docker DNS → gets container IP
+6. cloudflared forwards the request to the grafana container
+7. Response goes back through the tunnel to Cloudflare → to the user
+
+**The key insight:** All of this happens over a single outbound connection from cloudflared. No inbound ports. Your home router doesn't know any of this is happening.
+
+---
+
+## Tailscale — Overlay Network
+
+Tailscale creates a WireGuard mesh between devices. Each device gets a `100.x.x.x` IP that works regardless of physical location or network.
+
+**Under the hood:**
+- WireGuard: a modern VPN protocol, UDP-based, very fast, cryptographically simple
+- Tailscale coordinates key exchange via their servers, but actual traffic is peer-to-peer
+- Works behind NAT via UDP hole-punching (most of the time)
+- Falls back to relay servers (DERP) if direct connection isn't possible
+
+**Why this matters for the homelab:**
+- kscloud1's Postgres and Redis bind to `100.123.x.x` (Tailscale IP), not `0.0.0.0`
+- Even though kscloud1 has a public IP, the database is unreachable from the internet
+- Only devices on the tailnet can connect to it
+- Monk's Authentik connects to `100.123.x.x:5432` — traffic goes through the encrypted Tailscale tunnel
+
+---
+
+## Firewall Basics (ufw)
+
+ufw (Uncomplicated Firewall) manages Linux's netfilter/iptables rules.
+
+```bash
+ufw default deny incoming   # block all inbound by default
+ufw allow ssh               # allow SSH (port 22)
+ufw allow from 172.16.0.0/12 to any port 8000  # Docker containers → metrics API
+```
+
+On kscloud1: ufw blocks everything by default. The exception for `172.16.0.0/12` allows containers (which use 172.x.x.x addresses) to reach port 8000 on the host (where the metrics API runs in host network mode).
+
+Without that rule: the homepage container calls `host.docker.internal:8000` → kernel sees source `172.x.x.x` → ufw blocks it → System Status widget shows "Offline."
+
+---
+
+## What to Know Cold for CCNA
+
+- **Subnetting:** Practice calculating subnets. `/24`, `/25`, `/26`, `/27` etc. — know the host ranges by heart.
+- **OSI Model:** 7 layers. Know what each layer does and what protocols live there.
+- **TCP vs UDP:** TCP is reliable (handshake, acknowledgements). UDP is fast (no handshake, fire and forget). HTTP uses TCP. DNS uses UDP (mostly).
+- **The TCP 3-way handshake:** SYN → SYN-ACK → ACK. This is how every TCP connection starts.
+- **ARP:** How a device finds the MAC address for an IP on the same subnet.
+- **Default gateway:** The router. Packets destined for outside the local subnet go to the default gateway.
+- **NAT:** Network Address Translation. How your home router lets multiple devices share one public IP. Crucial to understand — it's why cloudflared uses outbound connections.
+
+---
+
+## What to Say About Networking
+
+> *"The homelab uses Cloudflare Tunnel for all inbound traffic, which means no ports are open on the home router. All nine public subdomains have DNS pointing to Cloudflare, and a cloudflared connector on each host maintains a persistent outbound tunnel. Internally, services communicate over a Docker bridge network using container DNS. A Tailscale overlay network connects monk and kscloud1 for private database access — the shared Authentik Postgres is bound only to the Tailscale interface so it's never exposed to the public internet."*
--- a/concepts/oauth2-oidc.md
+++ b/concepts/oauth2-oidc.md
@ -0,0 +1,171 @@
+# OAuth2 and OIDC — How SSO Actually Works
+
+This is the concept that most people get wrong. Understanding it cold will impress any interviewer.
+
+---
+
+## The Problem SSO Solves
+
+Without SSO: 9 services = 9 separate user databases. To add a friend:
+- Create account in Forgejo
+- Create account in Grafana
+- Create account in Open WebUI
+- Create account in Kavita
+- ... 9 times
+
+To remove their access: 9 places to deactivate.
+
+With SSO: 1 account in Authentik. Access to all 9 services. Deactivate once.
+
+---
+
+## OAuth2 — Authorization, Not Authentication
+
+OAuth2 is commonly misunderstood. It was designed for **authorization** (what can you access?) not **authentication** (who are you?).
+
+**The core flow (Authorization Code):**
+
+```
+1. You click "Sign in with Authentik" in Grafana
+
+2. Grafana redirects your browser to Authentik:
+   GET https://auth.kitestacks.com/application/o/authorize/
+   ?client_id=grafana
+   &redirect_uri=https://grafana.kitestacks.com/login/generic_oauth
+   &response_type=code
+   &scope=openid email profile
+   &state=random_string_to_prevent_csrf
+
+3. Authentik presents login page
+   You enter username + password
+   Authentik validates credentials against its database
+
+4. Authentik redirects your browser BACK to Grafana:
+   GET https://grafana.kitestacks.com/login/generic_oauth
+   ?code=abc123xyz  ← authorization code (short-lived, one-time use)
+   &state=random_string  ← must match what Grafana sent in step 2
+
+5. Grafana's backend (not browser) calls Authentik directly:
+   POST https://auth.kitestacks.com/application/o/token/
+   client_id=grafana
+   client_secret=<secret>
+   code=abc123xyz
+   grant_type=authorization_code
+
+6. Authentik validates: code exists? client_secret correct? 
+   Returns: access_token + id_token (JWTs)
+
+7. Grafana reads the user's info from the token
+   Logs the user in
+```
+
+**Why the code exchange in steps 5-6?**
+The authorization code goes through the browser (URL redirect) — visible in browser history, logs, etc. The actual tokens go server-to-server (Grafana backend → Authentik). This keeps tokens out of the browser.
+
+---
+
+## The `invalid_grant` Bug — Explained
+
+This is the exact bug you hit and fixed. Now understand why:
+
+In step 4, Authentik stores the `code=abc123xyz` in its database.
+In step 5, Grafana sends that code to Authentik to exchange for tokens.
+
+With active-active failover:
+- Step 4 might hit monk's cloudflared connector → monk's Authentik → code stored in monk's Postgres
+- Step 5 might hit kscloud1's cloudflared connector → kscloud1's Authentik → looks for code in kscloud1's Postgres → NOT FOUND → `invalid_grant`
+
+**The fix:** Both Authentik instances share ONE Postgres database (on kscloud1, via Tailscale). The code is always found regardless of which connector handles each request.
+
+This is a real distributed systems problem — **stateful operations across load-balanced nodes** — and you diagnosed and fixed it.
+
+---
+
+## OIDC — Adding Identity on Top of OAuth2
+
+OAuth2 tells you what a user can access. It doesn't tell you who they are.
+
+OpenID Connect (OIDC) is a layer on top of OAuth2 that adds identity. It introduces the **ID Token** — a JWT (JSON Web Token) that contains claims about the user:
+
+```json
+{
+  "sub": "user-uuid-from-authentik",
+  "email": "kenpat7177@gmail.com",
+  "name": "kenpat",
+  "preferred_username": "kenpat7177",
+  "iat": 1234567890,
+  "exp": 1234571490,
+  "iss": "https://auth.kitestacks.com/application/o/kavita/"
+}
+```
+
+The app reads these claims to create or update the user's local account. This is why when you log into Grafana with Authentik, Grafana knows your username and email without you creating a separate Grafana account.
+
+---
+
+## JWT — What It Is
+
+A JWT (JSON Web Token) is a signed, base64-encoded string with three parts:
+
+```
+eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJ1c2VyaWQiLCJlbWFpbCI6ImtlbnBhdEBraXRlc3RhY2tzLmNvbSJ9.SIGNATURE
+     HEADER                    PAYLOAD (claims)                          SIGNATURE
+```
+
+The signature is created by Authentik using a private RSA key. Any app can verify it using Authentik's public key (available at the JWKS endpoint). This means apps can validate tokens without calling Authentik again.
+
+**What this means practically:** Authentik signs the JWT. Grafana doesn't need to call Authentik to validate it — it just checks the signature with Authentik's public key. This is stateless authentication.
+
+---
+
+## The Discovery Document
+
+Every OIDC provider exposes a discovery document:
+
+```
+https://auth.kitestacks.com/application/o/grafana/.well-known/openid-configuration
+```
+
+This JSON file tells any OIDC client where to find every endpoint:
+- `authorization_endpoint` — where to send the user to log in
+- `token_endpoint` — where to exchange codes for tokens
+- `userinfo_endpoint` — where to get user details
+- `jwks_uri` — where to find the public keys for JWT validation
+
+Apps use this URL to auto-configure themselves. That's why you set `OPENID_PROVIDER_URL` in Open WebUI and it figured out the rest.
+
+---
+
+## Redirect URI — Why It Must Match Exactly
+
+In step 2 of the flow, Grafana sends:
+```
+&redirect_uri=https://grafana.kitestacks.com/login/generic_oauth
+```
+
+Authentik checks: is this redirect URI registered for this client? If not, it refuses — this prevents attackers from creating a malicious app that tricks users into sending their authorization codes to an attacker's server.
+
+This is why Karakeep broke until you fixed the redirect URI. Karakeep uses NextAuth.js with provider ID `custom`, so the actual callback path was `/api/auth/callback/custom` — not `/api/auth/callback/authentik`. The URI had to match exactly.
+
+---
+
+## Provider Patterns in This Homelab
+
+**Native OIDC** (app handles SSO itself):
+
+| App | How it works |
+|-----|-------------|
+| Grafana | Generic OAuth2 env vars (`GF_AUTH_GENERIC_OAUTH_*`) |
+| Open WebUI | `OAUTH_CLIENT_ID` + `OPENID_PROVIDER_URL` env vars |
+| Kavita | Settings UI → OIDC section (must use UI, not database) |
+| Karakeep | NextAuth.js with `custom` provider ID in `.env` |
+| Forgejo | OAuth2 authentication source in Forgejo admin UI |
+
+**The Authentik proxy pattern** (for apps with no native SSO support):
+Authentik acts as a reverse proxy in front of the app. The user authenticates with Authentik, and only authenticated requests are forwarded to the app. Uptime Kuma and Prometheus use this pattern (not yet fully deployed — requires Cloudflare route update).
+
+---
+
+## What to Say About SSO
+
+> *"I implemented single sign-on across all nine services using Authentik as the OIDC identity provider. Each service is registered as an OAuth2 client with a unique client ID and redirect URI. The OAuth2 authorization code flow means user credentials only ever go to Authentik — other services receive a signed JWT and never see the password. I hit a distributed systems issue in production where authorization codes were being invalidated by active-active load balancing across two hosts — I diagnosed it by tracing the OAuth2 flow and fixed it by sharing a single Postgres database between both Authentik instances over a private Tailscale network."*