kenpat/kitestacks-homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs: correct disaster recovery runbook (monk primary, kscloud1 active-active, Forgejo activity fix)

Browse source
This commit is contained in:
kenpat 2026-06-11 18:46:09 +00:00
parent 8a014d27bc
commit dbbb776808
1 changed files with 45 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

65
docs/disaster-recovery/RUNBOOK.md
View file

@ -2,23 +2,33 @@
## Purpose ## Purpose
This document describes how to restore the entire KiteStacks platform if the primary server (Assassin) fails. This document describes how to restore the entire KiteStacks platform if a
host fails. As of 2026-06-10, KiteStacks runs active-active across two hosts
plus Cloudflare Tunnel, so no single host is a hard dependency for the site
to stay up.
## Current Infrastructure ## Current Infrastructure
Primary Production: Primary Production:
- Host: Assassin - Host: monk
- IP: 192.168.1.205 - LAN IP: 192.168.1.205
Cloud Backup: Cloud Failover (PERMANENT, active-active - NOT cold standby):
- Host: kscloud1 - Host: kscloud1 (Hetzner VPS)
- Public IP: 5.78.233.28 - Public IP: 5.78.233.28
- Tailscale IP: 100.123.254.52
- Runs a full replica of all 9 services
assassin (T14): retired/OFF, no longer part of the topology.
Domains: Domains:
- www.kitestacks.com - www.kitestacks.com (+ ai, auth, gitforge, grafana, kavita, links, status, tasks)
- gitforge.kitestacks.com - www-backup.kitestacks.com / git-backup.kitestacks.com (kscloud1 direct
- www-backup.kitestacks.com A-records via local Caddy on port 80, separate from the Tunnel)
- git-backup.kitestacks.com
Cloudflare Tunnel:
- 3 connectors load-balance ACTIVE-ACTIVE across all 9 *.kitestacks.com
hostnames - no primary/backup priority.
## Recovery Priority ## Recovery Priority
@ -31,20 +41,35 @@ Domains:
## Current Backup Status ## Current Backup Status
Website Backup: Website:
- Operational on kscloud1 - Full replica running on kscloud1, served live via the Tunnel.
Forgejo Backup: Forgejo:
- Operational on kscloud1 - Full replica running on kscloud1, but with a SEPARATE database - repos and
commits pushed to monk's Forgejo do NOT appear on kscloud1's Forgejo (and
vice versa). Accepted tradeoff for uptime.
- The portal's Recent Activity widget on BOTH hosts queries monk's Forgejo
directly (FORGEJO_API_BASE -> http://100.85.209.116:3006 over Tailscale
from kscloud1, http://localhost:3006 on monk) so it stays consistent
regardless of which connector serves the page.
Git Repository: Authentik:
- Synced to both Forgejo instances - Shared Postgres+Redis hosted on kscloud1, reachable only over Tailscale
(100.123.254.52). Both monk's and kscloud1's authentik+worker use this
single database/cache - fixes invalid_grant SSO caused by active-active
routing splitting an OAuth flow across connectors.
Other stateful apps (kavita, karakeep, openproject, etc.):
- Fresh/separate data on kscloud1 - may show different/stale data depending
on which connector serves a request. Accepted as the cost of guaranteed
uptime.
## Validation Checklist ## Validation Checklist
- [ ] Website accessible - [ ] Website accessible (www.kitestacks.com)
- [ ] Backup website accessible - [ ] kscloud1 replica accessible (www-backup.kitestacks.com)
- [ ] Forgejo operational - [ ] Forgejo operational (gitforge.kitestacks.com)
- [ ] Backup Forgejo operational - [ ] kscloud1 Forgejo replica operational (git-backup.kitestacks.com)
- [ ] Authentik SSO works (auth.kitestacks.com)
- [ ] Cloudflare DNS verified - [ ] Cloudflare DNS verified
- [ ] Cloudflare Tunnel verified - [ ] Cloudflare Tunnel: all 3 connectors healthy