ops: add nightly Docker volume backup to SAMURAI

- scripts/backup-volumes.sh: tar each named volume via alpine, rsync to SAMURAI (Tailscale 100.74.x.x) at 02:00; 7-day retention; preflight checks Tailscale + SSH before starting - scripts/setup-samurai-ssh.sh: one-time SSH key install to SAMURAI - scripts/monk-backup.{service,timer}: systemd units for nightly schedule - docs/backup-setup.md: full setup instructions incl. Windows OpenSSH config and admin authorized_keys fix Phase 2 (MinIO S3 on SAMURAI) tracked as TODO in backup-volumes.sh. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 02:59:14 -05:00 · 2026-06-19 02:59:14 -05:00 · 5b3698191e
commit 5b3698191e
parent 4c28ed131a
5 changed files with 280 additions and 0 deletions
--- a/docs/backup-setup.md
+++ b/docs/backup-setup.md
@ -0,0 +1,100 @@
+# Docker Volume Backup: monk → SAMURAI
+
+Nightly rsync of named Docker volumes to SAMURAI (Windows 11, Tailscale).
+
+## Architecture
+
+```
+monk (T14s)
+└── Docker named volumes
+    ├── kite-ai_open-webui
+    ├── osticket_osticket_db / osticket_uploads
+    ├── portainer_data
+    ├── prometheus_prometheus-data
+    └── uptime-kuma_uptime-kuma
+         │
+         │  tar.gz via alpine container
+         │  rsync over SSH (Tailscale)
+         ▼
+SAMURAI (Windows 11, 100.74.x.x)
+└── C:\KiteBackups\monk\<TIMESTAMP>\
+    ├── kite-ai_open-webui.tar.gz
+    ├── osticket_osticket_db.tar.gz
+    └── ...
+
+7-day retention (older dirs pruned automatically)
+```
+
+## Phase 2 (TODO)
+
+Deploy MinIO on SAMURAI and push archives as S3 objects using `mc put`.
+
+## One-time setup
+
+### 1. Enable OpenSSH Server on SAMURAI
+
+In PowerShell (admin):
+```powershell
+Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
+Start-Service sshd
+Set-Service -Name sshd -StartupType Automatic
+# Allow Tailscale traffic (adjust rule name if needed)
+New-NetFirewallRule -Name "sshd-tailscale" -DisplayName "OpenSSH via Tailscale" -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22 -RemoteAddress 100.64.0.0/10
+```
+
+### 2. Install SSH key from monk
+
+```bash
+cd ~/kitestacks-homelab
+SAMURAI_USER=kenpat bash scripts/setup-samurai-ssh.sh
+```
+
+If your SAMURAI account is in the Administrators group, Windows ignores
+`~\.ssh\authorized_keys`. Run this in PowerShell admin instead:
+
+```powershell
+$key = Get-Content "$env:USERPROFILE\.ssh\authorized_keys" -ErrorAction SilentlyContinue
+if (-not $key) { $key = Get-Content "$env:ProgramData\ssh\authorized_keys" }
+Add-Content -Force "$env:ProgramData\ssh\administrators_authorized_keys" $key
+icacls "$env:ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "SYSTEM:(F)" /grant "BUILTIN\Administrators:(F)"
+```
+
+### 3. Create backup directory on SAMURAI
+
+```powershell
+New-Item -ItemType Directory -Path "C:\KiteBackups\monk" -Force
+```
+
+### 4. Install systemd units on monk
+
+```bash
+sudo cp ~/kitestacks-homelab/scripts/monk-backup.service /etc/systemd/system/
+sudo cp ~/kitestacks-homelab/scripts/monk-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now monk-backup.timer
+```
+
+Verify:
+```bash
+systemctl list-timers monk-backup.timer
+# Run immediately to test:
+sudo systemctl start monk-backup.service
+journalctl -u monk-backup.service -f
+```
+
+## Logs
+
+```bash
+tail -f /var/log/kitestacks/backup-volumes.log
+```
+
+## Restore a volume
+
+```bash
+# Copy archive back from SAMURAI
+scp -i ~/.ssh/id_ed25519_samurai kenpat@100.74.x.x:/cygdrive/c/KiteBackups/monk/<TIMESTAMP>/osticket_osticket_db.tar.gz /tmp/
+
+# Restore into a volume
+docker run --rm -v osticket_osticket_db:/target alpine sh -c \
+  "cd /target && tar xzf -" < /tmp/osticket_osticket_db.tar.gz
+```