kitestacks-homelab/homelab-mastery/build-guide/with-ai/08-monitoring.md

Step 8 — Monitoring

Track: With AI (Beginner)
Time for this step: 2–3 hours

Monitoring means knowing when something is wrong before your users tell you. In this step you will set up three layers of monitoring:

1. Grafana — beautiful dashboards showing CPU, RAM, disk, and network over time
2. Uptime Kuma — checks every 60 seconds that each service responds correctly
3. Conky — a desktop widget on your home computer showing live kscloud1 status

---

Monitoring Layer 1 — Grafana + Prometheus

You already deployed Grafana and Prometheus in Step 5. Now configure them properly.

Edit the Prometheus Config

Prometheus needs to know where to collect metrics from. Tell it about both machines:

nano ~/kitestacks-live/docker/prometheus/prometheus.yml

Add this content:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'monk-node'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance: 'monk'

  - job_name: 'kscloud1-node'
    static_configs:
      - targets: ['YOUR_VPS_IP:9100']
        labels:
          instance: 'kscloud1'

Replace YOUR_VPS_IP with your VPS's public IP address.

On kscloud1, make sure node-exporter is configured to be reachable publicly:

# In node-exporter's docker-compose.yml on kscloud1
ports:
  - "0.0.0.0:9100:9100"

Restart Prometheus:

cd ~/kitestacks-live/docker/prometheus
docker compose restart prometheus

Configure Grafana Provisioning

Tell Grafana to automatically load Prometheus as a data source and load the Node Exporter Full dashboard:

Create ~/kitestacks-live/docker/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: 000000001
    url: http://prometheus:9090
    isDefault: true

Create ~/kitestacks-live/docker/grafana/provisioning/dashboards/dashboards.yml:

apiVersion: 1
providers:
  - name: default
    folder: KiteStacks
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

The Node Exporter Full dashboard (id 1860) can be imported from Grafana's dashboard library:

1. Log in to grafana.yourdomain.com
2. Left menu → Dashboards → Import
3. Enter ID: 1860
4. Select your Prometheus datasource
5. Import

You should now see CPU, RAM, disk, and network graphs for both monk and kscloud1. Switch between them using the "instance" dropdown at the top of the dashboard.

---

Monitoring Layer 2 — Uptime Kuma

You set up Uptime Kuma in Step 5. Now add monitors for all your services.

Log in to status.yourdomain.com and add an HTTP monitor for each service:

Monitor Name	URL	Check Interval
Main Website	https://www.yourdomain.com	60s
Authentik	https://auth.yourdomain.com	60s
Forgejo	https://gitforge.yourdomain.com	60s
KiteAI	https://ai.yourdomain.com	60s
Karakeep	https://links.yourdomain.com	60s
Kavita	https://kavita.yourdomain.com	60s
Grafana	https://grafana.yourdomain.com	60s
BookStack	https://wiki.yourdomain.com	60s
OSTicket	https://tasks.yourdomain.com	60s
Portainer	https://portainer.yourdomain.com	60s
kscloud1	(ping to kscloud1 IP)	60s
Monk	(ping to monk's Tailscale IP)	60s

Then create a Status Page:

1. Status Pages → New Status Page
2. Title: "KiteStacks Status"
3. Slug: homelab
4. Add all monitors to it

Push Uptime Kuma to kscloud1:

The Conky widget on your desktop reads kscloud1's Uptime Kuma, not monk's. Push monk's database to kscloud1 after setting up monitors:

Ask your AI: "How do I copy a Docker named volume's SQLite database from one machine to another using Python's sqlite3.backup() method?"

---

Monitoring Layer 3 — Conky Desktop Widget

Conky is a program that draws information on your desktop background in real time. Your KiteStacks widget shows whether each service on kscloud1 is up (green dot) or down (red dot), refreshed every 15 seconds.

Install Conky

sudo apt install conky-all

Install the Widget Script

The widget script reads Uptime Kuma's API and formats the output for Conky. The script is at ~/.local/bin/kitestacks-uptime-widget.sh in the homelab repo.

Copy it to your machine:

mkdir -p ~/.local/bin
cp ~/kitestacks-homelab/apps/conky/kitestacks-uptime-widget.sh ~/.local/bin/
chmod +x ~/.local/bin/kitestacks-uptime-widget.sh

Edit the script to use your kscloud1's Tailscale IP:

nano ~/.local/bin/kitestacks-uptime-widget.sh

Change the KUMA_URL line:

KUMA_URL="http://100.123.x.x:3001"   # kscloud1's Tailscale IP

Enable the Conky Config

cp ~/kitestacks-homelab/apps/conky/kitestacks-uptime.conf ~/.config/conky/kitestacks-uptime.conf
conky -c ~/.config/conky/kitestacks-uptime.conf -d

The widget should appear in the top-right corner of your desktop, showing a dot for each service — green for up, red for down.

Ask your AI: "How do I make Conky start automatically when I log in to my Ubuntu desktop?"

---

Setting Up Alerts

Uptime Kuma can send you a notification on your phone when a service goes down.

Option 1: ntfy (recommended — self-hosted) You have ntfy running as a container. Set up an ntfy notification in Uptime Kuma:

• Notification Type: ntfy
• URL: your ntfy server URL
• Topic: choose a topic name (e.g., homelab-alerts)

Install the ntfy app on your phone and subscribe to your topic.

Option 2: Email Configure email notifications in Uptime Kuma using your email address.

Ask your AI: "How do I configure Uptime Kuma to send notifications via ntfy?"

---

Checkpoint

• Prometheus is collecting metrics from both monk and kscloud1
• Grafana shows Node Exporter Full dashboard with both hosts
• Uptime Kuma has monitors for all 11 services
• Uptime Kuma status page is live at status.yourdomain.com/status/homelab
• Uptime Kuma database has been pushed to kscloud1
• Conky widget is showing on your desktop with live service status
• You receive a notification when you manually pause a service in Uptime Kuma

---

Congratulations — Your Homelab Is Complete

You have built a production homelab with:

• 11 self-hosted services running in Docker
• Single sign-on via Authentik
• Cloud failover on a Hetzner VPS
• Private networking over Tailscale
• Real-time monitoring via Grafana and Uptime Kuma
• A live desktop status widget

Everything you built here maps directly to enterprise cloud engineering skills. Every concept has a certification that covers it in depth.

Your next step: certifications/roadmap.md