homelab-mastery/concepts/linux.md

# Linux — Commands and Concepts to Own

You've been running Linux commands without fully owning them. This fixes that.

---

## The Filesystem

Everything in Linux is a file. The filesystem tree starts at `/` (root):

```
/
├── etc/         Configuration files (system-wide)
├── home/        User home directories (/home/kenpat)
├── opt/         Optional/third-party software (kscloud1 services live here)
├── proc/        Virtual filesystem — running processes, kernel info
│   ├── uptime       System uptime in seconds
│   └── net/route    Routing table (used by metrics API to find active interface)
├── sys/         Virtual filesystem — hardware/kernel info
├── var/         Variable data — logs, databases, cache
└── usr/         User programs and libraries
```

---

## File Permissions

Every file has three permission sets: **owner**, **group**, **others**.

```bash
ls -la ~/kitestacks-live/docker/kavita/config/kavita.db
-rw-r--r-- 1 kenpat kenpat 2.4M Jun 11 kavita.db
```

Breaking it down:
- `-` — it's a file (not a directory `d` or symlink `l`)
- `rw-` — owner (kenpat): read + write
- `r--` — group (kenpat): read only
- `r--` — others: read only

**chmod** changes permissions. **chown** changes owner.

Why this mattered: When syncing kavita.db to kscloud1, you ran `chown 1000:1000` because the Kavita container runs as user ID 1000. If the file is owned by the wrong user ID, the container can't write to it.

```bash
chmod 644 kavita.db      # rw-r--r--
chmod 755 script.sh      # rwxr-xr-x (executable)
chown 1000:1000 kavita.db  # set owner to UID 1000, GID 1000
chown -R kenpat:kenpat ./  # recursive (-R) on a directory
```

---

## Processes

```bash
ps aux                   # all running processes
ps aux | grep forgejo    # find forgejo processes
kill 1234               # send SIGTERM to PID 1234 (polite stop)
kill -9 1234            # send SIGKILL (force kill, no cleanup)
```

**systemctl** manages systemd services (services that start on boot):
```bash
systemctl status tailscaled     # is tailscale running?
systemctl restart tailscaled    # restart it
systemctl enable tailscaled     # start on boot
journalctl -u tailscaled        # logs for tailscale service
```

Your containers don't use systemd — Docker manages them with `restart: unless-stopped`.

---

## Networking Commands

```bash
# What ports is this machine listening on?
ss -tlnp                 # TCP listening, numeric, with process
ss -tlnp | grep :3006   # is Forgejo's port bound?

# Test connectivity
ping 8.8.8.8             # can I reach Google DNS?
curl -I https://auth.kitestacks.com  # HTTP headers from Authentik
curl -s http://localhost:8000/api/health  # test metrics API

# DNS lookup
dig www.kitestacks.com   # full DNS query details
nslookup gitforge.kitestacks.com  # simpler DNS lookup

# Firewall
sudo ufw status          # what rules are active?
sudo ufw allow 22/tcp    # allow SSH
sudo ufw deny 3306/tcp   # block MySQL from outside
```

---

## Piping and Redirection

The `|` (pipe) sends output of one command as input to another:

```bash
docker ps | grep forgejo           # filter docker ps output
cat prometheus.yml | grep job      # find job lines in config
docker logs authentik 2>&1 | grep ERROR  # show only errors
```

`2>&1` redirects stderr (error output, stream 2) to stdout (stream 1) — so errors appear in the same stream as normal output and can be piped.

`>` redirects output to a file (overwrites):
```bash
pg_dump authentik > authentik-backup.sql
```

`>>` appends to a file:
```bash
echo "new line" >> ~/.ssh/config
```

---

## SSH

SSH (Secure Shell) gives you a terminal session on a remote machine.

```bash
ssh kenpat@5.78.x.x                    # basic SSH
ssh -i ~/.ssh/id_ed25519_kscloud1 kenpat@5.78.x.x  # specify key
ssh -L 5099:localhost:5000 kenpat@5.78.x.x          # local port forward
```

**Local port forward** (`-L local:remote_host:remote_port`):
`ssh -L 5099:localhost:5000 kenpat@kscloud1` means:
- Traffic to YOUR localhost:5099
- Gets tunneled through the SSH connection
- And hits kscloud1's localhost:5000

You used this to access kscloud1's Kavita instance (running on port 5000) from your browser at http://localhost:5099 — without opening that port to the internet.

---

## sudo and Non-Interactive Usage

`sudo` runs a command as root. It normally prompts for your password.

**The kscloud1 problem:** In automated scripts, there's no terminal to enter a password. Solution:
```bash
echo YOUR_PASSWORD | sudo -S command
# -S reads password from stdin instead of terminal
```

You used this to run ufw commands non-interactively. In real production environments, this is handled differently (sudoers file with NOPASSWD for specific commands, or SSH key-based service accounts).

---

## Grep, Sed, Awk

**grep** finds lines matching a pattern:
```bash
grep "error" /var/log/syslog          # lines containing "error"
grep -i "error" logfile               # case-insensitive
grep -n "AUTHENTIK" docker-compose.yml # show line numbers
grep -r "p12217177" /opt/kitestacks/  # recursive search in directory
```

**sed** (stream editor) modifies text:
```bash
sed 's/old_text/new_text/g' file.txt  # replace all occurrences
sed -i 's/old/new/g' file.txt         # -i edits file in place
```

**awk** processes structured text (columns):
```bash
grep PG_PASS .env | cut -d= -f2-      # get value after = (including trailing =)
# cut -d= splits on =, -f2- means "field 2 and everything after"
# This is why: PG_PASS=abc= → if you use -f2, you get "abc" (loses trailing =)
# With -f2-, you get "abc=" (correct)
```

---

## What to Say About Linux

> *"All services run on Linux hosts. I'm comfortable with file permissions, process management, SSH configuration (including local port forwarding for secure access to non-exposed services), firewall rules with ufw, and command-line tools like grep, curl, and docker CLI. I diagnosed and fixed a network configuration issue on the cloud VPS where ufw's default-deny policy was blocking Docker container traffic to a host-network-mode service."*