Compute & networking

What runs

k3s on three nodes (k3s-1, k3s-2, k3s-3), on Debian 13.
kube-vip in ARP mode, which gives LoadBalancer Services a stable address on the home network. k3s's built-in ServiceLB is turned off from the first install.
Traefik ingress on one kube-vip address. Every public hostname resolves to it, and Traefik gets certificates from Let's Encrypt. It sees each visitor's real IP and logs every request as JSON.
CrowdSec in front of every site: it bans attackers' IPs, blocks requests that exploit known vulnerabilities, and records what a full web application firewall would catch (Edge protection, below).

Why this choice

k3s is a full Kubernetes distribution in one small binary, and that suits a handful of small machines.

TODO: why k3s over kubeadm or Talos.

kube-vip gives ingress one address that doesn't belong to any single node. One kube-vip pod, chosen by leader election, announces the address. If that node goes down, another pod takes over the announcement.

TODO: why kube-vip over ServiceLB, in your words.

Edge protection

internet ─▶ router ─▶ VIP (kube-vip) ─▶ Traefik ─▶ CrowdSec bouncer (plugin)
                                                     ├─ banned IP? ─▶ 403
                                                     └─ AppSec (WAF) ─▶ known exploit? ─▶ 403
                                                                      └─ OWASP CRS ─▶ recorded only
CrowdSec agent ◀── Traefik's access logs        CrowdSec API ◀── community blocklist

Three layers, each switched on only after it was proven:

Layer	What it does	Blocks?
IP reputation	bans IPs that scan or brute-force, from Traefik's logs, plus IPs that thousands of other CrowdSec users have reported	yes
Virtual patching	blocks requests that exploit known CVEs, with rules updated daily, often before the app itself is patched	yes
OWASP Core Rule Set	flags generic attacks (SQL injection, XSS, path traversal)	not yet: it only records

Why CrowdSec. I wanted Coraza, the open-source WAF engine that runs the OWASP Core Rule Set, in front of Traefik. Its Traefik plugin is no longer maintained for production, by its author's own note, and Traefik's native Coraza is only in the commercial Traefik Hub. CrowdSec's WAF runs Coraza inside, is maintained, and adds the IP-reputation layer.

Real client IPs first. By default, Kubernetes rewrites every visitor's address on the way in, so Traefik saw one internal IP for everyone and wrote no access logs at all (a setting was silently ignored). An IP ban is useless without real IPs, so that came first: access logs on, and the Service set to deliver traffic only to Traefik's own node (externalTrafficPolicy: Local). Minutes later, the logs showed a scanner probing the bare home IP.

Fails open. If CrowdSec is down, requests pass uninspected rather than every site going dark, and an alert says so. For a homelab, availability wins. The plugin's default is the opposite, so this is set explicitly.

Home is never banned, but still inspected. Devices at home, and pods calling the lab's own sites, reach them through the router, so they arrive as the home IP. That IP is on a never-ban list; it isn't trusted to skip inspection, so a stale entry after the ISP changes it can't open a hole.

Nothing downloaded at start-up. Traefik plugins normally download when Traefik starts. With CrowdSec on every route, a failed download would take every site down, so the plugin's source is pinned in git and unpacked into Traefik's pod.

What this proves

Checked on 2026-10-23, from a phone on cellular and from home:

Check	Result
Client IPs before, and after `Local`	every request was `10.42.0.0`; after, the phone's carrier IP, and a scanner within minutes
Request for `/.env` (a secrets probe)	`403`, recorded as a blocked exploit
Banning the phone's IP	`403` on protected sites within 60 seconds; unprotected sites unaffected
OWASP CRS on the same probe	recorded as a file-inclusion attempt, nobody banned
Every site, a login through Zitadel, `git push`, a CI image push to the registry	all pass
CrowdSec's WAF made unreachable	every site still `200` at normal speed; protection back 2 seconds after

Trade-offs and known limits

One network. ARP mode only works when every node shares one Layer 2 network. Across routed networks, kube-vip needs BGP mode.
Same interface name on every node. kube-vip runs as one DaemonSet, so every pod gets the same interface setting. Debian names interfaces by hardware position, so each node is renamed to lan0 first. Without that, the address stops working whenever a node with a different name wins the election.
Traefik config lives in k3s. k3s installs Traefik itself. Changes go in a HelmChartConfig so k3s keeps them, not in edits that k3s would revert.
Real IPs cost a short blip when Traefik moves. With Local, the address follows the Traefik pod, so a node reboot or drain means about 8 seconds of downtime while it moves.
A kube-vip race took the sites down once. Switching the Service to Local through k3s's config file updated it twice in quick succession, and kube-vip left the address unannounced: a 3½-minute outage. I reproduced it on a throwaway Service, found the fix (restart the kube-vip pod on the wrong node, about 9 seconds), then made the real change with a single update and no downtime.
New bans take up to a minute. Traefik keeps a local copy of the ban list and refreshes it every 60 seconds, so no request ever waits on CrowdSec.
The Core Rule Set only records, for now. Its first day flagged ordinary git push traffic, so it needs Gitea's git endpoints excluded before it's allowed to ban. Large uploads are inspected up to 10 MB.