Observability

Being built. This is the design for the roadmap item Alerts to chat. The proof goes here once it works.

Today a failed backup, Argo CD sync or crashing pod shows up only if I look. The goal: cluster health in Grafana behind the same login as everything else, and every failure, plus every backup result, posted to chat.

How messages flow

flowchart LR
  subgraph cluster["k3s cluster"]
    subgraph jobs["Nightly backups"]
      GJ["Gitea backup Job
tar → upload"]
      MJ["MinIO backup Job
tar → upload"]
      PB["Postgres Backup
zitadel-db, postgres"]
    end
    JN["job-notifier
any opted-in job"]
    KSM["kube-state-metrics
+ app metrics"]
    P["Prometheus
alert rules"]
    AM["Alertmanager"]
    G["Grafana"]
    AR["Argo CD
Notifications"]
  end
  Z["Zitadel login"]
  HC["healthchecks.io"]
  subgraph slack["Slack"]
    SB["backups channel"]
    SA["alerts channel"]
    SD["deploys channel"]
  end
  Mail["Email"]
  Me(("Me"))

  GJ -- "done / failed" --> JN
  MJ -- "done / failed" --> JN
  PB -- "completed / failed" --> JN
  JN -- "✅ / ❌ every run" --> SB

  jobs -. "last success time" .-> KSM
  KSM --> P
  P -- "missed backup, notifier down,
WAL failing, pods, nodes, disks" --> AM
  AM --> SA
  AM -- "Watchdog ping, every 5 min" --> HC
  HC -- "pings stopped" --> SA
  HC -- "pings stopped" --> Mail

  AR -- "sync failed, degraded" --> SD

  Me -- "browser" --> Z --> G
  G --> P

What runs

• kube-prometheus-stack: Prometheus, Alertmanager and Grafana, with node-exporter, kube-state-metrics, the standard Kubernetes dashboards and about 150 alert rules. One upstream chart, pinned, deployed by Argo CD in the platform group.

• Grafana logs in through Zitadel, like every other app. The Zitadel roles grafana-admin and grafana-viewer map to Grafana's own roles. Grafana keeps no database: its data sources and dashboards come from git.

• Prometheus and Alertmanager have no public address. They have no login of their own, so they're reached with kubectl port-forward, which already needs the Zitadel login.

• The metrics the apps already have:

  App	Metrics	Comes with
  Gitea	/metrics, with a scrape token	a Grafana dashboard
  MinIO	its cluster metrics endpoint, with a scrape token	its official dashboard and alerts
  CloudNativePG	one monitor per database cluster	its own dashboard and alert rules, including WAL archiving and backup age
  Argo CD	its chart's ServiceMonitors	a dashboard; alerts when an app stays out of sync
  Traefik, cert-manager	their charts' metrics	dashboards; an alert when a certificate is 14 days from expiry

• Slack, free plan. One app, homelab, with a webhook per channel: #alerts, #deploys and #backups. The free plan keeps 90 days of history and allows 10 apps; this uses 2 (homelab, and Healthchecks).

Job messages: job-notifier

Prometheus alerts on bad states, so it can't say "the backup worked". job-notifier is a small Go controller that watches the cluster and posts one message per finished run, pass or fail. Backups are its first users, but any job can opt in.

A job opts in with metadata, not code, and needs no Slack secret of its own:

# on the CronJob's jobTemplate: a CronJob's own labels aren't copied to its Jobs
metadata:
  labels:
    notify.obiwalker.com/channel: backups      # opts in, and picks the channel
  annotations:
    notify.obiwalker.com/on: all               # all (the default) or failure
    notify.obiwalker.com/title: Gitea backup   # optional; defaults to the CronJob's name

Why a label and annotations. Labels are what Kubernetes can select on; annotations hold settings.

• The label is the opt-in. The notifier watches with a label selector, so the API server sends it only the Jobs that opted in, never every Job in the cluster. kubectl get jobs -A -l notify.obiwalker.com/channel lists everything that reports.

• Annotations hold free text: a title can have spaces, and on is a setting, not something to select on.

• Prometheus reads the same label. kube-state-metrics exports it, so the "no successful run in 26 hours" alert covers exactly the jobs that opted in. There's no second list to keep.

• Details without coupling. If a container writes a short summary to its termination message, the notifier adds it. The backup upload steps write the file name and size. A job that writes nothing still gets its name, duration and result.

• Why outside the job. A job can't report every way it fails. Gitea's tar step runs before the upload, so if tar fails, a message sent at the end of the job never goes out. The same goes for a job killed for memory or time. Watching from outside catches all of them, and the Slack webhook is sealed once, not in every namespace.

• No duplicates. After posting, the notifier marks the object with an annotation, so a restart doesn't post the same message twice.

Why a controller and not an operator. An operator owns a custom resource and keeps something in a desired state. The notifier owns nothing: it watches and reports. A custom resource would add an API to version and keep up, and labels already do the job. It's built on controller-runtime, the library operators use, so an operator remains for an idea that really owns state.

Alertmanager

• Routing by severity: critical and warning go to #alerts; info is dropped.

• Grouping: by alert and namespace, so ten crashing pods in one app are one message, not ten.

• Inhibition: a node that's down mutes the pod alerts on that node, and a critical alert mutes its matching warning. One cause, one message.

• Repeats: an alert that's still firing is posted again every 4 hours if critical, every 24 hours if a warning. Resolved messages are on.

• Quiet hours: warnings wait from 22:00 to 07:00. Critical alerts always go through.

• Links: each message links to its Grafana dashboard and its section of the runbook.

• Silences for planned work, with amtool over port-forward.

The dead man's switch

If the cluster is down, Alertmanager is down with it and can't say so. kube-prometheus-stack has an alert, Watchdog, that is always firing on purpose. Alertmanager sends it to healthchecks.io every 5 minutes, and each ping proves that Prometheus, Alertmanager and the internet connection all work.

If the pings stop for 10 minutes, healthchecks.io posts to #alerts and emails me. It talks to Slack directly, from outside the house, so the message arrives with the power or the internet out. It posts again when the pings come back.

It has to run outside the cluster to mean anything, so it isn't self-hosted. The free plan has 20 checks; this uses one.

What the messages look like

#backups
 ▌✅ Gitea backup                                        04:02
 ▌gitea-2026-10-22T040000Z.tar.gz · 604 MB · 2m41s

 ▌❌ MinIO backup failed                                 04:31
 ▌Container upload exited 1 after 12s
 ▌> upload failed: AccessDenied when calling PutObject
 ▌kubectl -n tools logs job/minio-backup-29342430

 ▌✅ Postgres backup: zitadel-db                         05:00
 ▌Backup 20261022T050000 · 3m12s

#alerts
 ▌🔥 FIRING · critical · KubePodCrashLooping             14:07
 ▌Pod obiwalker/gitea-7c9… restarted 5 times in 10 min · node k3s-2
 ▌📊 Dashboard  ·  📖 Runbook

 ▌✅ RESOLVED · KubePodCrashLooping                      14:21
 ▌Firing for 14 min

 ▌⚠️ FIRING · warning · BackupMissed                     06:00
 ▌CronJob tools/minio-backup has no successful run in 26 h

#deploys
 ▌❌ Sync failed: gitea                                  10:12
 ▌Revision 3f2a91c · "bump gitea to 1.27.4"
 ▌🔗 Open in Argo CD

The times and names are examples; real messages will replace them once it runs.

Trade-offs and known limits

• One Prometheus, on one node's disk. Its data doesn't support NFS, so it uses local disk: 15 days of history, capped at 20 GB, on a 30 GB volume. The smallest node has about 200 GB free, so the cap is there to stop runaway growth, not because space is short. If that node is lost, history starts again; the alert rules and dashboards are in git.

• Grafana changes made in the browser don't last. Dashboards come from git, so a change made in the UI is lost at the next restart unless it's committed.

• job-notifier is one more thing to run. If it's down, no backup messages arrive. Prometheus alerts when it's down, and the 26-hour rule catches a missed backup either way.

Next

• Tekton pipeline runs through job-notifier, once backups prove the pattern.