Storage

What runs

Two StorageClasses, each for a different kind of data:

StorageClass Backed by Used for
obiwalker-nfs (default) NFS from nfs-1, through the NFS CSI driver App files: Gitea repositories, MinIO, gateway data
local-path Each node's own disk Databases: every Postgres instance

Both use reclaimPolicy: Retain. Deleting a claim leaves the data in place until I remove it by hand. Nothing is lost to a mistyped command.

Why local disk was added: redundancy

At first, everything lived on NFS, Postgres included. That had two problems:

  1. CloudNativePG advises against running Postgres on NFS.

  2. One instance on nfs-1 was a single point of failure. If the NFS server or the one database pod went down, so did every app that needed the database.

The fix moves where redundancy comes from: from the disk to the database. Each Postgres cluster now runs 3 instances, each on a different node (required pod anti-affinity), on that node's local disk. CloudNativePG's streaming replication keeps them in sync. If a node is lost, its instance is rebuilt from the others, not reattached. Databases no longer depend on nfs-1 at all.

The move, measured

The move happened in place, with no dump and restore:

  1. Take a backup.

  2. Add replicas on local disk.

  3. Switch the primary over to one of them.

  4. Destroy the old NFS instance and let it rebuild on local disk.

Apps connect through the -rw Service, which follows the primary, so no app config changed.

On the first cluster (Zitadel's, 2026-10-16), the app saw:

Event Database errors
In-place primary restart (the default) ~25 s
Switchover to a replica ~13 s

So every cluster now sets primaryUpdateMethod: switchover. Rolling updates promote a replica instead of restarting the primary.

TODO: confirm the shared postgres cluster's move is finished. The runbook still shows it in progress.

Trade-offs and known limits

  • Local volumes can't grow in place. Resizing means a new instance.

  • NFS is still one server. App files on nfs-1 don't have the database's redundancy. Gitea's and MinIO's volumes are backed up to S3 every night instead, each on its own write-only role (Gitea).

Next

  • A 4th node: the 3 instances stay on separate nodes, and a lost node's instance has a spare node to rebuild on.