Data platform

The platform doesn't run "the database". It gives each app a way to have its own Postgres, run the same way every time. Each app's own choices (a cluster of its own, or a tenant on a shared one) are on that app's page.

What runs

  • CloudNativePG, a Kubernetes operator for Postgres. One operator runs every Postgres cluster.

  • Barman Cloud plugin: continuous WAL archiving plus a nightly base backup to S3, kept for 14 days.

  • cert-manager, for the mTLS between the plugin and the operator.

  • pgAdmin, behind the same Zitadel login as everything else.

Two clusters run today: zitadel-db for Zitadel, and postgres, shared by tenants such as Gitea. Each has three instances, one per node, on that node's local disk (Storage).

Why a Postgres operator

An app that needs a database declares a Cluster object in its chart, and the operator does the rest:

  • High availability: a primary and streaming replicas, with failover.

  • Rolling updates by switchover: the primary moves to a replica instead of restarting (Storage).

  • Databases and roles as objects: each tenant's database and login are declared in git like everything else.

  • Backups as config: each chart holds its own backup store and schedule.

Without the operator, each app would bring its own way of running, backing up and upgrading Postgres.

TODO: anything you'd add on why CloudNativePG over other operators, or over a managed database such as RDS.

Why the Barman Cloud plugin

CloudNativePG used to have backups built in. That's deprecated and goes away in version 1.31, replaced by plugins. The Barman Cloud plugin is the replacement: it runs Barman, Postgres's long-standing backup tool, against S3.

Each cluster's chart declares three things:

  • an object store: where in S3 the backups go, and how long they're kept;

  • the plugin on the cluster, as its WAL archiver;

  • a scheduled backup: the nightly base backup.

WAL streams to S3 continuously, so a restore can go to any point in the last 14 days, not just last night's backup.

How a backup works

The plugin adds a sidecar container to every database pod. Postgres hands each finished WAL file to the sidecar, and the sidecar uploads it. The nightly base backup runs in the same sidecar.

The sidecar has no AWS key. It uses the pod's own identity:

database pod (service account = the cluster's name)
 ├─ postgres ──── each WAL file ────▶ backup sidecar
 └─ backup sidecar
      reads a token the kubelet writes into the pod and renews (1 h, for AWS only)
      trades it with AWS STS ──▶ the cluster's own IAM role ──▶ 1 h credentials
      uploads to s3://…//  (the only place that role can reach)

  • One role per cluster. postgres can't read or delete Zitadel's backups, and the other way around. A single shared key used to reach both.

  • Nothing stored. No key to seal, rotate or leak. Kubernetes issues the token, and AWS trusts the cluster to issue it (Identity).

  • No Zitadel on the path. Zitadel's own database backs up, and can be restored, while Zitadel is down.

The sidecar's settings can't add volumes of their own, so the token comes in through the cluster: the operator mounts it into every container in the pod, the sidecar included.

Backups you've restored

The restore drill builds a new cluster from S3 next to the live one, then checks the data. The live cluster is never touched. It's a manual drill, run at setup and again after any change to backups.

The last drill, on 2026-10-21, ran after the move to roles, with the old key turned off. Both clusters restored with their own roles:

Cluster Check Live Restored
zitadel-db Zitadel projects 10 10
postgres Gitea repositories 15 15

If the cluster itself is gone, the backups are ordinary Barman files. Anyone with admin AWS access can download them to a machine with Postgres and restore with Barman's own tools.

Trade-offs and known limits

  • The newest backup isn't restorable for up to five minutes. A base backup ends in a WAL file that's still being written. A quiet database only ships it after five minutes, so a restore right after a backup waits for it.

  • Restore names are fixed. Each role trusts the cluster's own service account and one for its restore, named -restore.

  • Changing credentials takes two steps. Switching a running cluster to new credentials in one change left the old primary unable to archive its last WAL, and it waited forever. Adding the new token first, then switching, avoids it. The runbook has the order.

Next

TODO: alerts when archiving or a nightly backup fails (Roadmap).