Zitadel

The identity provider every other app trusts (Identity). Because everything else depends on it, it's the most careful build in the lab.

Why Zitadel

Zitadel is an open-source identity provider I can run myself. It fit what the lab needed:

A project per app, with its own roles. Access to an app is a role, granted or removed in one place (Identity).
OIDC and SAML in one place. Every app uses OIDC, and AWS IAM Identity Center, which accepts only SAML, uses the same accounts.
Postgres. It stores everything in Postgres, which the lab already runs, backs up and restores.
An API for setup. A script creates each new app's project, roles, OIDC app and grants, so nobody has to click through the console.

Zitadel, with 2 replicas of the API and console plus its login UI, at auth.obiwalker.com.
Its own Postgres cluster (zitadel-db), separate from the shared one, with 3 instances, each on a different node's local disk.
Backups: WAL archiving and a nightly base backup to S3, 14 days kept. Restore drills check the restored data against the live database.

Install order. Zitadel's install hooks set up the database before anything else in the release runs. So the database and the secrets those hooks read come from a separate base chart, installed and healthy first.
Blast radius. If the shared database has a problem, logins still work. Every other app can still authenticate.

First in, measured. zitadel-db was the first cluster moved from NFS to local disk, and its cutover timings set the switchover policy for every cluster (Storage).
The master key never rotates. Zitadel encrypts stored secrets, such as the SMTP password, with it. It's sealed in git and backed up.
Self-registration is off. Users are invited, and they can't reach any app until they're granted a role in it.
Setup that isn't in git is written down. The login policy, domain and SMTP live in Zitadel's database, not in chart values, so the runbook lists them to redo after a rebuild.