Crossplane

What runs

• Crossplane in the home cluster, with the AWS S3, KMS and IAM providers. AWS resources are Kubernetes objects, deployed by Argo CD like everything else.

• Projects. A project is a bucket and everything that uses it. Each one asks for a BackupStore. Crossplane builds the bucket's settings, a KMS key of its own, and one IAM role per app that writes to it.

• Drift is put back. Every resource is compared with AWS every minute. A change made in the console is undone, not found months later by a terraform plan.

What the per-project keys cost, and why WAL still ships every few minutes: FinOps: KMS keys for backups.

Why this choice

Long-lived shared infrastructure should be reconciled, not applied. In an enterprise with hundreds of AWS accounts, the VPC, the backup buckets and the roles around them usually live in a Terraform module, applied once per account. Each account then has its own state file and its own pinned module version. Drift is found only when someone runs plan, and changing every account means hundreds of pipeline runs. A controller that holds the desired state fixes both: change the definition once, and every copy converges and stays converged.

Terraform still has a job: the trust anchors. Something has to exist before Crossplane can act: the cluster's identity provider in AWS, the roles Crossplane uses, and the guardrails on them. In an enterprise, that's account vending. Here, it's a small script, reviewed and run by an admin. Crossplane owns everything inside a project.

A platform API, not raw AWS objects. A project asks for a BackupStore with a few fields: its writers, their prefixes, write-only or read-write, and how long to keep objects. Which encryption, which bucket policy and which role guardrails is decided once, by the platform, in the Composition. That's also why it's Crossplane and not AWS Controllers for Kubernetes (ACK). ACK maps AWS APIs one to one, so it has no layer to decide this in.

Three identities, three jobs

provider pod token ─▶ AWS STS ─▶ deployer role for     builds, never reads
                                  ├─ bucket obiwalker-: settings, policy, encryption
                                  ├─ KMS key tagged project=
                                  └─ app roles on /workload//, capped by the boundary
backup pod token ──▶ AWS STS ─▶ app role                         reads/writes, never builds

• Setup script (trust anchors): creates one deployer role per project, the deployer policy, and the permissions boundary.

• Deployer role (Crossplane): one per project, named by that project's ProviderConfig. It can build the vault, but never open it.

• App role (the workload): one per writer. It can use its own prefix of the bucket, and nothing else. It can't create or change anything.

No AWS access key exists anywhere in this chain. Every pod, Crossplane's included, trades its own short-lived Kubernetes token with AWS STS (Identity).

Least privilege, enforced by AWS

One deployer policy serves every project. It scopes itself with the role's own project tag, so adding a project adds no policy to review.

• Bucket settings only, never objects. A deployer can create and configure its own bucket. It can't read, write or delete a single object.

• Key administration only, never use. It can create and manage keys tagged with its own project. It can't encrypt, decrypt or grant either.

• Roles only with the boundary. It can create roles only on its own path, and only with the permissions boundary attached. It can't remove a boundary. Without this, a role that can create roles can create an admin.

• No tagging for another project. Tags decide who may use a key, so a deployer can't create or tag anything with another project's tag, or remove a project tag.

• Nothing destroyed. No deployer can delete a bucket, schedule a key's deletion, or disable a key. Crossplane is also told never to delete them.

The key policy is the second lock, on the resource itself. Every statement names who it's for, so writing an IAM policy (even kms:*) grants nothing on a project key:

The caller's own IAM policy must allow it too. Nobody else can create a grant, which would hand out decryption without these conditions. If a role's trust were ever pointed at an outside account, that account still couldn't decrypt.

Write-only means write-only. Gitea's and MinIO's backup roles can upload but can't list, read or delete backups, so anything that takes over the cluster can't wipe them. They still need kms:Decrypt as well as GenerateDataKey, because large uploads come in parts and S3 needs both to put the parts together. Without s3:GetObject, they still can't read.

What this proves

TODO: results, after the checks have run: drift put back, each deployer guardrail denied, a cross-project key use denied, and the backups and restore drills passing on the new roles and keys.

Moving existing buckets over

The buckets already hold backups, so Crossplane adopts them in three steps:

1. Observe first. Crossplane only watches the existing bucket and its settings, and shows what it would change.

2. New roles. The writers move to app roles that Crossplane creates. An IAM role's path can't change, so these are new roles, not adopted ones.

3. Manage. Crossplane takes over the settings. New objects are encrypted with the project's own key. Old objects expire in 7 days, or leave Postgres's 14-day window, so nothing needs re-encrypting.

Trade-offs and known limits

• The cluster becomes part of the control plane. If Crossplane is down, nothing is corrected, though nothing breaks either: AWS keeps running what exists.

• One bad Composition reaches every project at once. Two projects make that fine here. At enterprise scale, Composition revisions roll a change out to a few projects first. That's the next demo, with VPCs across accounts (Roadmap).

• An untagged KMS key could be claimed. A deployer may tag a key at creation, and AWS can't tell that apart from tagging an existing, untagged key. Every key in the account carries a project tag.

• Crossplane can change who may decrypt. The deployer keeps the right to change its key's policy, so key-policy changes stay in git like everything else. AWS can't limit what a policy says, so the control is git review: Argo CD applies only what's merged. The alternative, making every key-policy change a manual step, was weighed and rejected for now.

• A deployer can rewrite its bucket policy for the same reason. Another account given access would get ciphertext only, because the key refuses other accounts. The org-level fix is an AWS Organizations resource control policy, with the second account.

• A deployer can list its bucket's object names, because Crossplane needs ListBucket to check that the bucket exists. It still can't read them.

• One account for now. A second account needs either the cluster's identity registered there too, or a hub role in between.