SDD 0015 - Secrets Management

The Steward Cluster Agent will bootstrap the cluster local Vault installation and in coordination with the Lieutenant - Management API set up the auto unseal, recovery keys and root tokens. An important aspect is to make sure Argo CD can only apply secrets once the cluster local Vault is ready.

The cluster local Vault will be set up to auto unseal using the Central Vault transit secret engine. This ensures restarting Vault pods won’t leave the cluster in an unusable state. The recovery keys generated during setup will be GPG encrypted and centrally stored. This ensures that even in the event of an outage of the Central Vault we’re still able to unseal cluster local Vaults.

The cluster local Vault will use the service account token from Steward to authenticate against the Central Vault. An initContainer running the Vault Agent will lease a token which can be used for auto unsealing.

The Raft storage backend will be used by the cluster local Vaults. This simplifies the setup and maintenance of the Vault setups. Even though this feature is still labeled as Beta, we’ll make use of it and will switch to another backend if it proves to be unstable.

There are two parts of Vault which need to be backed up: The storage backend and the unseal keys. Without both of them, either is unusable. The unseal keys will already be stored centrally as discussed. This leaves the storage backend to be backed up. The Raft storage backend provides a snapshot functionality for this very purpose.

The unseal keys will be GPG encrypted and distributed. The exact setup and requirements is to be defined.

The transit secret engine is used to auto unseal cluster local Vault instances. A separate key per cluster is used and the Kubernetes authentication method to authenticate the clusters. Since the steward cluster agent will have a service account token to authenticate to the Lieutenant API it can use the same token to authenticate to the central Vault. The setup of these parts will be done in coordination by Lieutenant and Steward.

The Central Vault will be configured with the Kubernetes auth method, against the central infrastructure cluster. Since the steward cluster agents will have a service account token from the infra cluster, they can use it to authenticate against this vault instance as well.

As a user of Syn I can store secrets in the secret management system and the application development description in Git, pointing to the necessary secrets available in the secrets management system. The system takes care of revealing the secret only when applying configuration to the platform. No secrets are stored in Git, especially not unencrypted.

An example: Storing a Kubernetes object of the kind "Secret" in Git only contains a pointer to the secret and not the secret itself. The configuration management system then takes care of retrieving the requested secret during the apply phase on the cluster itself, so the secret never leaves the cluster.

As a user of Syn I want to securely access secrets from my application running on the platform. Objects of the kind "Secret" aren’t automatically stored encrypted and are available in quasi-plaintext to the platform. The platform should provide means to access secrets in a more secure way.

As a user of Syn I don’t want to care about managing secrets and leave this work up to the platform. I want to have access to automatically generated secrets and use them for the purpose they’re meant for.

Examples:

As a user of Syn I want the possibility to rotate secrets. If credentials get leaked or otherwise compromised it should be easy and fast to rotate the affected secrets.

As a user of Syn I want the ability to see an audit trail for secrets. At the minimum I want to see when and by whom a certain secret was accessed.

As a user of Syn I’m using a CSI provider that supports encryption, I have to provide secretes (encryption key passphrase) the CSI provider can use for the LUKS encryption. The cloud provider must not be able to decrypt volumes (no access to the encryption keys, except VM in-memory).

Storing these secrets on the cluster without encryption is therefore a no-go as it would negate mostly all benefits from disk / volume encryption. Also using the the cloud provider’s KMS isn’t an option in the case where the cloud provider must not have access to the encryption keys.

From Vault’s view, all secrets are accessed by the GitOps tool. Since they’re stored in the Kubernetes secrets, Vault has no control over them anymore. This invalidates the whole audit and access control features of Vault.

This means that these secrets will be stored in plain text in the cluster’s etcd cluster, except if encryption at rest is configured. As this currently is only really supported on the major cloud providers, most clusters don’t set it up. An attacker with access to the data in etcd, be that the live data or from a backup, can access all the secrets.

Even though we currently don’t profit from most of the features Vault brings to the table, this approach enables us to move away from native Kubernetes secrets in the feature and make real use of the Vault features. Additionally it enables other applications on the cluster to use the Vault instance.