VSecM Architecture

edit this page on GitHub ✏️

Introduction

This section discusses VMware Secrets Manager architecture and building blocks in greater detail: We will cover VMware Secrets Manager’s system design and project structure.

You don’t have to know about these architectural details to use VMware Secrets Manager; however, understanding how VMware Secrets Manager works as a system can prove helpful when you want to extend, augment, or optimize VMware Secrets Manager.

Also, if you want to contribute to the VMware Secrets Manager source code, knowing what happens under the hood will serve you well.

Components of VMware Secrets Manager

VMware Secrets Manager (VSecM), as a system, has the following components.

SPIRE

SPIRE is not strictly a part of VMware Secrets Manager. However, VMware Secrets Manager uses SPIRE to establish an Identity Control Plane.

SPIRE is what makes communication within VMware Secrets Manager components and workloads possible. It dispatches x.509 SVID Certificates to the required parties to make secure mTLS communication possible.

Check out the official SPIFFE/SPIRE documentation for more information about how SPIRE works internally.

It Is More SPIFFE than SPIRE

Technically, any SPIFFE-compatible Identity Control Plane can be used with VMware Secrets Manager. However, only SPIRE comes as part of the default installation.

VSecM Safe

vsecm-safe stores secrets and dispatches them to workloads.

VSecM Sidecar

vsecm-sidecar is a sidecar that facilitates delivering secrets to workloads.

VSecM Sentinel

vsecm-sentinel is a pod you can shell in and do administrative tasks such as registering secrets for workloads.

Here is a simplified overview of how various actors on a VMware Secrets Manager system interact with each other:

High-Level Architecture

Dispatching Identities

SPIRE delivers short-lived X.509 SVIDs to VMware Secrets Manager components and consumer workloads.

VSecM Sidecar periodically talks to VSecM Safe to check if there is a new secret to be updated.

Open the image above in a new tab or window to see it in full size.

Creating Secrets

VSecM Sentinel is the only place where secrets can be created and registered to VSecM Safe.

Component and Workload SVID Schemas

SPIFFE ID format wor workloads is as follows:

spiffe://vsecm.com/workload/$workloadName
/ns/{{ .PodMeta.Namespace }}
/sa/{{ .PodSpec.ServiceAccountName }}
/n/{{ .PodMeta.Name }}

For the non-vsecm-system workloads that Safe injects secrets, $workloadName is determined by the workload’s ClusterSPIFFEID CRD.

For vsecm-system components, we use vsecm-safe and vsecm-sentinel for the $workloadName (along with other attestors such as attesting the service account and namespace):

spiffe://vsecm.com/workload/vsecm-safe
/ns/{{ .PodMeta.Namespace }}
/sa/{{ .PodSpec.ServiceAccountName }}
/n/{{ .PodMeta.Name }}

spiffe://vsecm.com/workload/vsecm-sentinel
/ns/{{ .PodMeta.Namespace }}
/sa/{{ .PodSpec.ServiceAccountName }}
/n/{{ .PodMeta.Name }}

Persisting Secrets

VSecM Safe uses age to securely persist the secrets to disk so that when its Pod is replaced by another pod for any reason (eviction, crash, system restart, etc.). When that happens, VSecM Safe can retrieve secrets from a persistent storage.

Age Algorithm Can We Swapped With SHA-256

The age algorithm can be swapped with SHA-256.

Check out the VSecM Configuration for more information.

Since decryption is relatively expensive, once a secret is retrieved, it is kept in memory and served from memory for better performance. Unfortunately, this also means the amount of secrets you have for all your workloads has to fit in the memory you allocate to VSecM Safe.

VSecM Safe Bootstrapping Flow

To persist secrets, VSecM Safe needs a way to generate and securely store the private and public age keys that are utilized for decrypting and encrypting the secrets, respectively.

• Key generation is conveniently provided by age Go SDK.
• After generation, the keys are stored in a Kubernetes Secret that only VSecM Safe can access.

Here is a sequence diagram of the VSecM Safe bootstrapping flow:

Note that, until bootstrapping is complete, VSecM Safe will not respond to any API requests that you make from VSecM Sentinel.

You Can Swap the Encryption Mechanism

In FIPS-compliant environments, you can use VMware Secrets Manager FIPS-compliant container images that use AES-256-GCM encryption instead of Age.

Check out the Configuration section for more details.

Also, you can opt-out from auto-generating the private and public keys and provide your own keys. However, when you do this, you will have to manually unlock VSecM Safe by providing your keys every time it crashes. If you let VSecM Safe auto-generate the keys, you won’t have to do this; so you can #sleepmore.

Again, check out the Configuration section for more details.

VSecM Safe Pod Layout

Here is what an VSecM Safe Pod looks like at a high level:

• spire-agent-socket: Is a SPIFFE CSI Driver-managed volume that enables SPIRE to distribute X.509 SVIDs to the Pod.
• /data is the volume where secrets are stored in an encrypted format. You are strongly encouraged to use a persistent volume for production setups to retrieve the secrets if the Pod crashes and restarts.
• /key is where the secret vsecm-safe-age-key mounts. For security reasons, ensure that only the pod VSecM Safe can read and write to vsecm-safe-age-key and no one else has access. In this diagram, this is achieved by assigning a secret-readwriter role to VSecM Safe and using that role to update the secret. Any pod that does not have the role will be unable to read or write to this secret.

If the main container does not have a public/private key pair in memory, it will attempt to retrieve it from the /key volume. If that fails, it will generate a brand new key pair and then store it in the vsecm-safe-age-key secret.

Template Transformation and K8S Secret Generation

Here is a sequence diagram of how the and VSecM Safe-managed secret is transformed into a Kubernetes Secret (open the image in a new tab for a larger version):

There are two parts to this:

• Transforming secrets using a Go template transformation
• Updating the relevant Kubernetes Secret

You can check VSecM Sentinel CLI Documentation for various ways this transformation can be done. In addition, you can check VSecM Secret Registration Tutorial for more information about how the Kubernetes Secret object is generated and used in workloads.

Liveness and Readiness Probes

VSecM Safe and VSecM Sentinel use liveness and readiness probes. These probes are tiny web servers that serve at ports 8081 and 8082 by default, respectively.

You can set VSECM_PROBE_LIVENESS_PORT (default :8081) and VSECM_PROBE_READINESS_PORT (default :8082) environment variables to change the ports used for these probes.

When the service is healthy, the liveness probe will return an HTTP 200 success response. When the service is ready to receive traffic, the readiness probe will return an HTTP 200 success response.

Conclusion

This was a deeper overview of VMware Secrets Manager architecture. If you have further questions, feel free to join the VMware Secrets Manager community on Slack and ask them out.