Certain apps may require their secrets to be mounted in “in-memory” volumes. Depending on the app, this could be the only way the app consumes secrets and configuration information.
When this is the only way to configure the app, especially when we don’t have access to the application’s configuration and source code, there won’t be any other feasible way to pass secrets to the app.
Here is a screencast that demonstrates this use case:
WORK IN PROGRESS
Use VSecM Sidecar and VSecM Init Container to provide the secrets the workload needs when needed.
Open the image in a new tab to see the full-size version:
We will assume our workload has the name example
, deployed to the example-apps
namespace, and is associated with the example-sa
service account,
having a label example-app
.
Here is a sample deployment manifest of such a workload:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
ClusterSPIFFEID
For VSecM to communicate with this workload, a ClusterSPIFFEID
is needed.
Here is how such a ClusterSPIFFEID may look like:
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
name: example
spec:
spiffeIDTemplate: "spiffe://vsecm.com\
/workload/example\
/ns/{{ .PodMeta.Namespace }}\
/sa/{{ .PodSpec.ServiceAccountName }}\
/n/{{ .PodMeta.Name }}"
podSelector:
matchLabels:
app: example-app
workloadSelectorTemplates:
- "k8s:ns:example-apps"
- "k8s:sa:example-sa"
When this ClusterSPIFFEID
is defined, our example pod will get a secure
x.509 certificate from the SPIFFE Workload API to talk to VSecM;
however, to fetch the certificate, we will need to modify the pod’s deployment
manifest slightly.
Let’s see that in the next section.
Here is the modified deployment manifest to consume SPIFFE Workload API:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
## <--- BEGIN CHANGE
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
## <-- END CHANGE
We added a special volume, and the [SPIFFE CSI Driver][spiffe-csi-drive] will handle the rest of the communication.
Here is the manifest without the change markers:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
Let’s say this workload needs an /opt/app/credentials
file as an initial
configuration file to execute its business logic. Based on this assumption,
let’s update the manifest accordingly to provide this file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
## <--- BEGIN CHANGE
volumeMounts:
- name: credentials-volume
mountPath: /opt/app/credentials
subPath: credentials
readOnly: true
## <--- END CHANGE
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
## <--- BEGIN CHANGE
- name: credentials-volume
emptyDir:
medium: Memory
## <--- END CHANGE
However, the volume is empty; our app will likely require it to be populated before it can be used. We will address this real soon.
Here’s the manifest without the change markers:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
volumeMounts:
- name: credentials-volume
mountPath: /opt/app/credentials
subPath: credentials
readOnly: true
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
- name: credentials-volume
emptyDir:
medium: Memory
We will populate the volume using VSecM Sidecar.
Let’s add VSecM Sidecar to our manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
volumeMounts:
- name: credentials-volume
mountPath: /opt/app/credentials
readOnly: true
## <--- BEGIN CHANGE
- name: sidecar
image: vsecm/vsecm-ist-sidecar:0.24.1
volumeMounts:
- mountPath: /opt/app/credentials
name: credentials-volume
- name: spire-agent-socket
mountPath: /spire-agent-socket
readOnly: true
env:
- name: VSECM_SIDECAR_SECRET_PATH
value: "/opt/app/credentials/secrets.json"
- name: SPIFFE_ENDPOINT_SOCKET
value: "unix:///spire-agent-socket/spire-agent.sock"
## <--- END CHANGE
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
- name: credentials-volume
emptyDir:
medium: Memory
In this setup, VSecM Sidecar will periodically poll VSecM Safe to fetch
the secret associated with the workload and update /opt/app/credentials
.
Help Needed
VSecM Sidecar currently creates a single file. If the application needs more than one file in the volume, you’ll need to create a specialized sidecar based on VSecM Sidecar.
There are upstream issues that will enable VSecM Sidecar to parse the incoming secret and create separate files in its associated volume.
If you need this functionality, you are welcome to contribute upstream.
Here’s the YAML manifest without change markers:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
containers:
- name: example-container
image: example-app:0.1.0
volumeMounts:
- name: credentials-volume
mountPath: /opt/app/credentials
readOnly: true
- name: sidecar
image: vsecm/vsecm-ist-sidecar:0.24.1
volumeMounts:
- mountPath: /opt/app/credentials
name: credentials-volume
- name: spire-agent-socket
mountPath: /spire-agent-socket
readOnly: true
env:
- name: VSECM_SIDECAR_SECRET_PATH
value: "/opt/app/credentials/secrets.json"
- name: SPIFFE_ENDPOINT_SOCKET
value: "unix:///spire-agent-socket/spire-agent.sock"
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
- name: credentials-volume
emptyDir:
medium: Memory
But now, we have another problem: The main application likely assumes the credentials are already there when it starts its lifecycle. The app will likely crash if the credentials are not there during its bootstrapping.
To fix this, we’ll need an init container that watches this volume and initializes the main app only after it is populated. We will implement this in the following section.
Let’s add an init container to complete our plan:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: example-apps
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
serviceAccountName: example-sa
## <--- BEGIN CHANGE
initContainers:
- name: vsecm-init-container
image: vsecm/vsecm-ist-init-container:latest
volumeMounts:
- name: credentials-volume
mountPath: /opt/vsecm/secrets
readOnly: true
## <--- END CHANGE
containers:
- name: example-container
image: example-app:0.1.0
volumeMounts:
- name: credentials-volume
mountPath: /opt/app/credentials
readOnly: true
- name: sidecar
image: vsecm/vsecm-ist-sidecar:0.24.1
volumeMounts:
- mountPath: /opt/app/credentials
name: credentials-volume
- name: spire-agent-socket
mountPath: /spire-agent-socket
readOnly: true
env:
- name: VSECM_SIDECAR_SECRET_PATH
value: "/opt/app/credentials/secrets.json"
- name: SPIFFE_ENDPOINT_SOCKET
value: "unix:///spire-agent-socket/spire-agent.sock"
volumes:
- name: spire-agent-socket
csi:
driver: "csi.spiffe.io"
readOnly: true
- name: credentials-volume
emptyDir:
medium: Memory
VSecM Init Container will now watch the volume and only initialize the main app when everything is ready.
Help Needed
Enhancing the VSecM Init Container to monitor volume changes is an outstanding upstream issue. Your contributions are welcome.
Until this issue is resolved, you might need to add a second init container to introduce some delay (~10-15 seconds, on average) for VSecM Sidecar to consume the secret and create the file needed.
VSecM Helm Charts can be configured to provide secure pattern-based random secrets
for workloads. For example, we can modify the initCommand stanza
of the sentinel Helm chart’s values.yaml
to register a random username and
password as secrets for our example workload.
# ./charts/sentinel/values.yaml
initCommand:
enabled: true
command: |
--
w:example
n:example-apps
s:gen:{"username":"admin-[a-z0-9]{6}","password":"[a-zA-Z0-9]{12}"}
t:{"ADMIN_USER":"{{.username}}","ADMIN_PASS":"{{.password}}"}
--
Integrating VSecM Sidecar and VSecM Init Container for secret management represents a forward-thinking solution to the challenge of securely providing secrets to Kubernetes workloads.
This approach embodies the principles of zero trust security by ensuring that secrets are dynamically managed and securely injected, thereby minimizing the risk of exposure. As Kubernetes continues to evolve, such patterns will be pivotal in addressing the complex security needs of cloud-native applications, making contributions and enhancements in this space invaluable.
Here are some key points covered in this use case:
ClusterSPIFFEID
s
and the SPIFFE Workload API, the system ensures secure
communication between the workload and VSecM. This method enhances security
by providing each pod with a unique identity and secure certificates for
authentication.