Secrets Store CSI driver in AKS

Author – Anurag Chauhan, Cloud Engineer

What is Secrets Store CSI driver?

The Secrets Store CSI Driver provides Kubernetes a service to mount multiple secrets, certs, and keys which are stored in enterprise-grade external secrets store in their pods as a volume

Prerequisites

• Azure subscription
• Azure CLI 2.31.0 or later.

How it works

Similar to Kubernetes secrets, on pod start and restart the Secrets Store CSI driver communicates with the provider using gRPC to retrieve the secret content from the an external Secrets Store specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as tmpfs and the secret contents in are written to the volume. On pod deletion, the corresponding volume is cleaned up and deleted.

Secrets Store CSI Driver

The Secrets Store CSI Driver is a daemonset that helps in facilitating the communication with every instance of Kubelet. Each driver pod has following containers: node-driver-registrar: It registers the CSI driver with Kubelet so that it knows on which unix domain socket to inform the CSI calls on. This sidecar container is provided by the Kubernetes CSI team. secrets-store: It implements the CSI Node service gRPC services which are described in the CSI specification. And it also performs mounting/unmounting the volume during creation and deletion of a pod.This component is maintained and developed in this repository. liveness-probe: It is responsible for monitoring the health of CSI driver and reports directly to Kubernetes. This enables Kubernetes to automatically detect the issues within driver and restart the pod to try and fix the issue. This sidecar container is provided by the Kubernetes CSI team.

Provider for the Secrets Store CSI Driver

• Azure Provider
• AWS Provider
• GCP Provider
• Vault Provider

Features

• Mounts the secrets, keys, and certificates to a pod by using a CSI volume
• Supports CSI inline volumes
• Mounts multiple secrets store objects as a single volume
• pod portability with the SecretProviderClass CRD
• Supports Windows containers
• Syncs with Kubernetes secrets
• It supports auto rotation of mounted contents and and syn it to Kubernetes secrets

Create an AKS cluster along with Azure Key Vault Provider for the Secrets Store CSI Driver support

First, create an Azure resource groupfollowed by an AKS cluster with Azure Key Vault provider for Secrets Store CSI Driver functionality, use the az aks create command with the azure-keyvault-secrets-provider add-on.

• az group create -n myResourceGroup -l eastus2
• az aks create -n myAKSCluster -g myResourceGroup –enable-addons azure-keyvault-secrets-provider –enable-managed-identity

Ensure the Azure Key Vault Provider for the installation of Secrets Store CSI Driver

• kubectl get pods -n kube-system -l ‘app in (secrets-store-csi-driver, secrets-store-provider-azure)’

Create or use an existing Azure key vault

In AKS cluster, we need to create an Azure key vault which will store the secret. But here we have to keep in mind that the key vault’s name should be globally unique and this Azure key vault can store keys, certifications, and secrets.

• az keyvault create -n -g myResourceGroup -l eastus2
• az keyvault secret set –vault-name -n ExampleSecret –value MyAKSExampleSecret

Provide with an identity to access the Azure key vault

The Secrets Store CSI Driver have two methods to access Azure key vault:

• An Azure Active Directory pod identity
• A user-assigned or system-assigned managed identity

Use pod identities

Azure Active Directory (Azure AD) pod-managed identities use AKS primitives to associate managed identities for the Azure identities and resources in Azure AD along with pods. Further these identities can be used to grant access to the Azure Key Vault Secrets Provider for Secrets Store CSI driver.

1. A cluster identity is to be created, then permissions are to be assigned followed by a pod identity.

Create an identity

You have to have the applicable permissions (owner) in your subscription to create the identity.

• az group create –name myIdentityResourceGroup –location eastus
• export IDENTITY_RESOURCE_GROUP=”myIdentityResourceGroup”
• export IDENTITY_NAME=”application-identity”
• az identity create –resource-group ${IDENTITY_RESOURCE_GROUP} –name ${IDENTITY_NAME}
• export IDENTITY_CLIENT_ID=”$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} –query clientId -otsv)”
• export IDENTITY_RESOURCE_ID=”$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} –query id -otsv)”

Assign permissions for the managed identity

• NODE_GROUP=$(az aks show -g myResourceGroup -n myAKSCluster –query nodeResourceGroup -o tsv)
• NODES_RESOURCE_ID=$(az group show -n $NODE_GROUP -o tsv –query “id”)
• az role assignment create –role “Virtual Machine Contributor” –assignee “$IDENTITY_CLIENT_ID” –scope $NODES_RESOURCE_ID

Create a pod identity

• export POD_IDENTITY_NAME=”my-pod-identity”
• export POD_IDENTITY_NAMESPACE=”my-app”
• az aks pod-identity add –resource-group myResourceGroup –cluster-name myAKSCluster –namespace ${POD_IDENTITY_NAMESPACE} –name ${POD_IDENTITY_NAME} –identity-resource-id ${IDENTITY_RESOURCE_ID}

Assign permissions to the new identity to read your key vault and consider its contents through the subsequent commands: # set a policy to access keys in your key vault

• az keyvault set-policy -n –key-permissions get –spn

# set a policy to access secrets in your key vault

• az keyvault set-policy -n –secret-permissions get –spn

# set a policy to access certs in your key vault

• az keyvault set-policy -n –certificate-permissions get –spn

Create a SecretProviderClass with the usage of the following YAML, the use of your own values for aadpodidbinding, tenantId, and the objects to retrieve out of your key vault:

• kubectl apply -f secretproviderclass.yaml

Create a pod via the use of the following YAML

• kubectl apply -f pod.yaml

Validate the secrets

## show secrets held in secrets-store

• kubectl exec busybox-secrets-store-inline — ls /mnt/secrets-store/

## print a test secret ‘ExampleSecret’ held in secrets-store

• kubectl exec busybox-secrets-store-inline — cat /mnt/secrets-store/ExampleSecret

Disable the Azure Key Vault Provider for Secrets Store CSI Driver on an existing AKS cluster

• az aks disable-addons –addons azure-keyvault-secrets-provider -g myResourceGroup -n myAKSCluster

Existing workloads will have no issues, if the add-on is disabled and will not see any updates in the mounted secrets. The scale up event happens if the pod restarts or a new pod is created, the pod will fail to start because the driver is no longer running state.