Kubernetes Workload Identity

Applications talk to each other, more often than anything else, to get the work done. Applications/Microservices typically have a fixed pattern of communication, for e.g, a backend server will most likely talk to a database server, a log server and take ingress traffic from the frontend server.

These fixed patterns of communications usually translate to access control rules and setting up these access control/authorization rules requires one to “identify” these applications aka workloads.

The Principle of Least Privilege (PoLP) states that every workload must be able to access only the information and resources that are necessary for its legitimate purpose. Thus the workloads need to be identified at a granular level using a unique and entire set of attributes so that the authorization frameworks have the flexibility to put in the access control rules as desired.

Zero-trust requires that the services within the network use strong authentication and authorization policies and ensure that the principle of least privilege is effectively followed even for the east-west traffic and even for the services within a particular application provider space. A strong Identity solution forms the basis to achieve this.

How is Identity handled today?

K8s Service Accounts

K8s service accounts are the popular choice in k8s-orchestrated pods for service-based identity. As an application developer, you could generate individual IAM service accounts for each application and then download and store the keys as a Kubernetes secret that you manually rotate. Every service account has its corresponding certificate which would be used for service authentication.

Service account keys expire every 10 years by default. Thus if you want short-lived secrets then you need to have a system to manually rotate the secrets. The possibility of long-term key exposure and the management overhead of key inventory and rotation usually is a major hindrance in the security promise of this system.

The onus of handling security falls on the shoulders of application developers who might be less equipped to handle the security primitives with their corresponding best practices. K8s ServiceAccountToken (or Projected Service Account Token) allows you to generate a token from a Service Account and specify additional constraints such as expected audience and token lifetime.

In general, ServiceAccount based solutions suffers from the following problems:

1. Service accounts may not take into consideration all the attributes of the workload. For e.g, if an application developer has clusters across multiple regions then the workloads could be spawned across these regions. A service account based identity cannot provide a mechanism for finer attributes such as the location of the pod to handle the authorization.
2. Service accounts or Service account tokens are valid only within a given cluster.
3. No server authentication! The client is authenticated by the server using the service account credentials but there is no way for the client to authenticate the server.
4. Dependency on the k8s control plane. The validation of tokens needs to be done with the K8s API server which means the data plane has an impact if there is any control plane downtime.

K8s Certificate APIs

K8s Certificate APIs allow a pod with appropriate RBAC permissions to send a CSR based on a locally generated key pair. The following call flow shows how a pod can acquire a signed certificate using k8s certificate APIs.

K8s Certificate API

The “Approver” could be a user or automated tooling that can help validate the CSR and then sign the certificate. This is no simple job since the failure to validate or attest the originator of the CSR could easily lead to a big hole in the overall security. The signed certificate eventually can be used for the mTLS handshake and at the end of the handshake the pods could use the certificate information to authorize the flow.
The application developer will have to handle the following aspects of the design:

1. Defining an “Approver”. On what basis will the approver attest or authorize the pod’s CSRs? Here the service provider has to put in a strong attestation framework that would authorize and allow certificate signing. The security promise of the identity will depend on this attestation framework.
2. What part of the certificate would the pods use to identify each other? Pods can belong to certain nodes, cloud-service providers, location, namespace, etc and this needs to be carried as part of the certificates. The application developer has to define this aspect and make sure that all the entities adhere to this certificate format.
3. Handling key rotations. The application developer has to put in place a system to rotate the keys on a periodic basis.

GKE Workload Identity: Cloud Provider managed Identity

GKE workload Identity was developed with one primary use case in mind.

“How to simplify multi-cluster GKE k8s workloads access the Google cloud services using Google Service Accounts (GSA)?”

1. Download the JSON blob for GSA and add it as k8s secret. This k8s secret had to be added to every GKE cluster that has a workload accessing the Google cloud service.
2. Tell the k8s workload to use the k8s secret to allow access to the Google cloud service.

The primary problem here is the same as the developer faces with any other credential burnt into the workload:

1. How do I rotate the credential in case of a compromise? GSA tokens are long-lived (default 10 years).
2. There could be thousands of user-managed google services, thus there could be thousands of GSA tokens that have to be managed by the developer.

GKE workload identity introduces a notion of Identity namespace. An identity namespace is enabled when the workload identity is enabled for the cluster. While the k8s-namespace is local to a given k8s cluster, the Identity namespace is valid across multiple clusters within a given project. The Identity namespace binds the k8s-namespace and k8s-service account to a given Identity.

As an example, consider the following deployment:

GKE Workload Identity: an example

Note that the identity namespace is the same across the cluster since the k8s-namespace and k8s-service account names are the same. The backend pods here are part of two different clusters belonging to the same project (project-acme) and the admin has used the same namespace and service account.
Note that there is no binding to the GSA (GCP Service Account) done as of now. The binding is done through a ServiceAccount annotation.

apiVersion: v1
kind: Service
metadata:
name: backend-app
namespace: backendns
labels:
app: backend-app
spec:
ports:
:::::::---
apiVersion: v1
kind: ServiceAccount
metadata:
name: roleX
namespace: backendns
annotation:
iam.gke.io/gcp-service-account: roleX@project-acme.iam.gserviceaccount.com

Once this is done, one needs to bind the GSA with the identity namespace using the ‘add-iam-policy-binding’ primitive provided as part of gcloud SDK.

gcloud iam service-accounts add-iam-policy-binding
— role roles/iam.workloadIdentityUser
— member “serviceAccount:project-acme.svc.id.goog[backendns/roleX]”
roleX@project-acme.iam.gserviceaccount.com

In short, GKE workload identity allows one to bind the k8s-namespace/service-account name to an Identity-namespace. This identity namespace could then be bound with the GSA thus enabling the k8s-workloads to use that GSA indirectly through their namespace+service-account name.

Using Shared Secrets

This solution comes up very often in the Identity discussion. What if we have a secrets manager such as Vault or k8s secrets manager that can keep the credentials and provide access to those credentials to the right applications. The Vault can be shared across multiple clusters and thus does not have to operate within the auspices of a given k8s cluster. For e.g., a MySQL client can retrieve the access credentials to the MySQL server from the Vault to access the MySQL server.

Fundamentally this does not solve the problem of Identity since it is still not established as to which application should be provided access to the credentials. How to put authorization rules for credential access? In the MySQL example above, how would the Vault know which application is the MySQL client? Thus a shared-secrets manager does not solve the problem of Identity. Keeping shared secrets securely in safe storage and limiting access to it is a different problem and one which also requires a strong notion of application identity.

To sum up

1. Getting Certificate Management right is hard, getting Identity handling right is even harder.
2. Given the thrust towards Zero-Trust Architecture, Identity will play a central role in any organization that intends to be securely online.
3. Application Identity is intricately tied to several other problems (such as Secrets Management, Authorization, etc) and thus there is no excuse for not doing it right.
4. There are several ways to get Identity wrong. Leave it to the experts. Developers should not be dealing with Security elements such as Authn, Authz, Accounting, Secrets management. These should be architectural elements transparently available for any application.

Now you can protect your workloads in minutes using AccuKnox, it is available to protect your Kubernetes and other cloud workloads using Kernel Native Primitives such as AppArmor, SELinux, and eBPF.

Let us know if you are seeking additional guidance in planning your cloud security program.