Admission controllers, one of your best friends for Kubernetes security

Kubemas was the Christmas edition of the Kubernetes London meetup. On that event I did a presentation about improving the security in Kubernetes clusters with admission controllers. I was humbled and at the same time excited for being part of a panel of super clever and popular security experts like Justin Cormack, Liz Rice, and Andrew Martin. You can find the full recorded event here

Here in this series of articles I want to share some of the options we have when leveraging admission controllers and additional tips for improving the security in Kubernetes clusters.

Kubernetes admission controllers overview

“Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether.” from the official source

Kubernetes API request lifecycle

There are multiple admission controllers enabled by default in Kubernetes and I will mention a few more details about the most interesting ones for me which are MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

“The admission control process has two phases: the mutating phase is executed first, followed by the validating phase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both.”

Order is key - welcome the mutating webhook reinvocation

One important aspect to be aware is that previous versions of the MutatingAdmission Webhook could be triggered without a predictable order and that was a problem for non-idempotent webhooks, then the order became the alphabetic one using the name of the webhook, from Kubernetes version 1.15 you can leverage the reinvocation feature to be able to re-run the webhook if other changes happen even after the first run of the webhook.

As an example, if you don’t create a mutating webhook with idempotence in mind you can have problems like one webhook creating resource limits but another webhook injecting a container in the deployment, as you might have realised if the resource limits are updated before the injection of additional containers then it won’t do its job consistently,

The last chance to catch non-compliant requests

The last line of defense before the API request execution finishes are the ValidatingAdmissionWebhooks. They just accept or reject rather than modify the objects like the MutatingAdmissionWebhooks.

Check the image below, if you define a policy in your company that explicitly says no containers running as root, you can leverage a ValidatingAdmissionWebhook which reviews the object and if something is not compliant will reject that object to be persisted in the cluster. One of the default admission controllers that will help you with this example is the PodSecurityPolicy one.

How default admission controllers can fortify your Kubernetes cluster

There are multiple ways the mutating and validating admission webhooks can help you improve the security in Kubernetes, I am going to mention a few native ones as I will write the other articles with more examples and deeper details. You can use these ones out of the box.

AlwaysPullImages - This admission controller modifies the image pull policy for every Pod. We have seen multiple attacks related to cached container images in the same node used by pods that should not have access to those. As an example this admission controller is useful in a multitenant cluster so that users can be assured that their private images can only be used by those who have the credentials to pull them.

EventRateLimit - This one helps with the A of the information security CIA triad concept (Availability). If you want to avoid disruption in your cluster due to the API server being flooded by event requests you can control that defining the rate limit of the requests. This can help you avoid a Denial of Service from malicious Pods but in addition, as I have seen in my experience with locked-down private clusters, it can mitigate issues with misbehaving workloads that have misconfigured super frequent requests to the API server.

Image source: https://sysdig.com/blog/monitor-kubernetes-api-server/

ResourceQuotas - Similar concept as the previous one in terms of taking care of the availability, this admission controller can help you with the expected distribution of resources. If you have a deployment without resource limits that can be problematic, I have seen workloads with memory leaks that quickly block the scheduling of new Pods due to no more memory available in the nodes/cluster. This admission controller ensures that all the Pods have resource limits defined within the desired threshold.

PodPreset - This admission controller could help you if you want to automatically inject data(like secrets, environment variables, and volume mounts) on every Pod using the appropriate labels. As an example the production database credentials can be injected rather than creating a secret config for each Pod. I see it useful because it centralises the credentials control and is convenient for the developers to just annotate with the desired label to get those secrets automagically injected and mounted.

PodSecurityPolicy - This is one of the most popular admission controllers for Kubernetes security, a lot of people using it never realise it is an admission controller but at the end of the day those details do not matter if it does the job well. The main job is to admit or deny any Pod that does not have the security-sensitive aspects of the pod aligned with the Cluster Pod Security Policy.

The table below lists the security properties that can be controlled and reviewed by the PodSecurityPolicy admission controller.

Uplifting Kubernetes security with custom admission controllers

Kubernetes admission controllers, in my opinion are one of the many native features we can leverage to have more confidence in what is being deployed in our clusters.

In the next articles I am going to talk about multiple practical examples including the ones below:

• Injecting a security context in all your pods
• Replace all container image tags for more precise container SHA256 digest
• Allow pulling images only from specific registries known to the enterprise, while denying unknown image registries
• Supply chain security - Allow to run only containers that have the proper attestations for the environment (Grafeas+Kritis / In-toto)

Open Policy Agent integration with Kubernetes admission controllers

I am very passionate about the new trend Policy-as-code, one of the main disruptors in this area has been Open Policy Agent helping to achieve that practice.

In the next articles I am going to talk about multiple practical examples of integrations with OPA including the ones below:

• From PodSecurityPolicies to OPA Gatekeeper
• Using custom Authorisation in Kubernetes with an OPA admission webhook
• Using OPA for Kubernetes Ingress validation
• Existing projects leveraging OPA in Kubernetes

Conclusion

As mentioned in the previous sections, admission controllers are very powerful and flexible. If you spend enough time you can have fine grained controls of what is being created or modified in your Kubernetes cluster, those controls help to give you the assurance that your Kubernetes cluster complies with the security policies you can enforce with admission controllers.

Follow me on Twitter (@diegocomas) and tell me which topic should be first for Episode II: Custom admission controllers or Open Policy Agent integration with Kubernetes admission controllers?

Photo by Henning Kesselhut