Secrets Management for Your Kubernetes Workloads

In the DevOps space, Kubernetes has become almost synonymous with simplification. The ability to simplify the management of containers with multiple features, such as scaling on demand, autoscaling, and configuration management, has helped make Kubernetes the most widely adopted solution for orchestrating containerized workloads.

However, there is one space that is not clearly dealt with in Kubernetes: secrets management.

In this article, we will explore at how traditional Kubernetes deployments are executed, then identify gaps and best practices in deploying and managing secrets for your Kubernetes workload.

Deploying Kubernetes workloads with Helm Charts, Kustomize, and operators

Kubernetes workloads primarily comprise deployments, statefulsets, and CronJobs. Out of these, 90% of people only run stateless workloads on Kubernetes, which are managed using a deployment object in Kubernetes, and a few may use CronJobs.

Implementing Helm Charts, Kustomize, and operators are a few common ways to deploy these workloads. We discuss each of these in more depth below.

Helm Charts

Helm charts are the most common way to deploy workloads on Kubernetes. Helm is a templating engine that can help omit basic resources so that only a significant configuration is exposed. The configuration can be as easy as an image to deploy or a number of replicas. Users simply provide a small YAML file with customized deployment values without all the complexity of the actual Kubernetes objects.

Kustomize

Kustomize provides a template-free approach to customize application configurations. Backed by a Kubernetes special interest group, it can be applied using the kubectl command and can itself apply environment-specific changes from one centralized code repository. While it lacks Helm’s templatization, Helm does not provide the ability to deploy different application configurations from one place.

Operators

Operators are native Kubernetes objects you can use to manage deployments. Unlike Helm and Kustomize, operators can both keep an eye on resources and take action to heal the system. They can run as a pod in the cluster and manage resources from inside the cluster.

Deploying a workload

The first step in deploying a workload on Kubernetes is quantifying the deployment. A deployment in Kubernetes consists of the following objects:

  • An image: In Kubernetes, images are the code piece; they can also contain information like configs, number of replicas, CPU, and memory.
  • Config maps: Let you map the configurations or settings of the application.
  • Credentials or important tokens: These are objects that have limited access and can be part of Kubernetes secrets.

This is not a complete list of objects; there can be other configurations that are related to the pipeline and used for the deployment automation process.

Distinction between encrypting, encoding complicate Kubernetes secrets

While Kubernetes provides a simple way of deploying code and configs, privacy and accessibility concerns are what complicate managing secrets in Kubernetes. By nature, secrets should be hidden from everyone except a select few. In Kubernetes, secrets are not encrypted, they are encoded. With traditional secrets encryption, only the person who has the key can see the data. With encoding, anyone can see the data.

However, there are software and open source solutions to both safely and simply manage secrets in Kubernetes.

Software as a Service (SaaS)

When building a secrets management platform, there are numerous security considerations that can be missed during implementation. A SaaS can protect against mistakes, and allow you to benefit from a fully compliant secrets management platform without having to build and manage it.

SaaS platforms like HashiCorp’s Vault, AWS Secrets Manager, and Azure Key Vault allow users to store and access control on tokens, secrets, keys, and certificates. SaaS platforms can provide different interfaces to view and manage these secrets using a CLI, HTTP APIs, and a UI. Multiple secret engines integrate with SaaS platforms, including MySQL, MongoDB, and the Lightweight Directory Access Protocol (LDAP).

To ensure privacy, SaaS platforms will typically support data encryption and provide an encryption-as-a-service (EaaS) platform. These platforms also allow for easy control over basic operations such as key rotation, access control, and auditing of all events.

Pros, cons, and considerations

Advantages: SaaS platforms are managed, so users don’t have to worry about their backup, failovers, and availability. They are also typically easy to integrate.

Disadvantages: Associated costs and dependency on the vendor for a fix whenever any vulnerability is discovered. Although most platforms are straightforward, some may also have tricky integration.

Another important aspect to be aware of is availability; you want your secrets management platform to be up all the time. So before deciding on one, make sure to check out the SLA.

Open-Source Management Strategies

There are a few open-source alternatives that can help address the challenges with Kubernetes for secrets management.

Secrets with CI/CD and GitOps

You can also implement Git and continuous integration/continuous delivery (CI/CD) for Kubernetes secrets. For this, you first need to remove access permissions for Kubernetes secrets from everyone. After that, you can make use of GitLab/GitHub environment variable features to inject the secrets into your CI/CD pipeline. You would then run Helm or direct kubectl commands to inject the secrets into Kubernetes clusters.

Another layer of protection that you can add is to encrypt the keys using openPGP encryption and put them in the environment variables of GitLab/GitHub. Then, you can use the Helm secrets plugin to deploy them. 

This mechanism does not have any automated key rotation. Instead, you will have to write the scripts to enable this. Also, you can only use it for Kubernetes; it does not support databases, LDAP, or any other kind of secrets engine.

When using CI/CD, the Kubernetes secrets are in your deployment pipeline without any extra cost. If you are able to manage the permissions at the Kubernetes level and GitLab level properly, it will be safe. There is also little effort involved.

However, there is still manual work that goes into encrypting the secrets and putting them in the pipeline. The person who does this task will be able to access the secrets they encrypt. Further, if there is an access control issue, anyone who has access to Kubernetes secrets objects will be able to see the secrets.

Sealed Secrets by Bitnami

Bitnami has an open-sourced project named Sealed Secrets. It lets you keep all of your secrets in your Git repository, which helps you easily rotate secrets, renew keys, etc. It has two components. First, there is the controller, which runs on the Kubernetes cluster as an operator, and second, you have the client-side utility called kubeseal.

With kubeseal, you can encrypt secrets and keep them in Git; only the controller is able to decrypt the secrets. Keep in mind that anyone who has access to Kubernetes secrets can view these secrets. Sealed Secrets also let you save private keys and certificates.

There will have to be a person responsible for creating the encrypted secrets with kubeseal and then committing them in Git.

Using Kustomize and KSOPS plugin

This approach is very similar to CI/CD and Sealed Secrets. Here, you rely on the KSOPS Kustomize plugin and then implement GPG keys to encrypt the secrets and deploy them to the Kubernetes cluster. In the deployment pipeline, the KSOPS plugin will decrypt the secrets and create the Kubernetes secrets object, which can then be accessed by the applications and workload.

K8s community and secrets management

Built by a K8s special interest group keeping secrets management in mind, the Kubernetes Secrets Store CSI Driver provides a way to integrate your secrets store and mount the secrets as a volume in containers. This is done using the container storage interface, and right now only four providers have an implementation for this: the three major clouds (AWS, Azure, GCP) and Vault. It is also used by AWS Secrets Manager and Azure Key Vault for managing the lifecycle of Kubernetes Secrets.

Should you handle K8s secrets management yourself or rely on a SaaS platform?

The many options for managing secrets in Kubernetes can be overwhelming. If we look at the broader picture, the decision-making process can be facilitated by taking into account the following factors:

  1. Cost and features
  2. Open-source or managed
  3. Team size required
  4. Ecosystem

Cost can be a major factor if you decide to go with a SaaS platform, but the money spent can be very easily justified due to the amount of effort SaaS saves as well as the features it provides. If you have a team to deal with a secrets management tool, you can explore open-source solutions.

Whichever direction you choose, building your secrets management system is a very important design decision. Once built, it can serve you for a very long time. None of the above options is right or wrong. At the end of the day, your choice will depend on the requirements, resources, and capabilities of the team that is going to build and manage the secret management system.

Resources

Page: Cloud Monitoring | Cloud Native Application Observability | Cloud Infrastructure | AppDynamics

Datasheet: Cloud Native Application Observability

Document: Kubernetes and App Service Monitoring

Document: Cloud Native Application Observability Modules

Video: Cloud Native Application Observability: Kubernetes Overview

dCloud Sandbox: Getting started with AWS Observability in Cloud Native Application Observability