Menu

Subscribe

Helm Value Store - Simplifying multi-cluster Helm Deployments

When deploying applications to Kubernetes, there are currently a lot of options. One of the most popular options is Helm. For those who are unfamiliar, Helm allows users to create “Charts” that define how to deploy an application. These Charts are made up of templates for things like Deployments, ConfigMaps, Secrets and other Kubernetes resources. When a Chart is installed, users of Helm select a file that defines a set of values (the values file) which are used to replace templated sections of the manifests defined in the Chart.

Helm Charts have become a very common way to define how an application should look from the perspective of Kubernetes. They can be made as static or dynamic as you like and there is a lot of great tooling built around Helm as it has become more stable since its 2.0 release. However, there are still some operational issues that are left as an exercise to the user. At Skuid, the biggest of these issues was what to do with all of our values files. While storing them in a Git repository was an option, we didn’t like the idea of having to manage encryption whenever we needed to update these files. Also, diffs become somewhat useless because encryption obscures the meaningful changes.

Instead of kicking the can down the road and dealing with it later, we developed a Helm plugin called Helm Value Store. Written in Go, this plugin allows users to manage their values files in a “helm native” way. It has subcommands for creating, updating and deleting values and also supports installing a Helm chart using values stored in a remote datastore like DynamoDB or GCP Datastore.

This post is meant to show how Skuid uses Helm Value Store and explain some of the advantages we’ve found while working with it over the last year.

How it works

Helm Value Store (or helm-value-store) plugs into Helm to add extra functionality around managing values files. By default, helm-value-store will use DynamoDB as a backing datastore. A helm-value-store release is made up of a few pieces of information.

  1. The Helm Chart you want to install. If you can helm install the Chart, you can use it with helm-value-store.
  2. The version of the chart you want to use.
  3. The name of the Helm release.
  4. The namespace to install the release in.
  5. Any labels you would like to apply. These can be used to apply releases in bulk.
  6. The values to associate with your release.

The release can simply be created by issuing the following command:

$ helm value-store create --name my-release \
	--namespace my-namespace \
	--chart myrepo/mychart \
	--version 0.1.0 \
	--labels environment=prod \
	--labels region=us=west-2 \
	--values my-values-file.yaml

A record will then be created in your datastore and a unique ID will be assigned to it. You can use this ID to install your charts. If we issue a list command, we can see all of the releases we currently manage with helm-value-store.

$ helm value-store list
UniqueId                              Name                  Namespace    Chart                       Version  Labels                                     Values
6fad4903-58ec-446f-bda4-bd39c4ff96aa  alertmanager          default      skuid/alertmanager          0.1.0    map[region:us-west-2 environment:prod]     1.1K
8795d237-adac-4b91-b55b-bb0f1e258a32  exporter              default      skuid/prom-node-exporter    0.1.0    map[region:us-west-2 environment:prod]     279B
22c8f1e8-82fc-4eb0-b1f9-2c8d50b2df3b  prom1                 default      skuid/prometheus            0.1.2    map[region:us-west-2 environment:prod]     1.1K
ad01e6d4-05ec-4f18-ba6a-87cd49e6be25  alertmanager          default      skuid/alertmanager          0.1.0    map[environment:test region:us-west-2]     0
84c28f16-0bc2-4384-9e21-8077e3320aad  exporter              default      skuid/prom-node-exporter    0.1.0    map[environment:test region:us-west-2]     274B
fa718433-d76e-4edd-b263-9c50246c2f80  prom1                 default      skuid/prometheus            0.1.2    map[environment:test region:us-west-2]     0
080f9a8a-10dd-4c2f-8588-8c3c4980553f  alertmanager          default      skuid/alertmanager          0.1.0    map[region:eu-central-1 environment:prod]  1.3K
49582465-85fd-49ce-9778-4bf9d1162a2e  exporter              default      skuid/prom-node-exporter    0.1.0    map[environment:prod region:eu-central-1]  272B
34754bde-3114-43ca-bb23-1d4e16f12f95  prom1                 default      skuid/prometheus            0.1.2    map[environment:prod region:eu-central-1]  0

$ helm value-store list -s environment=test
UniqueId                              Name                  Namespace    Chart                       Version  Labels
ad01e6d4-05ec-4f18-ba6a-87cd49e6be25  alertmanager          default      skuid/alertmanager          0.1.0    map[environment:test region:us-west-2]     0
84c28f16-0bc2-4384-9e21-8077e3320aad  exporter              default      skuid/prom-node-exporter    0.1.0    map[environment:test region:us-west-2]     274B
fa718433-d76e-4edd-b263-9c50246c2f80  prom1                 default      skuid/prometheus            0.1.2    map[environment:test region:us-west-2]     0

As you can see, we have multiple relesaes of the same chart for different environments. The chart and version are the same, but the underlying values are different. Installing a chart is as easy, if not easier, than installing your chart with plain ol’ Helm.

# this will install skuid/alertmanager:0.1.0 with the values for environment=test,region=us-west-2
$ helm value-store install --uuid ad01e6d4-05ec-4f18-ba6a-87cd49e6be25

If you ever need to update your values, you can use get-values to download the values and write them to a file. Once you’ve made your updates, you can then use the update command to replace them.

$ helm value-store get-values --uuid ad01e6d4-05ec-4f18-ba6a-87cd49e6be25 > alertmanager.values.yaml
$ vim alertmanager.values.yaml # make your edits
$ helm value-store update --uuid ad01e6d4-05ec-4f18-ba6a-87cd49e6be25 -f alertmanager.values.yaml
$ helm value-store install --uuid ad01e6d4-05ec-4f18-ba6a-87cd49e6be25 # rollout your new changes

Under the hood, helm-value-store is doing everything helm would do when installing a chart. However, instead of using a local values file, it will download the values from the release store and use those to compile the Chart templates.

Taking it one step further - UI/Server

As we began using helm-value-store internally, we found that the workflow became a bit cumbersome when rolling changes out to multiple cluster. To use Helm, you need to access to the Tiller server running in your cluster. For us, this meant jumping between VPN servers to install changes in different regions. To make matters worse, iterative edits to values became more difficult to manage as each edit requires 3 steps to roll out. It was also difficult to see, at a glance, the state of these values. We needed a better solution. Fortunately, we work at Skuid.

If you aren’t familiar with Skuid, you can see what we’re about here. Skuid is a cloud based UI/UX platform as a service. You can use Skuid to build apps on top of your data, wherever it lives. Using the power of Skuid and its native DynamoDB integration, we built a custom interface for managing our values across all of our environments. This interface gives our team a common place to make changes and roll them out.

To do this, we added an API server to helm-value-store. This server exposes a /apply endpoint which invokes an install. It accepts the unique ID of your release and installs it into your target environment. We then launched an instance of this server in each of our Kubernetes clusters. To secure our new API, we rely on Google OAuth to authenticate connections to the service. When interacting with our new interface, users authenticate with Google, and subsequent requests to our API add an Authorization header obtained by Skuid. Authorization is then performed based on the email domain of the user as well as their membership to a specific Google Group. This prevents unauthenticated and unauthorized access which is actually a step above the capabilities of Tiller, Helm’s server side component.

With the introduction of this interface, we’ve been able to transform the way we manage Helm chart installations on a day-to-day basis. We can update values and roll changes out without having to jump between VPNs or issuing a single command.

Conclusion

Helm has been a great tool for us when it comes to managing resource in Kubernetes. Since the majority of those resources do not need CI/CD pipelines, managing them with our UI has made working with them even better. We plan to open source our UI for anyone who wants to use it in the coming weeks. We look forward to sharing the power of Skuid and helm-value-store with the rest of the Kubernetes community!

If you have any suggestions or would like to see additional functionality in helm-value-store, please let us know! We are always accepting pull requests!