{{announcement.body}}
{{announcement.title}}

The Art of the Helm Chart: Patterns from the Official Kubernetes Charts

DZone 's Guide to

The Art of the Helm Chart: Patterns from the Official Kubernetes Charts

The installation and management of Helm Charts can introduce a level of complexity that you may not have previously anticipated.

· Cloud Zone ·
Free Resource

Helm Charts package up applications for installation on Kubernetes clusters. Installing a Helm Chart is a bit like running an install wizard, so Helm Chart developers face some of the same challenges faced by developers producing installers:

  • What assumptions can be made about the environment that the install is running into?

  • Is the application meant to be able to interact with other applications?

  • What configurations need to be made available to the user and how should they be offered?

But these questions come with Helm-specific twists. To see why, let’s start with a picture of what happens when a user runs a helm install. Then we can move on to see how some of the official Kubernetes charts deal with these questions.

A Picture of a Helm Install

I want to install MySQL in my cluster. But I don’t want the MySQL version that the stable/MySQL chart sets in its values.yaml file in the official charts repo. So I create my own values.yaml file called "mysql-values.yaml" with just one line:

imageTag: “5.7.10”

And then I run helm install stable/mysql --values=mysqlvalues.yaml .

Helm generates a unique release name for me ("ignorant-camel") and MySQL is deployed in my cluster. The output of kubectl describe pod ignorant-camel-mysql-5dc6b947b-lf6p8   tells me that my chosen imageTag  has been applied.

Actually, I didn’t need to install anything in my cluster to see that my selection for imageTag  would be applied. I could have run helm install stabe/mysql --values=mysqlvalues.yaml --dry-run --debug   and Helm would have simply shown me the content of the Kubernetes deployment descriptor yaml it generated without installing anything.

The process of getting to the generated Kubernetes deployment descriptors can be better understood by thinking of the structure of a Helm Chart:

├── Chart.yaml
├── README.md
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── secrets.yaml
│   └── ...more yaml...
└── values.yaml

When the user runs  helm install stable/mysql  then the entries from the values.yaml in the chart and the Helm release information (such as unique release name) get injected into the templated yaml resource descriptors as the template is evaluated and rendered into pure Kubernetes deployment descriptors. When the user runs  helm install stable/mysql  with parameters or a values file, then the parameters or values file will be overlaid on the values in the chart. We should think of the in-chart values file as setting defaults that can be overridden.

So the values.yaml is the main interface between us as chart developers and our users. What we expose in the values.yaml determines what users can and can’t do with our charts.

Our charts and our values.yaml files also need to satisfy another kind of scenario. Another chart developer might like our app and decide they want to include it inside a package they want to release. So they add our chart to the "requirements.yaml" of a new chart they create, along with a bunch of other charts. They override some of our defaults using their values.yaml. And then they distribute it to users who do Helm install with that chart.

So our chart goes into other charts and they feed onto users in a charts food chain. Seeing how to satisfy our consumers in this food chain is one of the key challenges of writing Helm Charts.

The Challenges of Writing Helm Charts

Applications often have lots of configuration options and Kubernetes clusters can be configured in lots of different ways. So when writing a Helm Chart we naturally face questions like:

  • What if I forget to expose a configuration parameter in the values.yaml? What if my application parses configuration dynamically so that I can’t say upfront what all the possible parameter names will be?
  • How can I allow users to use resources defined not directly in my chart but in a chart in which my chart is used as a subchart?
  • What if another chart developer is using my chart in their own and they need to add significant sections inside a resource that I defined (e.g. add an extra container to a pod or a custom initialization script)?
  • How can I allow my users to expose the app externally in the various ways that they might want?

We can learn a lot about how to deal with these kinds of problems from the charts in the official Helm Charts repository. Let’s take a look at some of the patterns that those charts use, so that we can understand how to use the same patterns in our own charts.

Before diving in though it’s worth keeping in mind that the public charts are rather advanced. If you’re worrying about some of the above questions when developing your first Helm Charts, try to park the more difficult worries to begin with. Start by familiarising yourself with the Helm documentation and try to build a chart that works for just the simplest cases first. You can then add advanced options later.

So let’s try to get a snapshot of patterns from the official charts. This won’t be comprehensive and I expect new options to become available and new patterns to emerge when Helm 3 adds scripting with Lua. But it will give us look at the current state of the art.

1. Patterns for Exposing Configuration Parameters

Let’s say we’ve defined a Deployment resource in our template and we’ve configured the env section to allow some environment variables to be set from the values.yaml:

- name: ENV_VAR1
  value: {{ .Values.var1 }}
- name: ENV_VAR2
  value: {{ .Values.var2 }}

Now users of our chart can override these values in their values.yaml or with  --set var1=foo  . But what will our users be able to do if we’ve overlooked one? Or even worse, what if our application can dynamically parse configuration options (e.g. it might read ENV_VAR1  and create an internal variable called var1  )? Then there isn’t even a finite set of configuration option names to expose. So how do we let users set the variable names as well as the values?

As the Helm Charts developer guide says, we could create a configmap with a range function. A nice example of this is in the stable/unbound chart. It contains a configmap that defines its unbound.conf file. It mounts this file into Pods created by its Deployment. Inside the configmap it has entries such as:

{{- range .Values.localRecords }}
local-data: "{{ .name }} A {{ .ip }}"
local-data-ptr: "{{ .ip }} {{ .name }}"
{{- end }}


And its values.yaml lets the entries in localRecords be set as a list e.g.:

localRecords:
- name: "fake3.host.net"
  ip: "10.12.10.10"
- name: "fake4.host.net"
  ip: "10.13.10.10"


The Sonarqube chart applies a similar approach directly to environment variables, defining some explicitly and allowing further variables to be set using an extraEnv  collection:

{{- range $key, $value := .Values.extraEnv }}
 — name: {{ $key }}
   value: {{ $value }}
{{- end }}


So that variables would be set in a values.yaml like:

extraEnv:
- ENV_VAR1: var1
- ENV_VAR2: var2

Many of the official charts define an extraEnv but in slightly different ways. The Buildkite chart defines it differently, without using range. Instead, it takes the relevant values.yaml section and simply puts it in the template:

{{- if .Values.extraEnv }}
{{ toYaml .Values.extraEnv | indent 12 }}
{{- end }}

So this means that instead of setting extraEnv entries in the values.yaml as simple pairs we would also need to name each of the keys (name) and values (value) in the pairs like:

extraEnv:
 — name: ENV_VAR1
   value: "var1"
 — name: ENV_VAR2
   value: "var2"


The Keycloak chart does this differently again:

{{- with .Values.keycloak.extraEnv }}
{{ tpl . $ | indent 12 }}
{{- end }}

So it treats extraEnv as a string, sends it through the tpl function so that it is evaluated as part of the template and ensures that the correct indenting is applied. This is interesting as it allows values to be set like:

extraEnv: |
 — name: KEYCLOAK_LOGLEVEL
   value: DEBUG
 — name: HOSTNAME
   value: {{ .Release.Name }}-keycloak


Normally it wouldn’t be possible to use a template directive like  {{ .Release.Name }}  inside the values.yaml, but here we can because the content will go through tpl . This could be a big advantage in cases where we’re including the chart inside a parent chart that we’re developing and we need to refer to a service that is part of the same parent chart (more on this in the next section).The disadvantages are that it’s a bit more abstract and that the content in the values.yaml is treated as a string there so formatting issues aren’t necessarily found by your editor.

2. Referencing Mutually-Deployed Resources

Typically resources deployed through Helm are prefixed with a release name, so that multiple releases can be installed from the same chart (and in the same namespace) without naming conflicts between the installed resources. This means that template directives need to be used to prefix the resource names when one resource within a chart needs to refer to another, or to resources within the same parent chart.

A common case where a mutually-deployed resource needs to be referred to is a database secret. The Xray chart, for example, includes within it the option to deploy a Postgres database. It sets up its indexer component’s Deployment with credentials by pointing to the Postgres secret (which itself is part of the chart as a subchart and therefore also prefixed with the same release name):

{{- if .Values.postgresql.enabled }}
 — name: POSTGRES_USER
   value: {{ .Values.postgresql.postgresUser }}
 — name: POSTGRESS_PASSWORD
   valueFrom:
     secretKeyRef:
       name: {{ .Release.Name }}-postgresql
       key: postgres-password
 — name: POSTGRESS_DB
   value: {{ .Values.postgresql.postgresDatabase }}
 {{- else }}
...


Here the Xray chart knows which database it needs to connect to since it is itself including the Postgres chart. But what if we were developing a chart for an application where the user can choose to supply a database? Then it might be necessary to allow the user to include our chart inside a parent chart alongside a database of their choosing. How would we allow the user to set database configuration then?

One option would be the extraEnv  approach we saw already with the Keycloak chart. Then the user could set our chart’s extraEnv  to point to Postgres even if we’ve not defined it explicitly in our original chart. The values.yaml would have an entry like:

extraEnv: |
 — name: POSTGRES_USER
   value: {{ .Values.postgresql.postgresUser }}
 — name: POSTGRESS_PASSWORD
   valueFrom:
     secretKeyRef:
       name: {{ .Release.Name }}-postgresql
       key: postgres-password
 — name: POSTGRESS_DB
   value: {{ .Values.postgresql.postgresDatabase }}

The  |  is there because the section is treated as a string to be passed explicitly to the tpl  function.

A similar question emerges if a release name prefix needs to be used within an entry in a file that is loaded into a configmap. A typical way of loading a whole file into a configmap is to use .Files.Get  . However, like the values.yaml, the contents of files loaded in this way are then not part of the template and so template directives can’t be used in them. What does allow template directives to be used inside a file is to load the file content with .Files.Get and pass that into tpl. Then the data section of the configmap would have entries like:

conf_file1: {{ tpl (.Files.Get "files/conf_file1") . | quote }}


Or to load into a Secret we need to encode the content in base64:

conf_file1: {{ tpl (.Files.Get "files/conf_file1") . | b64enc | quote }}


Instead of just one file, we can load a set of files into a ConfigMap using  .Files.Glob  :

{{ (tpl (.Files.Glob "files/*").AsConfig . ) | indent 2 }}


Though we can’t quite apply the same approach with AsSecret  as then the content would be encoded before going through tpl  . To load a set of files into a secret we could use Glob to find the files and Get to load them:

{{ range $path, $bytes := .Files.Glob "files/*" }}
{{ base $path }}: '{{ tpl ($root.Files.Get $path) . | b64enc }}'
{{ end }}

3. Empowering Our Fellow Chart Developers

The use of extraEnv  in the Keycloak chart suggests a possible pattern for dealing with other kinds of definitions that might need to be injected into a chart. For example, the Keycloak chart needs to allow a user to package the Keycloak chart inside a parent chart that also supplies a user-defined a JSON file that the user needs to be able to mount into Keycloak Pods. The chart supports this by exposing extraVolumes :

{{- with .Values.keycloak.extraVolumes }}
{{ tpl . $ | indent 8 }}
{{- end }}


And extraVoumeMounts :

          volumeMounts:
            - name: scripts
              mountPath: /scripts
{{- with .Values.keycloak.extraVolumeMounts }}
{{ tpl . $ | indent 12 }}
{{- end }}


The user can then point to their secret (which contains the JSON file) and mount it through the values.yaml:

extraVolumes: |
 — name: custom-secret
   secret:
     secretName: custom-secret
extraVolumeMounts: |
 - name: custom-secret
   mountPath: "/realm/"
   readOnly: true


Effectively the configuration of volumes and volumeMounts  is externalized to the values.yaml. The chart also applies this pattern to allow users to inject further resources into the template such as initContainers  and even adding entire additional containers (or "sidecars"). This pattern gives chart users a lot of power and flexibility, especially for other chart developers using our chart.

The pattern can be especially powerful when combined with the Keycloak’s exposing of other parameters such as a preStartScript variable that is used within the chart’s initialization script:

{{- with .Values.keycloak.preStartScript }}                           
echo 'Running custom pre-start script...'                       
{{ . | indent 4 }}                         
{{- end }}


The user can use whatever shell commands they like in  .Values.keycloak.preStartScript  in their values.yaml. So this approach makes it possible for users to not just set their own configuration paramters but also load their own files and run their own custom scripts using those files.

4. Exposing Externally in Different Ways

The starter Helm Chart generated by  helm create  includes a Service specification but not an Ingress. Many of the public charts do define an Ingress resource. This needs to be configurable as users might not want to use ingress. So the RabbitMQ chart, like many others, wraps its whole Ingress resource definition with:

{{- if .Values.ingress.enabled }}
...
{{-end}

This is in line with the review guidelines for the official charts repo, which recommends that ingress should be disabled by default.

The RabbitMQ chart, for example, needs to provide the option whether to expose it by setting a host-based ingress rule:

rules:
  {{- if .Values.ingress.hostName }}
  - host: {{ .Values.ingress.hostName }}
    http:
  {{- else }}
  - http:
  {{- end }}


(Actually, the user might want multiple hosts to route to the service so the review guidelines suggest using a range function for hosts. The RabbitMQ chart just offers a single host.)

The RabbitMQ chart also allows for the host to be not set (the else condition above). In that case it’s likely that the user will instead override the path (so that RabbitMQ is exposed on a unique route, distinct from other exposed services):

- path: {{ default "/" .path }}
  backend:
    serviceName: {{ template "rabbitmq.fullname" . }}
    servicePort: {{ .Values.rabbitmq.managerPort }}


It’s also especially important that the user has flexibility to set the annotations on an ingress resource, since these are used to control routing configuration options. The review guidelines suggest supporting this with toYaml :

{{- with .Values.ingress.annotations }}
 annotations:
{{ toYaml . | indent 4 }}
{{- end }}


This allows the user of the chart to set annotations in the values.yaml such as:

annotations:
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/rewrite-target: /


So the user can set whatever annotations they need to on the ingress through the values.yaml without the chart needing to be aware of what those annotations might be. Just exposing these annotations can amount to giving the power to apply script-like configuration on the ingress routing. For example, with an NGINX ingress a user can apply rules to set headers in custom ways through configuration snippets:

annotations:
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/rewrite-target: /
  nginx.ingress.kubernetes.io/configuration-snippet: |
     more_set_headers 'Access-Control-Allow-Origin: $http_origin';    

What We've Learned About the Art of the Helm Chart

A good Helm Chart needs to anticipate what level of flexibility is needed by its users. More flexibility tends to mean either more complexity or greater abstraction or both. This can hamper chart readability and put more burden on chart users. The challenge is to choose the tools that best fit with what users need for that particular chart.

There are other concerns to balance, too, that we've not touched on, such as testing and security. This has just been a look at a particular slice of the official charts. I've tried to focus on patterns that seem particularly helpful to me in making sure that users can do what they need to do with your charts. The official Kubernetes charts have been extremely helpful to me in working on the Helm Charts for the Activiti project. Hopefully, the explanation in this post can help encourage others to dive into the official repo and take inspiration from its charts.

Topics:
helm ,kubernetes ,deployment ,cloud ,devops ,docker ,installation and configuration ,app deployment ,tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}