🕴🏽 🧜🏼 🕧 Helm device and its pitfalls 📛 🙋 😪

Typhon freight hauler concept, Anton Swanepoel

My name is Dmitry Sugrobov, I’m a developer at Leroy Merlin. In the article I’ll tell you why Helm is needed, how it simplifies working with Kubernetes, what has changed in the third version and how to use it to update applications in production without downtime.

This is a compendium based on a speech at the @Kubernetes Conference by Mail.ru Cloud Solutions conference - if you do not want to read, watch the video.

Why we use Kubernetes in production

Leroy Merlin is a leader in the DIY retail market in Russia and Europe. Our company has more than one hundred developers, 33,000 internal employees and a huge number of people visiting hypermarkets and the site. In order to make all of them happy, we decided to adhere to standard approaches in the industry. Develop new applications using microservice architecture; use containers for isolation of environments and proper delivery; and for orchestration use Kubernetes. The price of using orchestrators is rapidly getting cheaper: the number of engineers with technology is growing in the market, and providers are appearing to offer Kubernetes as a service.

Everything that Kubernetes does, of course, can be done in other ways, for example, by coating scripts with some Jenkins and docker-compose, but why complicate life if there is a ready-made and reliable solution? Therefore, we came to Kubernetes and have been using it in production for a year now. We now have twenty-four Kubernetes clusters, the oldest of which is over a year old, with about two hundred pods in it.

Curse a lot of YAML files in Kubernetes

To run the microservice in Kubernetes, create at least five YAML files: for Deployment, Service, Ingress, ConfigMap, Secrets - and send it to the cluster. For the next application, we will write the same package of jamliks, with the third - one more and so on. We multiply the number of documents by the number of environments, we already get hundreds of files, and this is not taking into account dynamic environments.

Adam Reese, a core maintainer Helm, introduced the concept of “ development cycle at Kubernetes ”, which looks like this:

Copy YAML - copy the YAML file.
Paste YAML - paste it.
Fix Indents - fix indents.
Repeat - repeat again.

The option is working, but you have to copy YAML files many times. To change this cycle, and came up with Helm.

What is Helm

Firstly, Helm is a package manager that helps you find and install the programs you need. To install, for example, MongoDB, you do not need to go to the official site and download binaries, just run the command helm install stable/mongodb.

Secondly, Helm is a template engine that helps to parameterize files. Let's get back to the situation with YAML files in Kubernetes. It’s easier to write the same YAML file, add some placeholders into it, into which Helm will substitute the values. That is, instead of a large set of small boxes, there will be a set of templates (templates) into which the necessary values will be substituted at the right time.

Third, Helm is a deployment wizard . With it, you can install, roll back and update applications. Let's figure out how to do it.

How to use Helm to deploy native applications

Install Helm client on the computer, following the official instructions . Then create a set of YAML files. Instead of specifying specific values, we will leave placeholders, which Helm will fill with information in the future. A set of such files is called the Helm Chart. It can be sent to the Helm console client in three ways:

specify daddy with templates;
pack the archive in .tar and point to it;
put the template in a remote repository and add a link to the repository in the Helm client.

Still need a file with values - values.yaml. Data from there will be substituted into the template. Let's create it too.

The second version of Helm has an additional server application - Tiller. It hangs outside Kubernetes and waits for requests from the Helm client, and when called, substitutes the necessary values in the template and sends it to Kubernetes.

Helm 3 is simpler: instead of processing templates on the server, information is now processed entirely on the side of the Helm client and sent directly to the Kubernetes API. This simplification improves cluster security and facilitates a rollout scheme.

How it all works.

We start the team helm install. Specify the name of the application release, give the path to values.yaml. At the end, we indicate the repository in which the chart lies and the name of the chart. In the example, these are “lmru” and “bestchart”, respectively.

helm install --name bestapp --values values.yaml lmru/bestchart

The execution of the command is possible only once, when re-executing installit must be used instead upgrade. For simplicity, instead of two commands, you can execute a command upgradewith an additional key --install. At the first execution, Helm will send a command to install the release, and in the future will update it.

helm upgrade --install bestapp --values values.yaml lmru/bestchart

Pitfalls of deployment of new versions of the application with Helm

At this point in the story, I play with the audience in “Who Wants to Be a Millionaire,” and we figure out how to get Helm to update the application version. Watch the video .

When I studied the work of Helm, I was surprised by the strange behavior when trying to update versions of running applications. I updated the application code, uploaded a new image to the docker-registers, sent a command to deploy, and nothing happened. Below are some not entirely successful ways to update applications. Studying each of them in more detail, you begin to understand the internal structure of the instrument and the reasons for this not obvious behavior.

Method 1. Do not change the information since the last launch

As the official website saysHelm, “Kubernetes charts are large and complex, so Helm is trying once again not to touch anything.” Therefore, if you update the latest version of the application image in the docker registry and run the command helm upgrade, then nothing will happen. Helm will think that nothing has changed and there is no need to send a command to update the application to Kubernetes.

Hereafter, the latest tag is shown solely as an example. When this tag is specified, Kubernetes will download the image from the docker registry each time, regardless of the imagePullPolicy parameter. Using latest in production is undesirable and causes side effects.

Method 2. Update LABEL in image

As it is written in the same documentation , "Helm will update the application only if it has changed since the last release." A logical option for this seems to be updating the LABEL label in the docker image itself. However, Helm does not look into the application images and has no idea about any changes in them. Accordingly, when updating labels in the Helm image, it does not recognize them, and the application update command in Kubernetes will not be received.

Method 3. Use the key--force

We turn to the manuals and look for the desired key. The key is most appropriate in meaning --force. Despite the talking name, the behavior is different than expected. Instead of a forced application update, its real purpose is to restore a FAILED release. If you do not use this key, then you need to execute commands sequentially helm delete && helm install --replace. Instead, it is suggested that you use a key --forcethat automates the sequential execution of these commands. More information in this pull request . In order to tell Helm to update the application version, unfortunately, this key will not work.

Method 4. Change labels directly in Kubernetes

Updating a label directly in a cluster using a command kubectl editis a bad idea. This action will lead to inconsistency of information between the running application and what was initially sent to the deployment. The behavior of Helm during the deployment in this case is different from its version: Helm 2 will not do anything, and Helm 3 will deploy a new version of the application. To understand the reason, you need to understand how Helm works.

How is Helm

To determine if an application has changed since the last release, Helm can use:

running application in Kubernetes;
new values.yaml and current chart;
Helm internal release information.

For the most curious: where does Helm store internal release information?

helm history, , Helm.

. :

Helm , Tiller ( — kube-system), ConfigMap, «OWNER=TILLER»:

Helm , , . . , , .

The second Helm, when it tries to understand whether an update is needed, uses only two sources of information: what it has been provided with now, and the internal release information that lies in ConfigMap.

The third Helm uses the three-way merge strategy: in addition to that information, it also takes into account the application that is working right now in Kubernetes.

For this reason, the old version of Helm will not do anything, since it does not take into account the application information in the cluster, but Helm 3 will receive the changes and send the new application to the deploy.

Method 5. Use the key --recreate-pods

Using the key, --recreate-podsyou can achieve what was originally planned to be obtained using the key --force. The containers will restart and, according to the imagePullPolicy: Always policy for the latest tag (more on that in the footnote above), Kubernetes will download and launch a new version of the image. This will not be done in the best way: not taking into account the StrategyType deployment, it will sharply turn off all old instances of the application and start launching new ones. During the restart, the system will not work, users will suffer.

In Kubernetes itself, a similar problem also existed for a long time. And now, 4 years after the opening of Issue , the problem was fixed, and starting with the 1.15 version of Kubernetes, the possibility of rolling-restart pods appears.

Helm just turns off all applications and launches new containers nearby. In production, this cannot be done so as not to cause a simple application. This is only necessary for development needs, it can only be done in stage environments.

How to update the version of the application using Helm?

We will change the values sent to Helm. As a rule, these are values substituted in place of the image tag. In the case of latest, often used for unproductive environments, the role of mutable information is the annotation, which is useless for Kubernetes itself, and for Helm will act as a signal for the need to update the application. Options for filling in the annotation value:

Random value using the standard function - {{ randAlphaNum 6 }}.
There is a nuance: after each deployment using a chart with such a variable, the annotation value will be unique, and Helm will assume that there are changes. It turns out that we will always restart the application, even if we have not changed its version. This is not critical, as there will be no downtime, but still unpleasant.
— {{ .Release.Date }}.
.
— . SHA SHA — {{ .Values.sha }}.
Helm , . , . , Helm , .

Helm , Docker Registry : .
--force .
--recreate-pods , : . , .
Kubernetes kubectl edit : , Helm.
Helm . Issues Helm , .
. , .

Thought from the category of "world peace", working in all areas of life: read the instructions before use, and not after. Only with complete information will it be possible to build reliable systems and make users happy.

Other related links:

This talk was first made at the @Kubernetes Conference by Mail.ru Cloud Solutions. Watch a video of other performances and sign up for Telegram event announcements Around Kubernetes at Mail.ru Group .

Helm device and its pitfalls