Get best of the week

Learning to deploy microservices. Part 3. Helm


Hello, Habr!


This is the third part in a series of articles “Learning to Deploy Microservices ,” and today we will talk about Helm 3. In the last part, we created Kubernetes configuration for a training project of 2 microservices (backend and gateway) and deployed all this in Google Kubernetes Engine. In this article, we will write a Helm chart for our system, create a repository for it based on GitHub Pages, and deploy a project in GKE using Helm.


:


  1. Spring Boot, Docker


    : Java 11, Spring Boot, Docker, image optimization


  2. Kubernetes Google Kubernetes Engine


    : Kubernetes, GKE, resource management, autoscaling, secrets


  3. Helm 3


    : Helm 3, chart deployment


  4. Jenkins


    : Jenkins configuration, plugins, separate configs repository



Helm — , , Kubernetes. Helm — yaml- , . Helm-. .


Helm Kubernetes , . values.yaml, Kubernetes-. , .


GitHub .



:


helm create <chart-name>

Helm. msvc-chart :


.
└── msvc-chart
    ├── charts
    ├── Chart.yaml
    ├── templates
    │   ├── backend.yaml
    │   ├── gateway.yaml
    │   ├── _helpers.tpl
    │   ├── NOTES.txt
    │   ├── secrets.yaml
    │   ├── service-account.yaml
    │   ├── tests
    │   │   └── interaction-test.yaml
    │   └── urls-config.yaml
    ├── values.schema.json
    └── values.yaml

  • charts/ — . , .


  • Chart.yaml — , .


  • *templates/*.yaml* — Kuberenetes.


  • templates/NOTES.txt — , .


  • *templates/tests/*.yaml* — Helm.


  • templates/_helpers.tpl — , . tpl.


  • values.yaml — .


  • values.schema.json — JSON- values.yaml.



values.yaml


— values.yaml. . .


backend:
  deployment:
    name:
    replicas: 2
  container:
    name:
    resources: {}
#      limits:
#        memory: 1024Mi
#        cpu: 500m
#      requests:
#        memory: 512Mi
#        cpu: 100m
  service:
    name:
    port: 8080
  image:
    name: anshelen/microservices-backend
    tag: latest
    pullPolicy: IfNotPresent
  hpa:
    enabled: false
    name:
    minReplicas: 1
    maxReplicas: 3
    targetCPUUtilizationPercentage: 50

gateway:
  deployment:
    name:
    replicas: 2
  container:
    name:
    resources: {}
#      limits:
#        memory: 1024Mi
#        cpu: 500m
#      requests:
#        memory: 512Mi
#        cpu: 100m
  service:
    name:
    port: 80
#   Can be one of ClusterIP, NodePort or LoadBalancer
    type: LoadBalancer
  image:
    name: anshelen/microservices-gateway
    tag: latest
    pullPolicy: IfNotPresent
  hpa:
    enabled: false
    name:
    minReplicas: 1
    maxReplicas: 3
    targetCPUUtilizationPercentage: 50

secrets:
  secret: default-secret

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the release and
  # chart names
  name:

, . , (№ gateway.service.type). , Kubernetes- (№ gateway.container.resources).


values.schema.json JSON-, . , , .


values.schema.json
{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "properties": {
    "backend": {
      "properties": {
        "deployment": {
          "properties": {
            "name": {
              "type": ["string", "null"]
            },
            "replicas": {
              "minimum": 1,
              "type": "integer"
            }
          },
          "type": "object"
        },
        "container": {
          "properties": {
            "name": {
              "type": ["string", "null"]
            },
            "resources": {
              "properties": {
                "limits": {
                  "properties": {
                    "memory": {
                      "type": ["string", "number"]
                    },
                    "cpu": {
                      "type": ["string", "number"]
                    }
                  },
                  "type": "object"
                },
                "requests": {
                  "properties": {
                    "memory": {
                      "type": ["string", "null"]
                    },
                    "cpu": {
                      "type": ["string", "null"]
                    }
                  },
                  "type": "object"
                }
              },
              "type": "object"
            }
          },
          "type": "object"
        },
        "service": {
          "properties": {
            "name": {
              "type": ["string", "null"]
            },
            "port": {
              "minimum": 1,
              "type": "integer"
            }
          },
          "type": "object"
        },
        "image": {
          "properties": {
            "name": {
              "type": "string"
            },
            "tag": {
              "type": "string"
            },
            "pullPolicy": {
              "enum": ["IfNotPresent", "Always", "Never"]
            }
          },
          "type": "object"
        },
        "hpa": {
          "properties": {
            "enabled": {
              "type": "boolean"
            },
            "name": {
              "type": ["string", "null"]
            },
            "minReplicas": {
              "minimum": 1,
              "type": "integer"
            },
            "maxReplicas": {
              "minimum": 1,
              "type": "integer"
            },
            "targetCPUUtilizationPercentage": {
              "minimum": 1,
              "maximum": 99,
              "type": "integer"
            }
          },
          "type": "object"
        }
      },
      "type": "object"
    },
    "gateway": {
      "properties": {
        "deployment": {
          "properties": {
            "name": {
              "type": ["string", "null"]
            },
            "replicas": {
              "minimum": 1,
              "type": "integer"
            }
          },
          "type": "object"
        },
        "container": {
          "properties": {
            "name": {
              "type": ["string", "null"]
            },
            "resources": {
              "properties": {
                "limits": {
                  "properties": {
                    "memory": {
                      "type": ["string", "number"]
                    },
                    "cpu": {
                      "type": ["string", "number"]
                    }
                  },
                  "type": "object"
                },
                "requests": {
                  "properties": {
                    "memory": {
                      "type": ["string", "null"]
                    },
                    "cpu": {
                      "type": ["string", "null"]
                    }
                  },
                  "type": "object"
                }
              },
              "type": "object"
            }
          },
          "type": "object"
        },
        "service": {
          "properties": {
            "name": {
              "type": ["string", "null"]
            },
            "port": {
              "minimum": 1,
              "type": "integer"
            },
            "type": {
              "enum": ["ClusterIP", "NodePort", "LoadBalancer"]
            }
          },
          "type": "object"
        },
        "image": {
          "properties": {
            "name": {
              "type": "string"
            },
            "tag": {
              "type": "string"
            },
            "pullPolicy": {
              "enum": ["IfNotPresent", "Always", "Never"]
            }
          },
          "type": "object"
        },
        "hpa": {
          "properties": {
            "enabled": {
              "type": "boolean"
            },
            "name": {
              "type": ["string", "null"]
            },
            "minReplicas": {
              "minimum": 1,
              "type": "integer"
            },
            "maxReplicas": {
              "minimum": 1,
              "type": "integer"
            },
            "targetCPUUtilizationPercentage": {
              "minimum": 1,
              "maximum": 99,
              "type": "integer"
            }
          },
          "type": "object"
        }
      },
      "type": "object"
    },
    "secrets": {
      "properties": {
        "secret": {
          "type": ["number", "string"]
        }
      },
      "type": "object"
    },
    "createAccount": {
      "properties": {
        "create": {
          "type": "boolean"
        },
        "name": {
          "type": ["string", "null"]
        }
      },
      "type": "object"
    }
  },
  "title": "Values",
  "type": "object"
}


Helm , Kubernetes.


— . Helm , (Release), (Values) (Chart). , secrets.secret {{ .Values.secrets.secret }}.


, "" ("pipeline") — . , : {{ .Chart.AppVersion | quote }}. .


, . define:


{{- define "msvc-chart.someFragment" -}}
...
{{- end -}}

include:


{{ include "msvc-chart.someFragment" . }}

{{- {{ , . . , {{- include "msvc-chart.someFragment" . | nindent 4 }} 4 .


.


_helpers.tpl


Helm . Kubernetes , . , . , , . — , , . Release.Name.



:


{{- define "msvc-chart.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}

.Chart.Name ('msvc-chart' ). , Helm.


:


{{- define "msvc-chart.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

, , 63 ( Kubernetes) . Helm. Kubernetes- ( DNS ).


, -Kubernetes:


{{- define "msvc-chart.labels" -}}
helm.sh/chart: {{ include "msvc-chart.chart" . }}
{{ include "msvc-chart.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}

, 'msvc-project', :


helm.sh/chart: msvc-chart-1.0.0
app.kubernetes.io/name: msvc-chart
app.kubernetes.io/instance: msvc-project
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm

Kubernetes


Kubernetes ServiceAccount. . , , . ServiceAccount . ServiceAccount , Jenkins .


:


  • serviceAccount.create=true, serviceAccount.name. .
  • serviceAccount.create=false, serviceAccount.name 'default', .

{{- define "msvc-chart.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}}
    {{ default (printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-") .Values.serviceAccount.name }}
{{- else -}}
    {{ default "default" .Values.serviceAccount.name }}
{{- end -}}
{{- end -}}

if default, , null.


Kubernetes , .


:


{{- define "msvc-chart.gateway.defaultName" -}}
{{- printf "gateway-%s" .Release.Name -}}
{{- end -}}

{{- define "msvc-chart.gateway.deployment.name" -}}
{{- default (include "msvc-chart.gateway.defaultName" .) .Values.gateway.deployment.name | trunc 63 | trimSuffix "-" -}}
{{- end -}}


, . , , Helm Kubernetes- . , . . Helm checksum/config, , . Helm , , .


:


{{- define "msvc-chart.propertiesHash" -}}
{{- $secrets := include (print $.Template.BasePath "/secrets.yaml") . | sha256sum -}}
{{- $urlConfig := include (print $.Template.BasePath "/urls-config.yaml") . | sha256sum -}}
{{ print $secrets $urlConfig | sha256sum }}
{{- end -}}

print , $.Template.BasePath (_helpers.tpl). include , - $secrets. ConfigMap, .


, include '. ', '\$' Template. Helm '. ' (scope), '\$' — , . , . . .



Secret (secrets.yaml)


apiVersion: v1
kind: Secret
metadata:
  name: {{ include "msvc-chart.secrets.defaultName" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
type: Opaque
stringData:
  secret: {{ .Values.secrets.secret }}

ConfigMap (urls-config.yaml)


apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "msvc-chart.urlConfig.defaultName" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
data:
  BACKEND_URL: "http://{{ include "msvc-chart.backend.service.name" . }}:{{ .Values.backend.service.port }}/"

ServiceAccount (service-account.yaml)


{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ include "msvc-chart.serviceAccountName" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: "{{ include "msvc-chart.serviceAccountName" . }}-binding"
  labels:  
    {{- include "msvc-chart.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
  name: {{ include "msvc-chart.serviceAccountName" . }}
  namespace: {{ .Release.Namespace }}
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io
{{- end -}}

serviceAccount.create=true, .


Deployments (backend.yaml gateway.yaml)


gateway.yaml , , backend.yaml — . , HorizontalPodAutoscaler', . , gateway.container.resources.requests.cpu — . Kubernetes .


gateway.yaml:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "msvc-chart.gateway.deployment.name" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
    tier: gateway
spec:
  replicas: {{ .Values.gateway.deployment.replicas }}
  selector:
    matchLabels:
      {{- include "msvc-chart.selectorLabels" . | nindent 6 }}
      tier: gateway
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: {{ include "msvc-chart.propertiesHash" . }}
      labels:
        {{- include "msvc-chart.selectorLabels" . | nindent 8 }}
        tier: gateway
    spec:
      serviceAccountName: {{ include "msvc-chart.serviceAccountName" . }}
      containers:
        - name: {{ include "msvc-chart.gateway.container.name" . }}
          image: "{{ .Values.gateway.image.name }}:{{ .Values.gateway.image.tag }}"
          imagePullPolicy: {{ .Values.gateway.image.pullPolicy }}
          envFrom:
            - configMapRef:
                name: {{ include "msvc-chart.urlConfig.defaultName" . }}
          env:
            - name: SECRET
              valueFrom:
                secretKeyRef:
                  name: {{ include "msvc-chart.secrets.defaultName" . }}
                  key: secret
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 3
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            {{- toYaml .Values.gateway.container.resources | nindent 12 }}

---

apiVersion: v1
kind: Service
metadata:
  name: {{ include "msvc-chart.gateway.service.name" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
    tier: gateway
spec:
  type: {{ .Values.gateway.service.type }}
  ports:
    - port: {{ .Values.gateway.service.port }}
      protocol: TCP
      targetPort: 8080
      name: http
  selector:
    {{- include "msvc-chart.selectorLabels" . | nindent 4 }}
    tier: gateway

---

{{- if .Values.gateway.hpa.enabled -}}
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "msvc-chart.gateway.hpa.name" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
    tier: gateway
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "msvc-chart.gateway.deployment.name" . }}
  minReplicas: {{ .Values.gateway.hpa.minReplicas }}
  maxReplicas: {{ .Values.gateway.hpa.maxReplicas }}
  targetCPUUtilizationPercentage: {{ .Values.gateway.hpa.targetCPUUtilizationPercentage }}
{{- end -}}

"checksum/config", , ServiceAccount (spec.template.spec.serviceAccountName). toYaml backend.container.resources values.yaml .


.


backend.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "msvc-chart.backend.deployment.name" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
    tier: backend
spec:
  replicas: {{ .Values.backend.deployment.replicas }}
  selector:
    matchLabels:
      {{- include "msvc-chart.selectorLabels" . | nindent 6 }}
      tier: backend
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        {{- include "msvc-chart.selectorLabels" . | nindent 8 }}
        tier: backend
    spec:
      serviceAccountName: {{ include "msvc-chart.serviceAccountName" . }}
      containers:
        - name: {{ include "msvc-chart.backend.container.name" . }}
          image: "{{ .Values.backend.image.name }}:{{ .Values.backend.image.tag }}"
          imagePullPolicy: {{ .Values.backend.image.pullPolicy }}
          ports:
            - containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 3
          resources:
            {{- toYaml .Values.backend.container.resources | nindent 12 }}

---

apiVersion: v1
kind: Service
metadata:
  name: {{ include "msvc-chart.backend.service.name" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
    tier: backend
spec:
  ports:
    - port: {{ .Values.backend.service.port }}
      protocol: TCP
      targetPort: 8080
      name: http
  selector:
    {{- include "msvc-chart.selectorLabels" . | nindent 4 }}
    tier: backend

---

{{- if .Values.backend.hpa.enabled -}}
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "msvc-chart.backend.hpa.name" . }}
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
    tier: backend
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "msvc-chart.backend.deployment.name" . }}
  minReplicas: {{ .Values.backend.hpa.minReplicas }}
  maxReplicas: {{ .Values.backend.hpa.maxReplicas }}
  targetCPUUtilizationPercentage: {{ .Values.backend.hpa.targetCPUUtilizationPercentage }}
{{- end -}}

NOTES.txt


NOTES.txt , , (helm status <name>).


URL- . , , . , .


1. Get the application URL by running these commands:
{{- if contains "NodePort" .Values.gateway.service.type }}
  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "msvc-chart.gateway.service.name" . }})
  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.gateway.service.type }}
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "msvc-chart.gateway.service.name" . }}'
  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "msvc-chart.gateway.service.name" . }} -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
  echo http://$SERVICE_IP:{{ .Values.gateway.service.port }}
{{- else if contains "ClusterIP" .Values.gateway.service.type }}
  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name= {{ .Chart.Name }},app.kubernetes.io/instance={{ .Release.Name }},tier=gateway" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:8080
{{- end }}
2. Get service account token:
  export TOKEN=$(kubectl get serviceaccount {{ include "msvc-chart.serviceAccountName" . }} -n {{ .Release.Namespace }} -o go-template --template='{{`{{range .secrets}}{{.name}}{{"\n"}}{{end}}`}}')
  export TOKEN_DECODED=$(kubectl get secrets "$TOKEN" -n {{ .Release.Namespace }} -o go-template --template '{{`{{index .data "token"}}`}}' | base64 -d)
  echo $TOKEN_DECODED

Chart.yaml


. — version appVersion. version — , . appVersion — . type 'application' 'library' — .


apiVersion: v2
name: msvc-chart
version: 1.0.0
description: Microservices project
type: application
sources:
  - https://github.com/Anshelen/microservices-backend
  - https://github.com/Anshelen/microservices-gateway
  - https://github.com/Anshelen/microservices-deploy
maintainers:
  - name: Anton Shelenkov
    email: anshelen@yandex.ru
    url: https://shelenkov.herokuapp.com
appVersion: 1.0.0


Helm . helm test <name> (№ ). Kubernetes-, , http-. , .


, . Kubernetes- Job — , . , Job 3 ( backoffLimit). helm.sh/hook helm.sh/hook-delete-policy. Helm, , .


interaction-test.yaml:


apiVersion: batch/v1
kind: Job
metadata:
  name: "{{ .Release.Name }}-interaction-test"
  labels:
    {{- include "msvc-chart.labels" . | nindent 4 }}
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": hook-succeeded,hook-failed
spec:
  template:
    spec:
      containers:
      - name: test
        image: busybox
        command: ['wget']
        args:  ['{{ include "msvc-chart.gateway.service.name" . }}:{{ .Values.gateway.service.port }}/']
      restartPolicy: Never
  backoffLimit: 3

Helm


Helm : , , Helm, kubectl. Helm , . kubectl, Helm , .



, Kubernetes-:


helm install --dry-run --debug <name> <chart-folder>

, , , <name>:


helm install <name> <chart-folder>

values.yaml, --set <key>=<value>. . , yaml- -f <overriden-values.yaml>. -n <namespace> , .



:


helm upgrade <name> <chart-folder>

-f <file> --set <key>=<value>. Helm , . , --reuse-values. , Helm , --set --reuse-values , - . :


helm install <name> <chart-folder>
helm upgrade <name> <chart-folder> --set key=value --reuse-values

, , - , , :


helm install <name> <chart-folder> --set _=null


helm list — Helm' .
helm status <name> — .
helm history <name> — . . .
helm get values <name> — . --revision n, n- .
helm show values <chart-folder> — . values.yaml.
helm get manifest <name> — Helm' .



helm rollback <name> <revision-num> — .
helm uninstall <name> — .



Helm- . HTTP- S3-. GitHub Pages.


index.yaml , . — .


GitHub docs :


helm package msvc-chart/ -d docs/
helm repo index docs/ --url https://<github-username>.imtqy.com/<github-repo-name>/

docs. index.yaml . url URL- . , . , GitHub Pages, 'Source' 'master branch /docs folder'.



, GKE, Helm.


:


kubectl create namespace msvc-ns
kubectl config set-context --current --namespace=msvc-ns

. Helm msvc-repo (URL , Helm-):


helm repo add msvc-repo https://anshelen.imtqy.com/microservices-deploy/
helm repo update

:


helm install msvc-project msvc-repo/msvc-chart \
--set backend.container.resources.requests.cpu=50m \
--set backend.hpa.enabled=true \
--set gateway.container.resources.requests.cpu=50m \
--set gateway.hpa.enabled=true \
--set secrets.secret=secret

URL- . , , .


:


helm test msvc-project

:


helm upgrade msvc-project msvc-repo/msvc-chart \
--set secrets.secret=new-secret \
--reuse-values

- , .



Helm- , GKE. Helm , Git — , Docker, Helm.


In order to automate the process of delivering code to the cluster, in the next article we will configure the Jenkins continuous integration server and create a pipeline for our project.

More articles:


All Articles