DEVOXX UK. Kubernetes in production: Blue / Green deployment, autoscaling, and deployment automation. Part 1

Kubernetes is a great tool for running Docker containers in a clustered production environment. However, there are tasks that Kubernetes is unable to solve. With frequent deployments in a production environment, we need a fully automated Blue / Green deployment to avoid downtime in this process, which also requires external HTTP requests and SSL upload. This requires integration with a load balancer such as ha-proxy. Another task is the semi-automatic scaling of the Kubernetes cluster itself when working in the cloud, for example, partial reduction of the cluster scale at night.

Although Kubernetes does not have these features out of the box, it provides an API that can be used to solve such problems. Kubernetes cluster automated blue / green deployment and scaling tools were developed as part of the Cloud RTI open source project.

This video transcript describes how to configure Kubernetes along with other open source components to get a production-ready environment that accepts code from the git commit change commit without downtime in production.



Today we will talk about Kubernetes, in particular, about its automation. We will take a look at the basics of this system, and then move on to how to integrate Kubernetes into projects at the development stage. My name is Paul, I am from the Netherlands, I work for a company called Luminis Technologies and I am the author of the books “Designing Cloud Applications with OSGi” and “Modulating Java 9”. What I will talk about is very different from the theme of these books.
In the last year, most of my time was spent working on an infrastructure platform based on Kubernetes and creating tools for it, mainly on Golang. In addition, I am still continuing to work on the Java 9 book. My hobby is weightlifting. So let's get down to business.

Why take care of Kubernetes? First of all, because it allows you to run Docker in clusters. If you do not use Docker, then you are not in the subject. If you work with Docker, then you know that it is mainly designed to run containers on your own machine. It improves the work with many networks, however, all this is limited to one machine, so there is no way to run containers in a cluster.



The creators of Docker are constantly expanding its capabilities, but if you really need to run containers in production, you will need something like Kubernetes, because Docker with its command line will not help.

Let's talk about the basics of Kubernetes. Before we look at using the API for automation, we need to understand the concept of how this platform works. A typical cluster includes the following elements. The master node manages and controls the operation of the entire cluster and decides where the containers will be planned and how they will be launched. It launches the API server, which is the most important element of the platform’s work, later we will consider this issue in detail.



There is a whole bunch of Node work nodes that contain Pods pods with Docker containers, and distributed storage etcd. Nodes is the place where your containers will run. Kubernetes does not work with Docker containers directly, but uses an abstraction called Pod for this. Working nodes launch a bunch of containers, and the Master-node makes sure that they work. In addition, there is a distributed keyd-value type etcd storage, which stores all information about the cluster. Etcd is fault tolerant, so at least consists of 3 independent nodes. This ensures that the wizard works without losing state.

After starting Kubernetes with all of the above components, scheduling containers are deployed. Deployment means that containers start working in a cluster. To do this, first of all, you need an object called Replication Controller, launched on the master server, which allows you to create and monitor the status of several instances of hearths.



To configure the replication controller to create, for example, 5 replicas of containers in five different work nodes, you need to have so many nodes. In addition to the fact that the controller starts the operation of these containers, it monitors their status. If due to lack of memory one of the containers fails, the controller will record a decrease in the number from 5 to 4 and start a new container on one of the available nodes. This means that when Kubernetes is running, you do not start containers on any specific nodes, but set the required state - configure the platform to deploy 5 containers and transfer control to the replication controller, which takes care of maintaining this state. So Replication Controller is a very important concept.

As I said, Kubernetes does not work with containers directly. He does this through Pod - an abstraction on top of containers. It is important that Kubernetes operates not only with Docker containers, but also, for example, its alternative - rkt containers. This is not official support, because if you look at the Kubernetes technical documentation, it only talks about Docker containers. However, as the platform develops, compatibility with containers of other formats is added to it.



Each under can contain many containers with the same life cycle. This means that we will not be able to run containers for various purposes in the same pod, because all of them can exist only in the same time period. They start simultaneously and simultaneously stop their work. So if you work with microservices, you cannot put them all in the same sub. The containers inside the hearth “see” each other on the local host.

If you, for example, are deploying a web service, then the pod for this application will contain the nginx container, and nginx is not modified, but directly out of the box. In order for this nginx to work, we will need to add our own HTML files and put them in a separate container for web files. Both of these containers are placed in one common sub and begin to work together, as if they existed on the same physical machine, so they see files of the same file system located on the same local host. The services that provide the interaction of cluster components operate using env vars environment variables.

The next thing that provides under is networking. If you run several containers on the same physical machine, which can also be located in the cloud, and all these containers use ports, for example, we use 3 Java applications that work with the same port 8080, then this is fraught with conflict. To avoid it, you will need port mapping port mapping, which will greatly complicate the process of deploying applications.



Therefore, it would be great if each launched container had its own virtual IP address with access to all available ports, without thinking about their possible conflict. This is exactly what Kubernetes provides.

Each time he creates a pod, he creates his own virtual IP address for it. Ports are not shared between the rest of the pods, so there is no conflict. The only thing working at this IP address is what this container does. It all simplifies.
However, this raises a new problem: if your virtual IP address changes every time you restart the containers, which often happens, then how can you use these containers? Suppose we have 2 different pods that communicate with each other, but their IP addresses change all the time. Kubernetes uses a concept called services to solve this problem. These services configure proxies on top of your hearths and continue to use virtual IP addresses of a certain range, but these are fixed addresses that do not change all the time.

Therefore, when pods want to communicate with each other, they do this not directly, but through services, using these fixed IP addresses.



These services organize the traffic of all replicas exchanged between the cluster components. Therefore, services are very important to ensure communication between components.
The Services concept greatly facilitates the multi-component deployment process when multiple components are part of a larger application, such as the microservice architecture, although I personally don’t like dealing with microservices.



Each component you have can be deployed as a separate sub with its own life cycle. They are independent of each other and have separate namespaces, their own IP addresses and port number ranges, can be independently updated and scaled. Using services greatly facilitates inter-component communication. The following slide shows an example application of several components.



The first under Frontend is a user interface that uses several backend services, or replicas of backend services. You see a bundle of several such pods for which services are used as a proxy. You should not care about the performance of each pod, because if one of them fails, Kubernetes will automatically restart the new one. The backend service uses a number of other services located in different pods, and all of them use the concept of services for interaction. All of this works great, so if you use this architecture, Kubernetes can easily deploy it. If the architecture is built on a different principle, you may have problems.

The next important concept to be introduced is the Namespaces namespace.



These are isolated spaces within Kubernetes that contain hearths, replication controllers, and services. Isolation means, for example, that when the development environment is in one namespace and production is in another, you use services with the same name to prevent conflicts between different namespaces. This way, you can easily place different environments in the same physical Kubernetes cluster.

Deploying the application is a necessary step before releasing the application in production. The Kubernetes documentation states that you must use the kubectl command-line tool to complete the deployment. This tool is constantly being improved and is perfect for deploying anything.



You see the yaml configuration files that you need to create for the API using the kubectl command. The next slide shows what a typical yaml file looks like.



In this file we set the replication controller. The first important part is the line with the number of replicas that we want to replicas: 3. This number is equal to the number of nodes to be used. In this case, the number 3 means that we want to run pods on 3 different machines in our cluster. The replication controller monitors the state of the system and, if necessary, automatically restarts the pods to ensure continuous operation of a given number of replicas.

At the bottom of the code are lines representing the specification of the hearth. It describes the ports, storage nodes, the name and image of the web server and other information necessary for our Docker container.

Finally, there is a whole bunch of metadata here, such as labels. Labels are key-value pairs and are added to objects like pods. They are used to group and select subsets of objects. They are very important for organizing API interactions, linking controllers, pods and services together.

Pod does not know which replication controller created it, and the replication controller does not know the list of pods that it creates. But both the pods and the controller have labels, the coincidence of which indicates that a specific sub belongs to a specific controller. This is a fairly weak, but flexible connection, which ensures the stable operation of the API and automated things.

Now you will see a short demo showing the work of kubectl, and then we will go deeper into the work of the load balancer and using the API. So, I enter the kubectl get pods command, but the system does not show any hearths because we haven't run anything yet. Next, I enter the ls command to view a list of available files.



Now I enter the name of the first file from the list and the configuration yaml file is displayed, similar to the one we just looked at. Let's use this file to create a hearth by typing kubectl create –f nginx-controller.yaml. As a result, we will have created under. By repeating the kubectl get pods command, you can see that this one works.



But this is just one under. Let's try to scale our controller by creating several replicas. To do this, I enter the command kubectl scale rc nginx - replicas = 5, where rc is the replication controller, and 5 is the required number of instances, or replicas of this controller. You see the progress of creating containers, and if you re-enter the kubectl get pods command after a couple of seconds, you can see that most of the created containers have already started working.



Thus, we increased the number of hearths or replicas working in our cluster to 5 copies. Also, for example, you can take a look at the IP address created for the first replica by entering the kubectl describe pod nginx-772ia command. This is the virtual IP address that is attached to this particular container.



His work will be provided by the services mentioned above. Let's see what happens if you destroy one of the working replicas, because in any case, we must ensure the work of a given number of copies. I enter the kubectl delete pod nginx-772ia command and the system reports that under with that identifier has been deleted. Using the kubectl get pods command, we see that 5 controller instances are working again, and instead of the remote replica nginx-772ia, a new one with the identifier nginx-sunfn has appeared.



This happened because Kubernetes noticed the failure of one of the replicas. In reality, this was not an accidental malfunction, but a deliberate action, so far as I personally deleted it. But since the number of replicas specified by the cluster configuration remained unchanged, the replication controller noted the disappearance of one of the instances and immediately restored the required state of the cluster by creating and starting a new container with a new ID. Of course, this is a stupid example when I delete a replica myself and wait for the controller to restore it, but similar situations occur in real life due to a container failure, for example, due to lack of memory. In this case, the controller will also restart under.

If we have several production scenarios in which the Java application is not configured correctly, for example, insufficient memory is installed, this can cause a crash both a few hours and a week after the program starts. In such cases, Kubernetes takes care that the workflow is not interrupted.

Let me remind you once again that all of the above is not applicable at the production stage, when you should not use the kubectl command line and other things entered from the keyboard to ensure the operation of the product. So before we start deploying automation, we need to solve another big problem. Suppose I have an nginx container running right now, but I don’t have access to it from the “outside world”. However, if it is a web application, then I can access it via the Internet. To ensure the stability of this process, a load balancer is used, which is located in front of the cluster.
We need to be able to manage Kubernetes services from the outside world, and this feature is not automatically supported right out of the box. For example, to ensure HTTP-balancing, you need to use SSL offloading, or SSL upload. This is the process of removing SSL-based encryption from inbound traffic that the web server receives in order to free the server from data decryption. To balance traffic, you can also use Gzip-compression of web pages.

Recent versions of Kubernetes, such as 1.02, have a new feature called Ingress. It serves to configure the load balancer. This is a new concept in the API where you can write a plugin that will configure any external load balancer you use.



Unfortunately, today this concept is not finalized and is offered in the alpha version. You can demonstrate how it will work in the future, but it will not match the way it works today. I use the balancer provided by Google Cloud Engine, however if you do not work with this service, you will have to do something yourself, at least at the moment. In the future, probably in a month, it will become much easier - you can use the ingress function to balance traffic.

Today, this problem is solved by creating a custom traffic balancer. First of all, you can use the ha-proxy located in front of the Kubernetes cluster. It runs outside the cluster on an external machine or set of machines. Next, it is necessary to provide automatic dynamic configuration of the ha-proxy, so as not to configure it manually when creating each new hearth. Similar work needs to be done for nginx, Apache and other servers, and the ha-proxy acts as an easily integrated tool.

The following slide shows how the load balancer node handles incoming HTTPS traffic, and the ha-proxy is located on an external machine or cluster of machines located outside of the Kubernetes cluster. SSL upload is also located here. The proxy detects the requested addresses and then passes the traffic to Kubernetes services, without being interested in the IP addresses of the hearths that can change over time. Next, services pass traffic to running pods with dynamic IP addresses.



If you use AWS, you can use the ELB load balancer concept. Elastic Load Balancing redirects traffic to healthy Amazon instances to ensure application stability. This mechanism is especially good if you need scalability of the load balancer.



In this case, the traffic is first sent to the AWS service, where SSL is unloaded, after which it enters the ha-proxy load balancer node.

You can do the same if your clusters are completely inside a private virtual VPN network. In this case, the use of AWS ELB fully ensures the security of the system. You do not need to implement SSL between various components outside the cluster. The only thing that will be connected with the outside world is our ha-proxy.



Conceptually, it looks quite simple, but how does it work in reality? How do I set up a ha-proxy so that it knows about Kubernetes services? If you look at ha-proxy in the same way as we consider nginx, you will notice that it still uses static configuration files. So if you want to add a new backend to it, which in this case is our Kubernetes services, we need to change the configuration file, reload ha-proxy, and only after that everything will work as it should. Of course, we do not want to do this manually. Therefore, you can use a small open-source tool called Confd, which manages local application configuration files using etcd data. If changes occur in the metadata store etcd, Confd automatically generates a configuration template,which reconfigures the ha-proxy in accordance with the new working conditions.

It may seem that this approach requires additional efforts, however, it is a very simple tool that works with templates, so its use is a rather trivial task.

26:00 min.

Continuation will be very soon ...

A bit of advertising :)

Thank you for staying with us. Do you like our articles? Want to see more interesting materials? Support us by placing an order or recommending to your friends, cloud VPS for developers from $ 4.99 , a unique analog of entry-level servers that was invented by us for you: The whole truth about VPS (KVM) E5-2697 v3 (6 Cores) 10GB DDR4 480GB SSD 1Gbps from $ 19 or how to divide the server? (options are available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).

Dell R730xd 2 times cheaper at the Equinix Tier IV data center in Amsterdam? Only we have 2 x Intel TetraDeca-Core Xeon 2x E5-2697v3 2.6GHz 14C 64GB DDR4 4x960GB SSD 1Gbps 100 TV from $ 199 in the Netherlands!Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - from $ 99! Read about How to Build Infrastructure Bldg. class c using Dell R730xd E5-2650 v4 servers costing 9,000 euros for a penny?