📛 🕑 🌷 Introducing Kubernetes CCM (Cloud Controller Manager) for Yandex.Cloud 🌖 🏬 👩🏼‍💻

In continuation to the recent release of the CSI driver for Yandex.Cloud, we are publishing another Open Source project for this cloud - Cloud Controller Manager . CCM is necessary not only for the cluster as a whole, but also for the CSI driver itself. Details about its purpose and some implementation features are under the cut.

Introduction

Why is this?

The motives that prompted us to develop CCM for Yandex.Cloud completely coincide with those already described in the announcement of the CSI driver. We serve many Kubernetes clusters from different cloud providers, for which we use a single tool. It implements numerous amenities "bypassing" the managed-solutions of these providers. Yes, we have a rather specific case and needs, but the developments created because of them can be useful to other users as well.

What is CCM in general?

As a rule, we prepare our environment for the cluster from the outside - for example, using Terraform. But sometimes there is a need to manage the surrounding cloud environment from a cluster . Such an opportunity is provided, and CCM implements it .

In particular, Cloud Controller Manager provides five basic types of interaction:

Instances - implements a 1: 1 relationship between a node object in Kubernetes ( Node) and a virtual machine in a cloud provider. To do this, we:
- fill in the field spec.providerIDin the object Node. For example, for OpenStack CCM, this field has the following format: openstack:///d58a78bf-21b0-4682-9dc6-2132406d2bb0. You can see the name of the cloud provider and the server’s unique UUID (virtual machine in OpenStack) of the object;
- nodeInfo Node . , instance type AWS;
- . , Node NotReady, , providerID. — Node, ;
Zones – failure domain Node, Pod’ ;
LoadBalancer – Service LoadBalancer , . , Yandex.Cloud NetworkLoadBalancer TargetGroup ;
Route – , .. Kubernetes pod IP- pod’. (VXLAN, GENEVE) :
Volume – PV, PVC SC. CCM, Container Storage Interface (CSI). CSI , , CSI-.

Previously, all the code interacting with the cloud lay mainly in the Kubernetes project Git repositories at the address k8s.io/kubernetes/pkg/cloudprovider/providers, but they decided to refuse this because of the inconvenience of working with a large code base. All old implementations were moved to a separate repository . For convenience of further support and development, all common components were also moved to a separate repository .

As with CSI, many large cloud service providers have already developed their CCM for cloud use in Kubernetes. If the supplier does not have CCM, but all the necessary functions are available through the API, then you can implement CCM yourself.

To write your CCM implementation, it’s enough to implement the necessary Go-interfaces .

ANDthat's what we got .

Implementation

How did you come to this

We started the development (or rather, even use) with the finished (!) CCM for Yandex.Cloud a year ago.

However, in this implementation we lacked:

authentication through the JWT IAM token;
Service controller support.

By agreement with the author (dlisin) in Telegram, we forked yandex-cloud-controller-manager and added the missing functions.

Key features

CCM currently supports the following interfaces:

the Instances ;
Zones ;
LoadBalancer .

In the future, when Yandex.Cloud starts working with the advanced features of VPC, we will add the Routes interface .

Loadbalanacer as the main challenge

Initially, we tried, as with other CCM implementations, to create a pair of LoadBalancerand TargetGroupfor each Servicewith a type LoadBalancer. However, Yandex.Cloud showed one interesting limitation: it cannot be used TargetGroupswith intersecting Targets(pair SubnetID- IpAddress).

Therefore, within the established CCM launched a controller that when changing the object Nodecollects information about all the interfaces on each virtual machine, groups them as belonging to certain NetworkIDcreates on TargetGroupon NetworkIDand making sure it's current. Subsequently, when creating an object Servicewith a type, LoadBalanacerwe simply attach the previously created one TargetGroupto the new ones NetworkLoadBalanacer.

How to start using?

CCM supports Kubernetes version 1.15 and higher. In the cluster, for its operation it is required that the flag --cloud-provider=externalbe set to the value truefor kube-apiserver, kube-controller-manager, kube-scheduler and all kubelet'ov.

All necessary steps for the installation itself are described in README . Installation boils down to creating objects in Kubernetes from manifests.

To use CCM you will also need:

specify in the manifest the identifier of the Yandex.Cloud directory ( folder-id);
service account for interacting with the Yandex.Cloud API. In the manifest, Secretyou must transfer the authorized keys to the service account. The documentation describes how to create a service account and get the keys.

We will be glad to receive feedback and new issues if you encounter any problems!

Summary

We have been using implemented CCM in five Kubernetes clusters over the past two weeks and plan to expand their number to 20 in the coming month. Using CCM for large and critical K8s installations is currently not recommended.

As in the case of CSI, we will be happy if the Yandex developers take on the development and support of this project - we are ready to transfer the repository at their request in order to deal with more specialized tasks for us.

Introducing Kubernetes CCM (Cloud Controller Manager) for Yandex.Cloud