Our experience working with data in etcd Kubernetes-cluster directly (without K8s API)

• The subnet is 192.168.*.*often used in office networks of clients, and even more often in home networks of developers. And then we get conflicts: home routers work on this subnet and the VPN push these subnets from the cluster to the client.
• We have several clusters (production, stage, and / or several dev clusters). Then in all of them by default there will be the same subnets for pods and services, which creates great difficulties for working with services in several clusters simultaneously.

Searching of decisions

The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf.
Since I still did not find the solution, i had to reset the entire cluster with kubeadm reset and init it again.

• Used by Flannel;
• There are clusters both in the clouds and on the iron;
• I would like to avoid the repeated deployment of all services in the cluster;
• There is a need to do everything with a minimum of problems;
• Kubernetes version is 1.16.6 (however, further actions will be similar for other versions);
• The main task is to 192.168.0.0/16replace it with a cluster deployed using kubeadm with a service subnet 172.24.0.0/16.

Add etcdhelper

• create a deserializer;
• compile a regular expression to replace CIDR;
• we pass through all services with the ClusterIP type in the cluster:
  
  
  • decode the value from etcd to the Go object;
  • using a regular expression, replace the first two bytes of the address;
  • assign the service an IP address from a new subnet;
  • create a serializer, convert the Go object to protobuf, write new data to etcd.

Practice

Change serviceCIDR

• installing the necessary software and assembling the patched etcdhelper;
• backup etcd and /etc/kubernetes.

• Modifying apiserver and controller-manager manifests
• reissue of certificates;
• ClusterIP services change in etcd;
• restart all pods in a cluster.

apt install etcd-client

• We put golang:
  
  

  GOPATH=/root/golang
  mkdir -p $GOPATH/local
  curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
  echo "export GOPATH=\"$GOPATH\"" >> ~/.bashrc
  echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
  echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

• We save ourselves etcdhelper.go, load the dependencies, collect:
  
  

  wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
  go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
  go build -o etcdhelper etcdhelper.go

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

1. Let's look at which domains and IP addresses the current certificate is issued:
   
   

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

2. Prepare the minimum config for kubeadm:
   
   

   cat kubeadm-config.yaml
   apiVersion: kubeadm.k8s.io/v1beta1
   kind: ClusterConfiguration
   networking:
     podSubnet: "10.244.0.0/16"
     serviceSubnet: "172.24.0.0/16"
   apiServer:
     certSANs:
     - "192.168.199.100" # IP-  

3. Let's delete the old crt and key, because without this a new certificate will not be issued:
   
   

   rm /etc/kubernetes/pki/apiserver.{key,crt}

4. Re-issue certificates for the API server:
   
   

   kubeadm init phase certs apiserver --config=kubeadm-config.yaml

5. , :
   
   

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

6. API- :
   
   

   docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

7. admin.conf:
   
   

   kubeadm alpha certs renew admin.conf

8. etcd:
   
   

   ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 

   
   ! , pod' /etc/resolv.conf CoreDNS (kube-dns), kube-proxy iptables . .
9. ConfigMap' kube-system:
   
   

   kubectl -n kube-system edit cm kubelet-config-1.16

   
   — clusterDNS IP- kube-dns: kubectl -n kube-system get svc kube-dns.
   
   

   kubectl -n kube-system edit cm kubeadm-config

   
   — data.ClusterConfiguration.networking.serviceSubnet .
10. kube-dns, kubelet :
    
    

    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet

11. It remains to restart all the pods in the cluster:
    
    

    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete pod \2/e'

Minimized Downtime

1. After changing the manifestations of the control plane, create a new kube-dns service, for example, with a name kube-dns-tmpand a new address 172.24.0.10.
2. Make ifin etcdhelper, which will not modify the kube-dns service.
3. Replace the address in all kubelets ClusterDNSwith a new one, while the old service will continue to work simultaneously with the new one.
4. Wait for pods with applications to roll either by themselves for natural reasons, or at an agreed time.
5. Delete the service kube-dns-tmpand change it serviceSubnetCIDRfor the kube-dns service.

Modification podNetwork

• we fix configs in kube-system;
• we fix the manifest of kube-controller-manager;
• we change podCIDR directly in etcd;
• reboot all nodes of the cluster.

kubectl -n kube-system edit cm kubeadm-config

kubectl -n kube-system edit cm kube-proxy

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

Total

PS

• “ Etcd 3.4.3: storage reliability and security research ”;
• “ Calico for the web at Kubernetes: getting to know and a bit of experience ”;
• “ 6 entertaining system bugs in the operation of Kubernetes [and their solution] ”;
• « Kubernetes».