Note perev. : The DNS problem in Kubernetes, or rather, parameter settings ndots, is surprisingly popular, and this is not the first year . In another article on this subject, its author, DevOps engineer from a large brokerage company in India, tells in a very simple and concise manner what it is useful to know for colleagues who use Kubernetes.
One of the main advantages of deploying applications in Kubernetes is the seamless discovery of applications. Intracluster interaction greatly simplified by service concept ( Service ), which is a virtual IP, supporting a set pod'ov IP-addresses. For example, if a servicevanillawants to contact the servicechocolate, it can access the virtual IP directly forchocolate. The question arises: who in this case will resolve the DNS query to chocolateand how?DNS name resolution is configured in the Kubernetes cluster using CoreDNS . Kubelet registers pods with CoreDNS as the name server in the files of /etc/resolv.confall pods. If you look at the contents of /etc/resolv.confany pod, it will look something like this:search hello.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.152.183.10
options ndots:5
This configuration is used by DNS clients to redirect requests to a DNS server. The file resolv.confcontains the following information:- nameserver : the server to which DNS queries will be routed. In our case, this is the address of the CoreDNS service;
- search: . ,
google.com mrkaran.dev FQDN ( ). , resolver' DNS, (FDQN) , «.», . resolver' . , mrkaran.dev. — (FQDN), mrkaran.dev — ; - ndots: ( ).
ndots , «» . , DNS-.
Let's see what happens when we request mrkaran.devin pod:$ nslookup mrkaran.dev
Server: 10.152.183.10
Address: 10.152.183.10
Non-authoritative answer:
Name: mrkaran.dev
Address: 157.230.35.153
Name: mrkaran.dev
Address: 2400:6180:0:d1::519:6001
For this experiment, I set the CoreDNS logging level to all(which makes it very verbose). Let's look at the pod’s logs coredns:[INFO] 10.1.28.1:35998 - 11131 "A IN mrkaran.dev.hello.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000263728s
[INFO] 10.1.28.1:34040 - 36853 "A IN mrkaran.dev.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000214201s
[INFO] 10.1.28.1:33468 - 29482 "A IN mrkaran.dev.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000156107s
[INFO] 10.1.28.1:58471 - 45814 "A IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 56 0.110263459s
[INFO] 10.1.28.1:54800 - 2463 "AAAA IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 68 0.145091744s
Fuh. Two things get attention here:- The request goes through all stages of the search until the response contains a code
NOERROR(DNS clients understand it and store it as a result). NXDOMAINmeans that no record was found for this domain name. Since mrkaran.devit is not an FQDN name (according to ndots=5), the resolver looks at the search path and determines the order of the queries; /etc/resolv.conf , IPv4 IPv6. , single-request resolv.conf.
: glibc , musl — , Alpine .ndots
Let's experiment with a little more ndotsand see how this parameter behaves. The idea is simple: it ndotsdetermines whether the DNS client considers the domain absolute or relative. For example, how, in the case of simple google, does the DNS client know if this domain is absolute? If set ndotsto 1, the client will say: “Oh, googlethere is not a single point; I’ll probably go over the entire search list. ” However, if requested google.com, the list of suffixes will be completely ignored, because the requested name satisfies the threshold ndots(there is at least one dot).Let's make sure of this:$ cat /etc/resolv.conf
options ndots:1
$ nslookup mrkaran
Server: 10.152.183.10
Address: 10.152.183.10
** server can't find mrkaran: NXDOMAIN
CoreDNS Logs:[INFO] 10.1.28.1:52495 - 2606 "A IN mrkaran.hello.svc.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000524939s
[INFO] 10.1.28.1:59287 - 57522 "A IN mrkaran.svc.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000368277s
[INFO] 10.1.28.1:53086 - 4863 "A IN mrkaran.cluster.local. udp 39 false 512" NXDOMAIN qr,aa,rd 132 0.000355344s
[INFO] 10.1.28.1:56863 - 41678 "A IN mrkaran. udp 25 false 512" NXDOMAIN qr,rd,ra 100 0.034629206s
Since there is mrkarannot a single point, the search was carried out throughout the list of suffixes.Note: in practice, the maximum value is ndotslimited to 15; Kubernetes defaults to 5.Production application
If an application makes many external network calls, DNS can become a bottleneck in case of active traffic, because when resolving a name, a lot of unnecessary queries are performed (before the system gets to the right one). Applications usually do not add a root zone to domain names, however this is quite a hack. That is, instead of asking api.twitter.com, you can hardcode api.twitter.com.(with a dot) in the application, which will prompt DNS clients to perform authoritative searches immediately in the absolute domain.Furthermore, starting with version Kubernetes 1.14, extension dnsConfigand dnsPolicygranted the status of a stable. Thus, when deploying a pod, you can decrease the valuendots, say, up to 3 (and even up to 1!). Because of this, each message within the node will have to include a complete domain. This is one of the classic compromises when you have to choose between performance and portability. It seems to me that worrying about this is only necessary if ultra-low latencies are vital for your application, since DNS results are also cached internally.References
For the first time I learned about this feature on the K8s-meetup on January 25th. There was a discussion there, including about this problem.Here are some links for further study:Note: I chose not to use digthis article. digautomatically adds a point (root zone identifier), making the domain “full” (FQDN), without first running it through the search list. He wrote about this in one of the previous publications . However, it is rather surprising that, in general, a standard flag has to be set for standard behavior.Good DNS'ing! See you later!PS from the translator
Read also in our blog: