Get best of the week

Understanding Network Policies with Calico



Calico Network Plugin provides a wide range of network policies with a unified syntax for protecting hosts on hardware, virtual machines and pods. These policies can be applied within the namespace or be global network policies applicable to the host endpoint (to protect applications running directly on the host - the server or virtual machine can be the host directly) or to workload endpoint(to protect applications running in containers or virtual machines hosted). Calico policies allow you to apply security measures to various points in the package path using parameters such as preDNAT, untracked, and applyOnForward. Understanding how these options work can help improve the security and performance of the system as a whole. This article explains the essence of these Calico policy settings (preDNAT, unraracked, and applyOnForward) applied to host endpoints, with an emphasis on what happens in packet processing paths (iptabels chains).

This article assumes that you have an understanding of how Kubernetes and Calico network policies work. If not, we recommend that you try the basic network policy tutorial and host protection tutorial using Calico before reading this article. We also expect you to have a basic understanding of iptables on Linux.

Calico global network policyallows you to apply a set of labels access rules (to host groups and workloads / pods). This is very useful if you use heterogeneous systems together - virtual machines, a system directly on the hardware, or the kubernetes infrastructure. In addition, you can protect your cluster (nodes) using a set of declarative policies and apply network policies to incoming traffic (for example, through the NodePorts service or External IPs).

At a fundamental level, when Calico connects a pod to a network (see diagram below), it connects it to a host using a virtual Ethernet (veth) interface. Traffic sent by pod arrives at the host from this virtual interface and is processed as if it came from a physical network interface. By default, Calico calls these interfaces caliXXX. Since the traffic comes through the virtual interface, it goes through iptables, as if the pod was at the distance of one hop. Therefore, when traffic comes / comes from the pod, it is forwarded from the point of view of the host.

On the Kubernetes node where Calico is running, you can map the virtual interface (veth) to workload as follows. In the example below, you can see that veth # 10 (calic1cbf1ca0f8) is connected to cnx-manager- * in the calico-monitoring namespace.

[centos@ip-172-31-31-46 K8S]$ sudo ip a
...
10: calic1cbf1ca0f8@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
...

[centos@ip-172-31-31-46 K8S]$ calicoctl get wep --all-namespaces
...
calico-monitoring cnx-manager-8f778bd66-lz45m                            ip-172-31-31-46.ec2.internal 192.168.103.134/32
calic1cbf1ca0f8
...



Given that Calico creates a veth interface for each workload, how does it apply policies? To do this, Calico creates hooks in various chains of the package processing path using iptables.

The diagram below shows the chains involved in processing packets in iptables (or the netfilter subsystem). When a packet arrives through a network interface, it first goes through the PREROUTING chain. Then a routing decision is made, and based on this, the packet passes through either INPUT (directed to host processes) or FORWARD (directed to pod or another node on the network). From the local process, the packet goes through the OUTPUT chain and then POSTROUTING before sending it over the cable.

Note that pod is also an external object (connected to veth) in terms of iptables handling. To summarize:

  • Forwarded traffic (nat, routed to / from pod) passes through the PREROUTING - FORWARD - POSTROUTING chains.
  • Traffic to the local host process goes through the PREROUTING - INPUT chain.
  • Traffic from the local host process goes through the OUTPUT - POSTROUTING chain.



Calico provides policy options for applying policies to all chains. With that in mind, let's look at the various policy settings available in Calico. The numbers in the options list below correspond to the numbers in the diagram above.

  1. Workload endpoint (pod) policy
  2. Host endpoint policy
  3. ApplyOnForward Option
  4. PreDNAT Policy
  5. Untracked Policy

Let's start by looking at how policies apply to workload endpoints (Kubernetes or OpenStack VMs pods), and then look at the policy options for host endpoints.

Workload endpoints


Workload Endpoint Policy (1)


This is an option to protect your kubernetes pods. Calico supports working with Kubernetes NetworkPolicy, but it also provides additional policies - Calico NetworkPolicy and GlobalNetworkPolicy. Calico creates a chain for each pod (workload) and hooks to the INPUT and OUTPUT chains for workload to the filter table of the FORWARD chain.

Host endpoints


Host Endpoint Policy (2)


In addition to CNI (container network interface), Calico policies provide the ability to directly protect the host. In Calico, you can create a host endpoint by specifying a combination of the host interface and, if necessary, port numbers. The application of policies for this entity is achieved using the filter table in the INPUT and OUTPUT chains. As can be seen from the diagram, (2) they are applied to local processes on the node / host. That is, if you created a policy that applies to the host endpoint, it will not affect the traffic going to / from your pods. But it provides a single interface / syntax for blocking traffic for your host and pods using Calico policies. This greatly simplifies the process of managing policies for a heterogeneous network. Configuring host endpoint policies to enhance cluster protection is another important use case.

ApplyOnForward Policy (3)


The ApplyOnForward option is available in the Calico global network policy to enable policies to apply to all traffic passing through the host endpoint, including traffic that will be forwarded by the host. This traffic includes being sent to a local pod or anywhere else on the network. Calico requires this setting to be enabled for policies that use PreDNAT and untracked, see the following sections. In addition, ApplyOnForward can be used to track host traffic when using a virtual router or software NAT.

Note that if you need to apply the same network policy for both host processes and pods, then you do not have to use the ApplyOnForward option. You just need to create a label for the desired hostendpoint and workload endpoint (pod). Calico is smart enough to apply labels-based policies, regardless of the type of endpoint (hostendpoint or workload).

PreDNAT Policy (4)


In Kubernetes, the ports of the service entity can be forwarded outward using the NodePorts option or, optionally (when using Calico), by declaring them through the Cluster IPs or External IPs options. Kube-proxy balances the incoming traffic bound to the service to the pods of the corresponding service using DNAT. Given this, how do you apply policies to traffic coming through NodePorts? In order for these policies to be applied before traffic is processed by DNAT (which is a mapping between host: port and corresponding service), Calico provides a parameter for globalNetworkPolicy called "preDNAT: true".

When pre-DNAT is enabled, these policies are implemented in (4) in the diagram — in the mangle table of the PREROUTING chain — immediately before DNAT. The usual order policy is not respected here, since the application of these policies occurs much earlier along the path of processing traffic. However, preDNAT policies respect the order of application between themselves.

When creating policies with pre-DNAT, it is important to be mindful of the traffic that you want to process and allow most to be rejected. Traffic marked as 'allow' in the pre-DNAT policy will no longer be checked by the hostendpoint policy, while traffic when the pre-DNAT policy fails will continue to flow through the rest of the chains.
Calico made it mandatory to enable the applyOnForward option when using preDNAT, because by definition the destination of the traffic has not yet been selected. Traffic can be directed to the host process, or it can be redirected to a pod or to another node.

Untracked Policy (5)


Networks and applications can have large differences in behavior. In some extreme cases, applications can generate many short-term connections. This can lead to a lack of memory for conntrack (the main component of the Linux network stack). Traditionally, to run this type of application on Linux, you need to manually configure or disable conntrack, or write iptables rules to bypass conntrack. Untracked policy in Calico is a simpler and more efficient option if you want to process connections as quickly as possible. For example, if you use massive memcache or as an additional measure of protection against DDOS .

Read this blog post (or our translation) for more information, including performance tests using the untracked policy.

When you set the “doNotTrack: true” option to Calico globalNetworkPolicy, it becomes a ** untracked ** policy and is applied at the very early stage of the Linux package processing pipeline. If you look at the diagram above, untracked policies are applied in the PREROUTING and OUTPUT chains in the raw table before connection tracking (conntrack) is started. When a package is allowed by an untracked policy, it is flagged to disable connection tracking for this package. It means:

  • An untracked policy is applied for each package. There is no concept of a connection (or stream). Lack of connections entails several important consequences:
  • , , , ( Calico conntrack, ).
  • untracked workload Kubernetes (pod’), pod’.
  • NAT ( ​​ NAT conntrack).
  • « » untracked- . , , untracked- ( ).
  • Untracked policies are applied at the very beginning of the packet processing pipeline. This is very important to understand when creating Calico policies. You can have a policy for a pod with order: 1 and an untracked policy with order: 1000. It will not matter. The untracked policy will be applied before the policy for the pod. Untracked policies respect the order of execution only among themselves.

Because one of the goals of the doNotTrack policy is to enforce the policy at the very early stage of the Linux package processing pipeline, Calico makes it mandatory to specify the applyOnForward option when using doNotTrack. Referring to the packet processing diagram, note that the untracked (5) policy is applied before any routing decisions. Traffic can be directed to the host process, or it can be redirected to a pod or to another node.

Summary


We looked at the various policy options (Host endpoint, ApplyOnForward, preDNAT, and Untracked) in Calico and how they apply to packet processing. Understanding the essence of their work helps in developing effective and secure policies. With Calico, you can use the global network policy, which applies to the label (a group of nodes and pods) and apply policies with various parameters. This allows security and network design specialists to conveniently protect immediately “everything” (endpoints types) using the same policy language with Calico policies.


Acknowledgment: I would like to thank Sean Crampton and Alex Pollitt for their review and for valuable information.

More articles:


All Articles