Get best of the week

How to limit the frequency of requests in HAProxy: step-by-step instructions


The author explains how to implement in HAProxy query speed limit (rate limiting) with a certain IP-addresses. The Mail.ru Cloud Solutions team translated his article - we hope that with it you will not have to spend as much time and effort on it as you had to spend it.

The fact is that this is one of the most popular methods of protecting a server from DoS attacks, but it is difficult to find clear instructions on the Internet how to configure it specifically. By trial and error, the author forced HAProxy to limit the frequency of requests on a list of IP addresses, which is updated in real time.

No prior knowledge is required to configure HAProxy, as all necessary steps are outlined below.

Open source and free HAProxy is a highly accessible load balancer and proxy server. In recent years, it has become very popular because it provides high performance with a minimum of resources. Unlike alternative programs, the nonprofit version of HAProxy Community Edition offers enough features for reliable load balancing.

This program is at first pretty hard to figure out. However, she has very meticulous and detailed technical documentation . The author says that this is the most detailed documentation among all open source programs that he has ever used.
So, here is a step-by-step guide.

Configuring a load balancer


To save time and not get distracted by infrastructure setup, take the Docker and Docker Compose images and quickly launch the main components.

The first task is to raise the working instance of the HAProxy load balancer with several Apache backend servers.

Clone the repository


$ git clone git@github.com:stargazer/haproxy-ratelimiter.git
$ cd haproxy-ratelimiter

You can look at Dockerfileand docker-compose.ymlwith installation parameters. Their discussion is beyond the scope of this article, so let’s dwell on the fact that they created a working HAProxy instance called loadbalancerwith two backend servers api01and api02. To configure HAProxy, we will initially use the file haproxy-basic.cfg, and then switch to haproxy-ratelimiting.cfg.

For simplicity, the configuration file has been haproxy-basic.cfgreduced to the bare minimum and cleared of excess. Let's look at it:

defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

frontend proxy
bind *:80

use_backend api

backend api
balance roundrobin

server api01 api01:80
server api02 api02:80

The section frontend proxysets HAProxy to listen on port 80 and forward all requests to the server pool apion the backend.

CATEGORY backend apispecifies backend pool apiwith two back-end servers api01and api02and corresponding addresses. The server for servicing each incoming request is selected by the load balancing algorithm roundrobin, that is, in fact, the two available servers are used in turn.

Let's launch all three of our containers


$ sudo docker-compose up

Now we have a container loadbalancerthat redirects requests to two backend api01and servers api02. We will get a response from one of them if we enter in the address bar http://localhost/.

It is interesting to refresh the page several times and see the logs docker-compose.

api01_1 | 192.168.192.3 - - [08 / Jan / 2019: 11: 38: 09 +0000] "GET / HTTP / 1.1" 200 45
api02_1 | 192.168.192.3 - - [08 / Jan / 2019: 11: 38: 10 +0000] "GET / HTTP / 1.1" 304 -
api01_1 | 192.168.192.3 - - [08 / Jan / 2019: 11: 38: 10 +0000] "GET / HTTP / 1.1" 304 -
api02_1 | 192.168.192.3 - - [08 / Jan / 2019: 11: 38: 11 +0000] "GET / HTTP / 1.1" 304 -
api01_1 | 192.168.192.3 - - [08 / Jan / 2019: 11: 38: 11 +0000] "GET / HTTP / 1.1" 304 -
api02_1 | 192.168.192.3 - - [08 / Jan / 2019: 11: 38: 11 +0000] "GET / HTTP / 1.1" 304 -

As you can see, two servers apiprocess requests in turn.

We now have a HAProxy instance with a very simple load balancing configuration, and we have some idea of ​​how it works.

Add a limit on the number of requests


To set a limit on the number of requests to the load balancer, you need to modify the configuration file in the HAProxy instance. Make sure that the container loadbalanceruses the configuration file haproxy-ratelimiter.cfg.

Just modify the Dockerfile to replace the configuration file.

FROM haproxy:1.7
COPY haproxy-ratelimiter.cfg /usr/local/etc/haproxy/haproxy.cfg

Setting limits


All settings are registered in the configuration file haproxy-ratelimiter.cfg. Let's study it carefully.

defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

frontend proxy
bind *:80

# ACL function declarations
acl is_abuse src_http_req_rate(Abuse) ge 10
acl inc_abuse_cnt src_inc_gpc0(Abuse) gt 0
acl abuse_cnt src_get_gpc0(Abuse) gt 0

# Rules
tcp-request connection track-sc0 src table Abuse
tcp-request connection reject if abuse_cnt
http-request deny if abuse_cnt
http-request deny if is_abuse inc_abuse_cnt

use_backend api
backend api
balance roundrobin

server api01 api01:80
server api02 api02:80

backend Abuse
stick-table type ip size 100K expire 30m store gpc0,http_req_rate(10s)

HAProxy offers a set of low-level primitives that provide more flexibility and are suitable for various use cases. Its counters often remind me of the accumulative register (adder) in the CPU. They store intermediate results, take different semantics as input, but in the end they are just numbers. To understand everything well, it makes sense to start from the very end of the configuration file.

Table Abuse


backend Abuse
stick-table type ip size 100K expire 30m store gpc0,http_req_rate(10s)


Here we set up a dummy backend called Abuse(“abuse”). Fictitious, because it is used only for stick-table, which the rest of the configuration can refer to by name Abuse. A stick-table is a table stored in the process memory, where for each record you can determine the lifetime.

Our table has the following characteristics:

  • type ip: Requests are saved in the table by IP address as a key. Thus, requests from the same IP address will refer to the same record. Essentially, this means that we are tracking IP addresses and related data.
  • size 100K: The table contains a maximum of 100 thousand records.
  • expire 30m: The record retention period is 30 minutes of inactivity.
  • store gpc0,http_req_rate(10s): The counter gpc0and the number of IP address requests for the last 10 seconds are stored with entries . With the help, gpc0we will track how many times the IP address is noticed in abuses. In fact, a positive counter value means that the IP address is already marked as suspicious. Let's call this counter abuse indicator.

In general, the table Abuseallows you to track whether the IP address is marked as suspicious, as well as the current frequency of requests from this address. Therefore, we have a history of records, as well as real-time information.

Now let's move on to the section frontend proxyand see what's new there.

ACL Functions and Rules


frontend proxy
bind *:80

# ACL function declarations
acl is_abuse src_http_req_rate(Abuse) ge 10
acl inc_abuse_cnt src_inc_gpc0(Abuse) gt 0
acl abuse_cnt src_get_gpc0(Abuse) gt 0

# Rules
tcp-request connection track-sc0 src table Abuse
tcp-request connection reject if abuse_cnt
http-request deny if abuse_cnt
http-request deny if is_abuse inc_abuse_cnt

use_backend api

Access Control Lists (ACLs) are function declarations that are called only when the rule is set.

Let's take a closer look at all three ACL entries. Keep in mind that they all explicitly reference a table Abusethat uses IP addresses as a key, so each function applies to the request IP address:

  • acl is_abuse src_http_req_rate(Abuse) ge 10: The function is_abusereturns Trueif the current request frequency is greater than or equal to ten.
  • acl inc_abuse_cnt src_inc_gpc0(Abuse) gt 0: The function inc_abuse_cntreturns Trueif the increment value is gpc0greater than zero. Since the initial value gpc0is zero, this function always returns True. In other words, it increases the value abuse indicator, essentially signaling abuse from this IP address.
  • acl abuse_cnt src_get_gpc0(Abuse) gt 0: The function abuse_cntreturns Trueif the value is gpc0greater than zero. In other words, he says if this IP address has already been spotted in abuses.

As mentioned earlier, ACLs are simply declarations, that is, function declarations. They do not apply to incoming requests until some rule is triggered.

It makes sense to look at the rules defined in the same section frontend. The rules are applied to each incoming request one by one - and run the functions from the ACL that we just defined.

Let's see what each rule does:

  • tcp-request connection track-sc0 src table Abuse: Adds a query to the table Abuse. Since the key is the IP address in the table, this rule adds to the list of IP addresses.
  • tcp-request connection reject if abuse_cnt: TCP-, IP- , abuse. , TCP- IP-.
  • http-request deny if abuse_cnt: , IP- . IP-, abuse.
  • http-request deny if is_abuse inc_abuse_cnt: , is_abuse inc_abuse_cnt True. , , IP- , IP- .

In essence, we introduce two types of checks: in real time and in the blacklist from the query history. The second rule rejects all new TCP connections if the IP address has been noticed in abuses. The third rule prohibits the service of HTTP requests for an IP address from the black list, regardless of the current frequency of requests from this address. The fourth rule ensures that HTTP requests from an IP address are rejected at the very moment as soon as the threshold for request frequency has been overcome. Thus, the second rule mainly works on new TCP connections, the third and fourth - on already established connections, the first being a historical check, and the second a real-time check.

Let's try the filter in action!


Now we can assemble and launch our containers again.

$ sudo docker-compose down
$ sudo docker-compose build
$ sudo docker-compose up

The load balancer should start before the two backend servers.

Let's direct our browser to http://localhost/. If we quickly refresh the page a dozen times, we will exceed the threshold of ten requests in a ten-second interval - and our requests will be rejected. If we continue to refresh the page, new requests will be rejected immediately - even before the TCP connection is established.

Questions


Why is the limit of ten requests per ten seconds?


The table Abusedetermines http_req_rate(10s), that is, the frequency of requests is measured in a window of ten seconds. A function is_abusefrom the ACL returns Trueat a request rate of ≥10 within the specified interval. Thus, abuse is considered the frequency of requests of ten or more requests in ten seconds.

In this article, for example, we decided to set a low limit to make it easier to check the operation of the limiter.

What is the difference between http-request and tcp-request connection rules?


From the documentation :

http-request: the http-request statement defines a set of rules that apply at the network layer 7 [OSI model]

From the documentation :
tcp-request connection: performing an action on an incoming connection depending on a condition at the network layer 4

HTTP-, TCP-?


Imagine that HTTP requests to the server send multiple TCP connections from the same IP address. The frequency of HTTP requests will quickly exceed thresholds. It is then that the fourth rule comes into effect, which discards requests and blacklisted the IP address.

Now it is entirely possible that HTTP connections from the same IP address remain open (see persistent HTTP connection ), and the frequency of HTTP requests has dropped below a threshold value. The third rule guarantees continued blocking of HTTP requests, since it is abuse indicatortriggered on this IP.

Now suppose that after a few minutes the same IP tries to establish TCP connections. They are dropped immediately, because the second rule applies: it sees the labeled IP address and immediately drops the connection.

Conclusion


At first, it can be difficult to understand the limitation of the speed of processing requests using HAProxy. To do everything right, you need a fairly "low-level" thinking and a number of non-intuitive actions. The documentation in this part is probably too technical and suffers from the absence of any basic examples. We hope that this guide will make up for the shortcoming and show the direction to everyone who wants to take this path.

What else to read :

  1. How fault-tolerant architecture is implemented in the Mail.ru Cloud Solutions platform .
  2. Top 10 Kubernetes Tricks and Tips .
  3. Our Telegram channel on digital transformation .

More articles:


All Articles