In this article I will talk about the history of development and the current state of technology for accelerating the distribution of content in TLS connections by transferring encryption to the core of the operating system, as well as about my contribution to the development of this direction.Background
Back in 2015, Randall Stewart and Scott Long from Netflix made a presentation at the AsiaBSDCon2015 conference on optimizing the distribution of encrypted content. The main message of the report is to transfer data encryption to the kernel of the operating system to reduce the number of data copies between kernel-space and user-space and use the optimized non-blocking sendfile () system call. The topic turned out to be very promising and already at the 2016 Netdev 1.2 conference, Dave Watson from Facebook made a presentation on the need to create TLS sockets in the Linux kernel, and Boris Pismenny, Ilya Lesokhin and Liran Liss from Mellanox made a presentationabout the ability to use TLS hardware acceleration in network cards. The topic was also discussed at HighLoad ++ 2017 by a speaker from Tempesta Technologies.Linux kernel support
The result of all these conversations was the appearance of Kernel TLS in the Linux 4.13 kernel (2017) with support for TLSv1.2 and the AES128-GCM cipher. Initially, encryption of only outgoing traffic was supported, decryption support appeared later in the Linux kernel 4.17 (2018). In version 5.1, they added support for TLSv1.3 and AES256-GCM, and in version 5.2 they also added the AES128-CCM cipher (2019).Support in user-space
In all the reports mentioned above, it was said that only encryption of useful data can be placed in the kernel, and all TLS approvals and control messages must be handled the same in user-space. And for this, a modified version of OpenSSL was used. However, at the time of the publication of the reports in the public domain there was no information about which modifications to this well-known library needed to be done to support the functionality. Perhaps the only available example of using Linux Kernel TLS was the blog article Filippo Valsorda Playing with kernel TLS in Linux 4.13 and Go, which appeared immediately after the release of the Linux 4.13 kernel. And, although it showed a valid example of the use of technology, it did not bring an understanding of how to use the technology in real projects. Indeed, very few people write a WEB server for their project on their own, usually everyone uses well-known and time-tested tools.Support in OpenSSL
The first discussions of the technology in OpenSSL appeared in the summer of 2017 shortly before the release of the Linux 4.13 kernel ( PR 3631 ), but the discussion process was very, very slow, the first real comments appeared in October (after the release of the 4.13 kernel), and the actual working versionstarted discussing in February 2018. By the time all corrections were agreed, the window for adding new functionality in OpenSSL 1.1.1 was closed, and support for Kernel TLS was moved to the next release. Since that time, Kernel TLS RX support has been added, and SSL_sendfile () is a call to the corresponding syscall with a little handling of possible situations in the TLS protocol. But now in the year 2020, OpenSSL 3.0 has not yet come out, TLSv1.3 is supported by the vast majority of browsers, and the AES128-GCM cipher is actively replaced by the more resistant AES256-GCM. So I took the liberty and sent a Pull Request to support new ciphers and TLSv1.3 in the hope that they would accept it before the new release of the library.Support in WEB-servers
But supporting Kernel TLS in the TLS library is not enough. Reports about the technology said that maximum efficiency can be obtained using sending without copying data to user-space - using the sendfile () system call. And the server application should be able to distinguish between situations where you can use sendfile () on a socket with TLS, and when you need to do “read the old way” read () / SSL_write (). Some progress towards adding functionality to Nginx appeared in April 2019, but no changes were accepted to the main code. The developers' position is that the API for these functions in OpenSSL has not yet been approved, and the actual code proposed in the patch does not seem to be sufficiently portable to different platforms. To be honest, the code not only does not look very nice, but it also contains errors that prevent Nginx from being built without additional corrections.I could not find Kernel TLS support in other web servers at all (maybe someone saw it - tell me in the comments).And what to do?
While the community is awaiting the release of OpenSSL 3.0, in order to start developing support for Kernel TLS in Nginx with a stable API, I went the other way. In my cozy corner of GitHub, I did 2 things:- I created an OpenSSL fork and backported everything related to Kernel TLS to the stable version of OpenSSL 1.1.1 (OpenSSL_1_1_1-ktls branch). Especially in order to be able to check the functionality in conditions of stable operation of the rest of the library. As stable versions become available, I try to rebase so that the fork is up to date.
- I created a fork of Nginx, in which I added (based on a patch from Mellanox) support for calling SSL_sendfile () in conditions when it is really possible and with the necessary SSL socket checks, and the ability to enable / disable the functionality through the configuration variable. In addition to this feature, in my fork there are also several patches that optimize the work of Nginx a bit and fix some bugs (master-feature branch). As far as possible, I try to rebase based on the master branch of the Nginx main repo, in order to keep the fork up to date.
I invite everyone to take part in testing. Comments and corrections to the code can be thrown at Issues on GitHub. To build this Nginx with this OpenSSL and the included Kernel TLS support, add parameters to the configure script:./configure --with-openssl=<OpenSSL-fork-dir> --with-openssl-opt="enable-ktls"
At the moment, I don’t have the opportunity to test this Nginx + OpenSSL bundle under heavy load to confirm the performance from the comment on the Nginx patch, so if suddenly someone can measure the difference in CPU load at high file upload speeds, it would be great to get graphs and add them into an article for a visual understanding of the effect.