I want to talk about the development process and share the proxy server, which I use to filter all kinds of garbage and other tasks that require viewing or intervention in browser traffic. Perhaps there is already a similar functionality somewhere, but I wanted to make it specifically for my needs with the possibility of real-time appending code to any trifle that is needed right away. Well, under the no-little things, too, but it's longer.
Initially, the task was to simplify visiting sites through a slow (about 5-10kb / s with lags) connection. There are two main areas: 1) to cut out everything that is not necessary (primarily advertising), and 2) to cache everything that can be cached without much harm to the functionality of the sites visited, even when the sites themselves do not allow caching in http headers, or even they obviously interfere by adding a question mark with a random number after urls of static files.
Warning: the implementation described below was made for Linux, it seems to work on other * nix, but even compiling it on Windows was not considered (although there is an opportunity to adapt, of course).
Draft: nginx + php-fpm
I did not want to spend all this time, so it was decided to quickly configure the nginx + php-fpm bundle, of which the first parsed incoming connections, including https and redirected them all, without analyzing hosts and URLs, to the same php script . There was also a small C program that converted the http-proxy protocol to regular http (s) traffic. That is, it turned requests GET http://host/path HTTP/1.xinto GET /path HTTP/1.x(the host name is still in the Host header) and proxied all https CONNECTs to the local nginx. As it turned out later, it http://hostwas not necessary to remove from http-requests, nginx accepts them in the same way as usual ones.
PHP-, , , fsockopen() ( SSL ), http-, header() echo. , Évelyne Lachance https://github.com/eslachance/php-transparent-proxy ( , , ).
-, URL , POST- , Content-Type — php- , $_POST[] , , ( ). php 5.4 enable_post_data_reading — off, 'php://input'. $_POST $_FILES, , , .
-, nginx , " " firefox . , . , . ( ), , .
. , host host/path ( php- ). , ( — ). , . GET 200, — . — , , ( , ) content-type . : http-, content-type . , , , , , .
, , , .
( - , — ).
3-4 - — nginx php C.
( ), -, . https, - ( ), SSL - php .
OpenSSL. , , . , -, ( ), - ( ), - API OpenSSL ( ), API, ( ) ssl- ( , , ).
OpenSSL SSL, - API TLS, , , (, , , , ), ASN1. , API , RFC . .
SSL/TLS-
, , SSL/TLS, , , . : , , , . .
, (accept) , SSL/TLS, ( SSL/TLS-, ), , . http- https-, , .
, : , , .
ssl-server-wrapper ssl-client-wrapper. , — , , SSL- , .
$ host ya.ru - ip-
ya.ru has address 87.250.250.242
$ ./ssl-client-wrapper
T/etc/ssl/certs/ca-certificates.crt - RootCA
C87.250.250.242:443/ya.ru - SNI
( , ):
DEBUG: cert[-1] subject = /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: cert[-1] issuer = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[0] subject = /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: cert[0] issuer = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[1] subject = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[1] issuer = /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
DEBUG: cert[2] subject = /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
DEBUG: cert[2] issuer = /C=PL/O=Unizeto Sp. z o.o./CN=Certum CA
trusted
DEBUG: cert[-1] issued by cert[1]
DEBUG: cert[1] issued by cert[2]
DEBUG: cert[2]=cert[*0] was not issued by something from trusted-cert-store - chain incomplete, trying to recover
DEBUG: cache miss at ./cert-pin/cache/9nqv0qm41tlxyel95bbods0famxjagbx.0
DEBUG: downloading AIA issuer cert: http://repository.certum.pl/ca.cer
DEBUG: cert[*0] issued by cert[*1]
DEBUG: cert[*1] is self-signed, chain ended
, "Certum CA" Root CA, , (cert[2]) http://repository.certum.pl/ca.cer ; (cert[*1]), . , Root CA "Certum Trusted Network CA", .
:
DEBUG: check wildcard '*.yandex.az' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.tm' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.com.ua' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.de' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.jobs' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.net' against needed name 'ya.ru'
DEBUG: check wildcard '*.xn--d1acpjx3f.xn--p1ai' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.com.ge' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.fr' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.fr' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.kz' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.aero' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.jobs' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.ee' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.com' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.tm' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.ru' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.ru' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.lv' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.lt' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.az' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.net' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.lt' against needed name 'ya.ru'
DEBUG: check wildcard 'ya.ru' against needed name 'ya.ru'
:
DEBUG: protocol = TLSv1.2
DEBUG: cipher = ECDHE-RSA-AES128-GCM-SHA256
DEBUG: verify_result = 0
DEBUG: server_cert subject: /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: server_cert issuer: /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
OK: connection to 87.250.250.242:443/ya.ru established
, http-
GET / HTTP/1.0
Host: ya.ru
Connection: close
HTTP/1.1 200 Ok
Accept-CH: Viewport-Width, DPR, Device-Memory, RTT, Downlink, ECT
Accept-CH-Lifetime: 31536000
(...)
, .
nginx , . , \r\n \n , , , Content-Length, . Chunked encoding — , http-, .
, , — , php, . ( , , ..), php- , , "" ( C PHP), " -" . ( ) "" "" include ( ).
, 2 . — - ( ), , .
SSL/TLS-, ssl-telnet. - ( -), . , . OpenSSL ( , - ). .
- : ( ), ( ), (, , — ). ( , ). , , .
- , , trusted root CA .
, , ssh port forwarding ( helpers/remote.c proxy.c "--proxy". . SSL/TLS , "" . , .
: , , , , raw- ( real-time ).
/etc/hosts : - , , .
, , , . ( , ), . " , — ", . , , , " - ".
- — , , . - ( offline.flag , ). , , , ( POST- ), , , — .
, . , handle/inc.rewrite.c
dashboard ( ) .
./dashboard --basedir=/path/to/proxy/base --highlight --loop
. , , , , , . , - — , " ".
(+)
(-) - OpenSSL
(+)
(+-) #define
(+-) IPv6
(-)
(-)
(-) Access-Control-Allow-Origin '*' — .
(-) ,
(-) ( raw ) — , localhost- , , - ( ) 10
/
C-, zlib ( gzip http-), openssl ( ssl-). Debian zlib1g-dev libssl-dev. () .
, ( ) (, Makefile, make) build-all.sh.
: fproxy/build2.sh, fproxy/config.h ( /etc/ssl/certs/ca-certificates.crt ). (helpers/) #define (config.h ), .
, , () . , ( ), — , ( #define-). build-all.sh .
.
. prepare-dir.sh — fproxy-target/ ( ).
— ( fproxy-target/, prepare-dir.sh).
-. , , , . " " , root , 0640 0750. world-readable , , - , , MITM.
dashboard BASE/log/dashboard .
127.0.0.10:3128. 127.0.0.10 ( ) , - .
— , , .
: .
, , "-", . , .
— (, , , ) , , .
, - .