🍭 🍨 💸 Proxy lokal untuk memfilter lalu lintas browser 🤽🏽 👨🏽‍🏭 🔂

Saya ingin berbicara tentang proses pengembangan dan berbagi server proxy, yang saya gunakan sendiri untuk memfilter semua jenis sampah dan tugas lain yang memerlukan tampilan atau intervensi dalam lalu lintas browser. Mungkin sudah ada fungsi serupa di suatu tempat, tetapi saya ingin melakukannya secara khusus untuk kebutuhan saya dengan kemungkinan menambahkan kode waktu-nyata ke hal-hal sepele yang sesaat. Nah, di bawah hal-hal kecil, juga, tapi lebih lama.

Awalnya, tugasnya adalah menyederhanakan situs yang dikunjungi melalui koneksi yang lambat (sekitar 5-10 kb / s dengan kelambatan). Ada dua bidang utama: 1) untuk memotong segala sesuatu yang tidak perlu (terutama iklan), dan 2) untuk men-cache semua yang dapat di-cache tanpa banyak merusak fungsi situs yang dikunjungi, bahkan ketika situs itu sendiri tidak mengizinkan caching di header http, atau bahkan mereka jelas mengganggu dengan menambahkan tanda tanya dengan nomor acak setelah url file statis.

Peringatan: implementasi yang dijelaskan di bawah ini dilakukan untuk Linux, tampaknya bekerja pada * nix lain, tetapi bahkan mengkompilasinya pada Windows tidak dipertimbangkan (walaupun tentu saja ada kesempatan untuk beradaptasi).

Konsep: nginx + php-fpm

Saya tidak ingin menghabiskan waktu ini, jadi diputuskan untuk dengan cepat mengkonfigurasi bundel nginx + php-fpm, di mana koneksi masuk yang diurai pertama kali, termasuk https dan dialihkan semuanya, tanpa menganalisis host dan URL, ke skrip php yang sama . Ada juga program C kecil yang mengkonversi protokol http-proxy ke lalu lintas http (s) biasa. Artinya, itu mengubah permintaan GET http://host/path HTTP/1.xmenjadi GET /path HTTP/1.x(nama host masih di header Host) dan memproksi semua https CONNECTs ke nginx lokal. Ternyata kemudian, http://hosttidak perlu menghapus dari permintaan http, nginx menerimanya dengan cara yang sama seperti yang biasa.

PHP-, , , fsockopen() ( SSL ), http-, header() echo. , Évelyne Lachance https://github.com/eslachance/php-transparent-proxy ( , , ).

-, URL , POST- , Content-Type — php- , $_POST[] , , ( ). php 5.4 enable_post_data_reading — off, 'php://input'. $_POST $_FILES, , , .

-, nginx , " " firefox . , . , . ( ), , .

. , host host/path ( php- ). , ( — ). , . GET 200, — . — , , ( , ) content-type . : http-, content-type . , , , , , .

, , , .

( - , — ).

3-4 - — nginx php C.

( ), -, . https, - ( ), SSL - php .

OpenSSL. , , . , -, ( ), - ( ), - API OpenSSL ( ), API, ( ) ssl- ( , , ).

OpenSSL SSL

, - API TLS, , , (, , , , ), ASN1. , API , RFC . .

SSL/TLS-

, , SSL/TLS, , , . : , , , . .

, (accept) , SSL/TLS, ( SSL/TLS-, ), , . http- https-, , .

, : , , .

ssl-server-wrapper ssl-client-wrapper. , — , , SSL- , .

$ host ya.ru                        -  ip-
ya.ru has address 87.250.250.242
$ ./ssl-client-wrapper
T/etc/ssl/certs/ca-certificates.crt    -      RootCA
C87.250.250.242:443/ya.ru              -       SNI

( , ):

DEBUG: cert[-1] subject = /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: cert[-1] issuer = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[0] subject = /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: cert[0] issuer = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[1] subject = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[1] issuer = /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
DEBUG: cert[2] subject = /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
DEBUG: cert[2] issuer = /C=PL/O=Unizeto Sp. z o.o./CN=Certum CA

trusted

DEBUG: cert[-1] issued by cert[1]
DEBUG: cert[1] issued by cert[2]
DEBUG: cert[2]=cert[*0] was not issued by something from trusted-cert-store - chain incomplete, trying to recover
DEBUG: cache miss at ./cert-pin/cache/9nqv0qm41tlxyel95bbods0famxjagbx.0
DEBUG: downloading AIA issuer cert: http://repository.certum.pl/ca.cer
DEBUG: cert[*0] issued by cert[*1]
DEBUG: cert[*1] is self-signed, chain ended

, "Certum CA" Root CA, , (cert[2]) http://repository.certum.pl/ca.cer ; (cert[*1]), . , Root CA "Certum Trusted Network CA", .

DEBUG: check wildcard '*.yandex.az' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.tm' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.com.ua' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.de' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.jobs' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.net' against needed name 'ya.ru'
DEBUG: check wildcard '*.xn--d1acpjx3f.xn--p1ai' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.com.ge' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.fr' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.fr' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.kz' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.aero' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.jobs' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.ee' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.com' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.tm' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.ru' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.ru' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.lv' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.lt' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.az' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.net' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.lt' against needed name 'ya.ru'
DEBUG: check wildcard 'ya.ru' against needed name 'ya.ru'

DEBUG: protocol = TLSv1.2
DEBUG: cipher = ECDHE-RSA-AES128-GCM-SHA256
DEBUG: verify_result = 0
DEBUG: server_cert subject: /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: server_cert issuer: /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
OK: connection to 87.250.250.242:443/ya.ru established

, http-

GET / HTTP/1.0
Host: ya.ru
Connection: close

HTTP/1.1 200 Ok
Accept-CH: Viewport-Width, DPR, Device-Memory, RTT, Downlink, ECT
Accept-CH-Lifetime: 31536000
(...)

, .

nginx , . , \r\n \n , , , Content-Length, . Chunked encoding — , http-, .

, , — , php, . ( , , ..), php- , , "" ( C PHP), " -" . ( ) "" "" include ( ).

, 2 . — - ( ), , .

SSL/TLS-, ssl-telnet. - ( -), . , . OpenSSL ( , - ). .
- : ( ), ( ), (, , — ). ( , ). , , .
- , , trusted root CA .
, , ssh port forwarding ( helpers/remote.c proxy.c "--proxy". . SSL/TLS , "" . , .
: , , , , raw- ( real-time ).
/etc/hosts : - , , .
, , , . ( , ), . " , — ", . , , , " - ".
- — , , . - ( offline.flag , ). , , , ( POST- ), , , — .
, . , handle/inc.rewrite.c
dashboard ( ) .
./dashboard --basedir=/path/to/proxy/base --highlight --loop
. , , , , , . , - — , " ".

(+)
(-) - OpenSSL
(+)
(+-) #define
(+-) IPv6
(-)
(-)
(-) Access-Control-Allow-Origin '*' — .
(-) ,
(-) ( raw ) — , localhost- , , - ( ) 10

/

C-, zlib ( gzip http-), openssl ( ssl-). Debian zlib1g-dev libssl-dev. () .

, ( ) (, Makefile, make) build-all.sh.

: fproxy/build2.sh, fproxy/config.h ( /etc/ssl/certs/ca-certificates.crt ). (helpers/) #define (config.h ), .

, , () . , ( ), — , ( #define-). build-all.sh .

. prepare-dir.sh — fproxy-target/ ( ).

	:
cache/	cache_real/.
cache_real/	, — 2 , .
saved/	saved_real/.
saved_real/	-, 2-. .
cert-pin/certs/	( , / , ).
cert-pin/queue/	, .
cert-pin/cache/	, ( , ).
dyn-certs/	.
hist/	( , ).
internal/	AIA .
log/	.
pem/	.
proxy_temp/	.
tmp/	.
dyn-cert-serial	( — '0' ).
mincache.date	0, .
offline.flag	0, "-" — saved_real/ , , — .
rules_*.txt	, .

— ( fproxy-target/, prepare-dir.sh).

-. , , , . " " , root , 0640 0750. world-readable , , - , , MITM.

dashboard BASE/log/dashboard .

127.0.0.10:3128. 127.0.0.10 ( ) , - .

— , , .

: .

, , "-", . , .

— (, , , ) , , .

, - .

Proxy lokal untuk memfilter lalu lintas browser

Konsep: nginx + php-fpm

SSL/TLS-

/

More articles: