👫 😯 ⚙️ 用于过滤浏览器流量的本地代理 💪🏾 🦖 👨🏻‍💻

我想谈谈开发过程并共享代理服务器，该服务器用于过滤各种垃圾和其他需要查看或干预浏览器流量的任务。也许某处已经有类似的功能，但是我想专门针对我的需要进行制作，并可以将代码实时地立即添加到任何需要的琐事上。好吧，在没有什么大不了的情况下，但是时间更长。

最初，任务是通过缓慢的连接（约5-10kb / s滞后）简化访问站点。主要有两个方面：1）删去所有不必要的内容（主要是广告），2）缓存所有可以缓存的内容，而不会对所访问的网站的功能造成很大的损害，即使这些网站本身不允许在http标头中甚至是在缓存时也是如此。它们显然会在静态文件的网址之后添加一个带有随机数的问号，从而产生干扰。

警告：下面描述的实现是针对Linux的，它似乎可以在其他* nix上运行，但是甚至没有考虑在Windows上进行编译（尽管有机会进行调整）。

草稿：nginx + php-fpm

我不想花所有的时间，因此决定快速配置nginx + php-fpm捆绑包，该捆绑包中的第一个解析传入连接（包括https）并将它们全部重定向，而无需分析主机和URL，并将其重定向到同一php脚本。还有一个小的C程序，将http-proxy协议转换为常规的HTTP（s）流量。也就是说，它将请求GET http://host/path HTTP/1.x转换为GET /path HTTP/1.x（主机名仍在Host标头中），并将所有https CONNECT代理到本地nginx。后来发现，http://host没有必要从http请求中删除，nginx以与通常相同的方式接受它们。

然后，PHP脚本收集回分解成不同变量的请求，并通过fsockopen（）连接到目标服务器（内置SSL支持），发送收集的http请求，获取响应，然后使用标头（）发送给浏览器。回声。该脚本不是从头开始编写的，这要归功于某个velen Lachance的https://github.com/eslachance/php-transparent-proxy（但是，引用的目的稍有不同，因此根本无法复制它）。

有两个问题

-, URL , POST- , Content-Type — php- , $_POST[] , , ( ). php 5.4 enable_post_data_reading — off, 'php://input'. $_POST $_FILES, , , .

-, nginx , " " firefox . , . , . ( ), , .

. , host host/path ( php- ). , ( — ). , . GET 200, — . — , , ( , ) content-type . : http-, content-type . , , , , , .

, , , .

( - , — ).

3-4 - — nginx php C.

( ), -, . https, - ( ), SSL - php .

OpenSSL. , , . , -, ( ), - ( ), - API OpenSSL ( ), API, ( ) ssl- ( , , ).

OpenSSL SSL

, - API TLS, , , (, , , , ), ASN1. , API , RFC . .

SSL/TLS-

, , SSL/TLS, , , . : , , , . .

, (accept) , SSL/TLS, ( SSL/TLS-, ), , . http- https-, , .

, : , , .

ssl-server-wrapper ssl-client-wrapper. , — , , SSL- , .

$ host ya.ru                        -  ip-
ya.ru has address 87.250.250.242
$ ./ssl-client-wrapper
T/etc/ssl/certs/ca-certificates.crt    -      RootCA
C87.250.250.242:443/ya.ru              -       SNI

( , ):

DEBUG: cert[-1] subject = /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: cert[-1] issuer = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[0] subject = /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: cert[0] issuer = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[1] subject = /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
DEBUG: cert[1] issuer = /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
DEBUG: cert[2] subject = /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
DEBUG: cert[2] issuer = /C=PL/O=Unizeto Sp. z o.o./CN=Certum CA

trusted

DEBUG: cert[-1] issued by cert[1]
DEBUG: cert[1] issued by cert[2]
DEBUG: cert[2]=cert[*0] was not issued by something from trusted-cert-store - chain incomplete, trying to recover
DEBUG: cache miss at ./cert-pin/cache/9nqv0qm41tlxyel95bbods0famxjagbx.0
DEBUG: downloading AIA issuer cert: http://repository.certum.pl/ca.cer
DEBUG: cert[*0] issued by cert[*1]
DEBUG: cert[*1] is self-signed, chain ended

, "Certum CA" Root CA, , (cert[2]) http://repository.certum.pl/ca.cer ; (cert[*1]), . , Root CA "Certum Trusted Network CA", .

DEBUG: check wildcard '*.yandex.az' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.tm' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.com.ua' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.de' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.jobs' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.net' against needed name 'ya.ru'
DEBUG: check wildcard '*.xn--d1acpjx3f.xn--p1ai' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.com.ge' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.fr' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.fr' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.kz' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.aero' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.jobs' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.ee' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.com' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.tm' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.ru' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.ru' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.lv' against needed name 'ya.ru'
DEBUG: check wildcard '*.yandex.lt' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.az' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.net' against needed name 'ya.ru'
DEBUG: check wildcard 'yandex.lt' against needed name 'ya.ru'
DEBUG: check wildcard 'ya.ru' against needed name 'ya.ru'

DEBUG: protocol = TLSv1.2
DEBUG: cipher = ECDHE-RSA-AES128-GCM-SHA256
DEBUG: verify_result = 0
DEBUG: server_cert subject: /C=RU/O=Yandex LLC/OU=ITO/L=Moscow/ST=Russian Federation/CN=*.yandex.az
DEBUG: server_cert issuer: /C=RU/O=Yandex LLC/OU=Yandex Certification Authority/CN=Yandex CA
OK: connection to 87.250.250.242:443/ya.ru established

, http-

GET / HTTP/1.0
Host: ya.ru
Connection: close

HTTP/1.1 200 Ok
Accept-CH: Viewport-Width, DPR, Device-Memory, RTT, Downlink, ECT
Accept-CH-Lifetime: 31536000
(...)

, .

nginx , . , \r\n \n , , , Content-Length, . Chunked encoding — , http-, .

, , — , php, . ( , , ..), php- , , "" ( C PHP), " -" . ( ) "" "" include ( ).

, 2 . — - ( ), , .

SSL/TLS-, ssl-telnet. - ( -), . , . OpenSSL ( , - ). .
- : ( ), ( ), (, , — ). ( , ). , , .
- , , trusted root CA .
, , ssh port forwarding ( helpers/remote.c proxy.c "--proxy". . SSL/TLS , "" . , .
: , , , , raw- ( real-time ).
/etc/hosts : - , , .
, , , . ( , ), . " , — ", . , , , " - ".
- — , , . - ( offline.flag , ). , , , ( POST- ), , , — .
, . , handle/inc.rewrite.c
dashboard ( ) .
./dashboard --basedir=/path/to/proxy/base --highlight --loop
. , , , , , . , - — , " ".

(+)
(-) - OpenSSL
(+)
(+-) #define
(+-) IPv6
(-)
(-)
(-) Access-Control-Allow-Origin '*' — .
(-) ,
(-) ( raw ) — , localhost- , , - ( ) 10

/

C-, zlib ( gzip http-), openssl ( ssl-). Debian zlib1g-dev libssl-dev. () .

, ( ) (, Makefile, make) build-all.sh.

: fproxy/build2.sh, fproxy/config.h ( /etc/ssl/certs/ca-certificates.crt ). (helpers/) #define (config.h ), .

, , () . , ( ), — , ( #define-). build-all.sh .

. prepare-dir.sh — fproxy-target/ ( ).

	:
cache/	cache_real/.
cache_real/	, — 2 , .
saved/	saved_real/.
saved_real/	-, 2-. .
cert-pin/certs/	( , / , ).
cert-pin/queue/	, .
cert-pin/cache/	, ( , ).
dyn-certs/	.
hist/	( , ).
internal/	AIA .
log/	.
pem/	.
proxy_temp/	.
tmp/	.
dyn-cert-serial	( — '0' ).
mincache.date	0, .
offline.flag	0, "-" — saved_real/ , , — .
rules_*.txt	, .

— ( fproxy-target/, prepare-dir.sh).

-. , , , . " " , root , 0640 0750. world-readable , , - , , MITM.

dashboard BASE/log/dashboard .

127.0.0.10:3128. 127.0.0.10 ( ) , - .

— , , .

: .

, , "-", . , .

— (, , , ) , , .

, - .

用于过滤浏览器流量的本地代理

草稿：nginx + php-fpm

SSL/TLS-

/

More articles: