Ich schlage vor, das Protokoll des Berichts von Andrei Borodin von Anfang 2020 "Odyssee: Architektur, Konfiguration, Ăśberwachung" kennenzulernen.
In jüngerer Zeit haben wir Version 1.0 unserer Abzieherverbindungen Odyssey veröffentlicht . Es soll die Probleme beim Verwalten von Verbindungen hoch geladener PostgreSQL-Installationen lösen. In diesem Bericht möchte ich über die Architektur und den Betrieb der Odyssee sprechen. Probleme, die in einem ziemlich langen Übergang zwischen 1.0rc und 1.0 behoben wurden, sind ebenfalls betroffen.
Hallo alle zusammen! Ich mache Postgres bei Yandex. Und in den letzten anderthalb Jahren habe ich Verbindungsmanagement betrieben. Ziemlich kritische Sache, weil sie jeder Anfrage im Wege steht. Auf Russisch sagen wir "Odyssee".
Connection pooler. . , . connection pooler . . – , . , connection pooler , . Connection pooler . , .
Postgres . , Postgres ., - Postrgres’. . . .
., Postgres. , , , 1 .
. , Postgres. , , Odyssey .
? , . Postgres , , – , .
, .
, Postgres . . - . , procArray.
- , , Application-side pooling.
- – Proxy pooler.
- -, , pooler.
- , .
Application-side pooling – , , .
, , instances , .
, . , , .
, , .
instance Postgres. , . , . , - – .
Application-side pooling . , ?
pooler, . .
Pgpool II. , . . . - , , .
Crunchy-Proxy. Crunchy Data Proxy pooler. Go, - .
PgBouncer. pooler , pooling. . , , .
- PgBouncer.
.
PgBouncer . PgBouncer , . . , - , , , : , , . , PgBouncer .
, , , , Bouncer .
, , PgBouncer . . CPU 97 % . , , Bouncer .
, . , PgBouncer PgBouncer. PgBouncer, . reuseport. .
PgBouncer, Bouncer, - .
PgBouncer, connection pool PgBouncer. , .
- PgBouncer, , c PgBouncer - .
PgBouncer TLS-. TLS- – , , . . . , , . TLS handshake . PgBouncer, , TLS , . , . . , , .
, , , Odyssey – , PgBouncer .
- , PgBouncer , . . pull request. . . , PgBouncer .
Odyssey PgBouncer . .
connection pooler, Odyssey.
Odyssey . Odyssey. Kiwi, Machinarium.
Kiwi Postgres. Postgres Proto3. Kiwi , , . libpq Postgres, , . connection pooler Postgres’ .
Machinarium – , .
Machinarium . – , , thread . . – , . , , .
, . Machinarium. – , , , . sleep. , csw_worker Mashinarium’, - 0 .
Machinarium - , .
, , PgBouncer. PgBouncer libevent, Machinarium – libevent, connection pooling, .
Odyssey . main-, . . main-, , . . TCP-, , . - , , , .
, router, wait_read sleep, , . -. . , . . . . , , , , , . , , . , .
, , , , PgBouncer PgBouncer. . . , show stats, show clients, show pulls .
Odyssey production instances Postgres. . , PgBouncer, , . PgBouncer .
, . Odyssey production- , Postgres profession Odyssey. , production , . . , , .
. Odyssey, Odyssey TLS-. TLS- . .
Odyssey , , . connection pooler , , TLS-, , connection pooler, , TLS-.
, , .
, , , 20 000-30 000 TLS-, pooler TLS- . TLS handshake . - TLS handshake, , 3 30 000 TLS handshake, .
accept , . . , -, , , handshake -.
, , , Postgres , GSSAPI.
, Postgres , -, TLS-, GSSAPI.
PSQL GSSAPI. PgBouncer, Odyssey 12 Postgres . , , . .
Pooler , GSSAPI, , . TLS-.
Odyssey . 1.0 storages, , storage . . , . , walsender, : « ?». : «, basebackup» «, , walsender ».
, break inEnhaced. 1.1, , , , , . . . , . , .
– connection pooler. Connection pooler . . pooler’ , 1 000 .
Odyssey , , 1 000 , 1 000 , . , . PgBouncer , , OLTP-. pooler , .
, , . , , , , 4- . , PgBouncer, .
, – ML. PgBouncer, Odyssey – . , . .
pooler , . - . . ML-, . , .
, , . , , , .
, , , . , connection pooler, . , Python - . , , . .
PgBouncer , – , . TLS handshake, 15 , PgBouncer . . , 1.0. , , .
client_max_routing TLS handshakes. , TLS-.
, , .
Odyssey , , TLS-. listen.
Odyssey – , . , , .
Listen , .
Storage – . , remote, local. Local – , . . , PgBouncer PgBouncer.
, – database user, , . , , . .
, Postgres – client_fwd_error. «no», Bouncer. . . - , , Bouncer. «yes», , Postgres.
Pool_cancel pool_rollback – , , , , cancel .
Application_name_add_host – Bouncer, , ip- application_name .
, , quantiles quantiles .
? Bouncer . , . 10 % , , , - . 10 % .
– , . 0,99 , 99 % 1,273 . , - , - - , - cron-. . , , – .
Odyssey. PgBouncer.
Utilization – Odyssey. 4 , , Odyssey 4- . 400 %, , , .
Saturation Errors – , Odyssey, , . . - , .
Odyssey , online restart. PgBouncer PgBouncer – PgBouncer. , , Odyssey .
reload Odyssey , , Odyssey , . , reload , restart , .
Odyssey , , – IPv4 IPv6 - PgBouncer Odyssey PgBouncer , Odyssey IPv6. , , IPv4 , . , , . , IPv4 , , .
Odyssey Open Source . . . , PgBouncer source-port, , - , . - , .
, SCRAM .
merge pause resume. , . . , . - pooler. , . , 11 12, 13 Postgres.
! . . Odyssey? . . hba , Postgres pg_hba.conf . .? GitHub , , .
, .
ip, hostname . ., . . , .
iptables .
, iptables – , . Odyssey – Bouncer, , , pg_hba .
. Issue GitHub.
, .
-. , .
, !
! metrics endpoint, Prometheus .
, Prometheus. , . : , . , , . – , . . Prometheus - , ? .
, Prometheus - , , , . endpoints metrics , Prometheus’. , , .
?
Zabbix . , , , , Bouncer , . endpoint .
. . Odyssey http-?
-, . , Bouncer.
. .
?
– , . .
. - Prometheus, , , , ?
– Odyssey , - , , , . , , . - ? , .
. – , . , . . . - , , . , , . , .
, , «connection pooler». - . , . .
, .
, .
, . , , , , , , . , pooler, . , . ? 2 . pooler? , pooler .
.
.
! . , - Prometheus, . , , - , , Prometheus, Cloudwatch, , ?
Odyssey?
call Odyssey. , . , , , . , , .
show stats. ?
- , .
, exporter Prometheus.
exporter ?
Exporter, .
. . , Postgres Odyssey : « ?». Odyssey , . exporter - http . , Odyssey. Postgres CentOS . , . Exporter – . . , , C, - , , Go. . ?
! ? ?
? , . . open source. , . -, . GitHub. . , .
, ?
GCC Clang.
. ?
C99.
, - ? ? , ?
20 000 . 10 000 – . . . - , , , , , - .
? 500 MB , - , - . , , .
10 000 , ?
, , . . . 10 000 , – ? . . TLS handshake – 100 . 100 , 10 . . 10 , 100 . , - . 100 – . 10 . TLS handshake – .
Odyssey.conf max clients. ? , instance Odyssey per_worker? ? . . user1 , , 50 Odyssey, user2 , , 100. ?
, . . , TLS handshake, , . . . , . TLS - , , . max clients – . , .
max clients 20 000 20 000 TCP . 20 001- , : «» TLS handshake. pool size. pool size «1», , , , , - . , 5300: too many clients for this pool, . . sorry, .
. . wait - pool size?
Database. database user. user pool size. Postgres. Postgres : «too many clients». Odyssey . , round trip postmaster. Postmaster , . , , pool size. pool size.
! . Bouncer : MD5 hash. , MD5 hash?
.
. . ?
, . . include .
, ?
, . Storage , listen , . . GitHub : « , . . - , ». , . , - . , , . .