Proponho me familiarizar com a decodificação do relatório do início de 2019 por Andrei Borodin "Backups da WAL-G. O que há em 2019?"
Olá a todos! Meu nome é Andrey Borodin. Sou desenvolvedor da Yandex. Eu estou interessado no PostgreSQL desde 2016, depois de conversar com os desenvolvedores, e eles disseram que tudo é simples - você pega o código fonte e compila, e tudo vai dar certo. E desde então não consigo parar - estou escrevendo todo tipo de coisas diferentes.
, , WAL-G. , PostgreSQL . , . , , .
, , , , . WAL-G? , , .
PostgreSQL, , , S3 Google Cloud .
, Yandex Object Storage, S3 .
Bucket. .
.
C aws-s3-key.
WAL-G.
? . , , , . . , - , , . . , . . .
, WAL-G, «backup list», . Object Storage, , . , , . , . , .
. WAL-G «backup-push» WAL-G pgdata . , , PostgreSQL , , «archive-mode».
, «archive_mode = on» «archive_command», WAL-G. - - WAL-G. , . , WAL-G. - , GitHub. WAL-G , , archive_command.
WAL-G High Availability Database management.
. Yandex Object Storage.
– Point in time recovery. . .
. ? , . . pg_basebackup , . , , .
. -, .
- , , pg_basebackup .
- . , , .
- , . - , .
- , . , contribution WAL-G. WAL-G, : « , - ?». WAL-G.
- , , . .
- .
- .
. , PostgreSQL. - , - , - .
, WAL-G. . .
, – , WAL-G .
WAL-G 4 . :
WAL-PUSH – .
WAL-FETCH – .
BACKUP-PUSH – .
BACKUP-FETCH – .
, WAL-G , . . listing , .
– -.
- , , . , WAL. WAL-, - . , , , - fail, 4 - 10 WAL. , - .
LSN-based – , , LSN LSN , , . , , -.
, .
API PostgreSQL . PostgreSQL WAL- WAL-. WAL- «WAL-FETCH», «WAL-PREFETCH», 8 , .
, archive_status WAL-. WAL . , WAL. , , , API PostgreSQL. PostgreSQL archive_status , WAL-. , , . PostgreSQL, , .
LSN-based WAL- , mode-time . , . , WAL-.
, WAL , , , . , . , delta-.
Delta- WAL-, , WAL. delta- .
, , , , WAL, , , , .
_delta_partial . , , WAL , , .
, , , , , , , LSN-based .
, WAL-G. , , , - – , . ( https://github.com/wal-g/wal-g/tree/master/internal/walparser)
. delta-. , , delta- , . , . delta-.
. . LSN-based, . , LSN-based delta- . . WAL-delta 20. . . , .
, , – PostgreSQL, «pg_prefaulter»
, , Stand-by restore command, WAL-G WAL-. , WAL . , SSD . WAL , . page-. , . WAL-G, , WAL . , . SSD-, , , , .
, .
, .
, WAL-delta . , . WAL-delta , . WAL-delta , . , . «instant WAL-delta».
, , , , . . . WAL, PostgreSQL , .
. , - , . . . WAL-delta. , , , - - , delta pg_basebackup, .
. . . - , , , .
, , .
WAL-G. , . , API MySQL . MySQL PITR WAL-, binlog. MySQL, - , binlog . - cron – - ?
MySQL restore command, , - -. , , , . , . , , - . (: MySQL )
, WAL-G .
, WAL-G . , . , WAL-G. - , High Availability , .
, WAL-G WAL-E. , WAL-G WAL-E , WAL-E. wal-push, . WAL-G , WAL-E . WAL-E archive-status, , , , WAL- , PostgreSQL .
WAL-G? PostgreSQL , , PostgreSQL , , mode-time md5 PostgreSQL – OK, , , .
WAL-E , archive command, WAL-G, WAL-E .
, , WAL-G , .
-, . , . , . , . . , WAL-G , smoke-: pg_dumpall /dev/null amcheck .
WAL-G WAL. . . . , , , . : « , ». WAL-G . ( — . — backup-mark https://github.com/wal-g/wal-g/blob/master/PostgreSQL.md)
PITR.
Google Summer of Code. , - Go «G», . , . , .
, . .
, WAL-G , . , , . pg_control . , - . , , . , .
offline-. , . . LSN, . LSN control . . . .
. . . - Go , bucket. pull request.
, , , , . . , , . , , PostgreSQL 9.4-9.5. , PostgreSQL, , . , , .
WAL-G , Database management. .
TODO. , , pull request, .
! ! , WAL-delta, , , full-page writes. , ? . , FPW ?
Full-page writes , . . . , , . , , , . FPW . , .
! . – ?
pull request. SSD NMVE . , . , . , , , . , pull request. ( 0.2.13)
. , WAL-G , . . ?
, , . , , . , . , , – Go.
! ! WAL-G GPG . , . - ? .
GitHub, .
. . ?
, WAL-G , , : « , ?». , . . . , , . GPG , . GPG, open PGP, WAL-G, . Libsodium ( 0.2.15). , , – , . - . (PGP encryption without external GPG — v0.2.9)
! ! . , pg_basebackup WAL log , . . . - ?
, .
, .
. . , issue GitHub?
, .
, Google Summer of Code, , , .
. issue GitHub. , -, . restore. , . issues. , . . , .
, - (CEPH) , . ? retry . . - , , . , , . retry — exponential backoff. , , . . . , . ? , , . , , , .
SLA? , ?
, , , . . . Amazon Google Cloud, Yandex Object Storage .
?
. - , , WAL-G. , . Object Storage, - listing . , folder. WAL-G – - , , , , . . . , , Object Storage - , .
Mail .
reproduce…
…
reproduce, , , retry , , . , , . , . . 16 , retry 8 , 4 , 2 . . - 7,5 , 7,5 . .
! workflow WAL-G? , , . , . , , . - delta- , . . - ?
API delta- . – max delta steps, . . , , backup-push, . , , 3, , backup-push, . , 3 .
. . , WAL-G, ?
, WAL-G, , .
, , pg_basebackup?
, , . Pg_basebackup . WAL-G , . warning, .
Pg_basebackup .
, . Pg_basebackup . , , , . pg_basebackup . , , .
, CephFS. Object Storage .
, , , . .
! . remote storage, , - -?
– . , , . . . , . 0.2. , . . . WAL-E, . delta-backup. . , , Google Cloud, S3- storage.
, . , WAL-G – - barmen? ?
, . WAL-G , , . , . , - , . , , . WAL-G . - , .
, storage. , , , , . , - . . , , . , - . . , , . - , , – , .
PS Uma nova versão 0.2.15 foi lançada , na qual você pode usar o arquivo de configuração .walg.json, localizado por padrão no diretório inicial do postgres. Você pode optar por sair dos scripts do bash. Um exemplo de .walg.json está nesta edição https://github.com/wal-g/wal-g/issues/545
Vídeo: