I propose to read the decoding of the report of the beginning of 2019 by Andrei Borodin "Backups from WAL-G. What is there in 2019?"
Hello everyone! My name is Andrey Borodin. I am a developer at Yandex. I have been interested in PostgreSQL since 2016, after I talked with the developers, and they said that everything is simple - you take the source code and compile, and everything will work out. And since then I can’t stop - I’m writing all kinds of different things.
One of the things I'm doing is the WAL-G backup system . In general, at Yandex we have been doing backup systems in PostgreSQL for a very long time. And you can find on the Internet a series of six reports on how we make backup systems. And every year they evolve a little, develop a little, become more reliable.
But today the report is dedicated not only to what we have done, but also about how simple it is and what it is. How many of you have already watched my WAL-G reports? It’s good that quite a few people didn’t look, because I will start with the simplest thing.
If suddenly you have a PostgreSQL cluster, and I think everyone has a couple with them, and suddenly there is no backup system yet, then you need to get any S3 storage or Google Cloud compatible storage.
, Yandex Object Storage, S3 .
Bucket. .
.
C aws-s3-key.
WAL-G.
? . , , , . . , - , , . . , . . .
, WAL-G, «backup list», . Object Storage, , . , , . , . , .
. WAL-G «backup-push» WAL-G pgdata . , , PostgreSQL , , «archive-mode».
, «archive_mode = on» «archive_command», WAL-G. - - WAL-G. , . , WAL-G. - , GitHub. WAL-G , , archive_command.
WAL-G High Availability Database management.
. Yandex Object Storage.
– Point in time recovery. . .
. ? , . . pg_basebackup , . , , .
. -, .
- , , pg_basebackup .
- . , , .
- , . - , .
- , . , contribution WAL-G. WAL-G, : « , - ?». WAL-G.
- , , . .
- .
- .
. , PostgreSQL. - , - , - .
, WAL-G. . .
, – , WAL-G .
WAL-G 4 . :
WAL-PUSH – .
WAL-FETCH – .
BACKUP-PUSH – .
BACKUP-FETCH – .
, WAL-G , . . listing , .
– -.
- , , . , WAL. WAL-, - . , , , - fail, 4 - 10 WAL. , - .
LSN-based – , , LSN LSN , , . , , -.
, .
API PostgreSQL . PostgreSQL WAL- WAL-. WAL- «WAL-FETCH», «WAL-PREFETCH», 8 , .
, archive_status WAL-. WAL . , WAL. , , , API PostgreSQL. PostgreSQL archive_status , WAL-. , , . PostgreSQL, , .
LSN-based WAL- , mode-time . , . , WAL-.
, WAL , , , . , . , delta-.
Delta- WAL-, , WAL. delta- .
, , , , WAL, , , , .
_delta_partial . , , WAL , , .
, , , , , , , LSN-based .
, WAL-G. , , , - – , . ( https://github.com/wal-g/wal-g/tree/master/internal/walparser)
. delta-. , , delta- , . , . delta-.
. . LSN-based, . , LSN-based delta- . . WAL-delta 20. . . , .
, , – PostgreSQL, «pg_prefaulter»
, , Stand-by restore command, WAL-G WAL-. , WAL . , SSD . WAL , . page-. , . WAL-G, , WAL . , . SSD-, , , , .
, .
, .
, WAL-delta . , . WAL-delta , . WAL-delta , . , . «instant WAL-delta».
, , , , . . . WAL, PostgreSQL , .
. , - , . . . WAL-delta. , , , - - , delta pg_basebackup, .
. . . - , , , .
, , .
WAL-G. , . , API MySQL . MySQL PITR WAL-, binlog. MySQL, - , binlog . - cron – - ?
MySQL restore command, , - -. , , , . , . , , - . (: MySQL )
, WAL-G .
, WAL-G . , . , WAL-G. - , High Availability , .
, WAL-G WAL-E. , WAL-G WAL-E , WAL-E. wal-push, . WAL-G , WAL-E . WAL-E archive-status, , , , WAL- , PostgreSQL .
WAL-G? PostgreSQL , , PostgreSQL , , mode-time md5 PostgreSQL – OK, , , .
WAL-E , archive command, WAL-G, WAL-E .
, , WAL-G , .
-, . , . , . , . . , WAL-G , smoke-: pg_dumpall /dev/null amcheck .
WAL-G WAL. . . . , , , . : « , ». WAL-G . ( — . — backup-mark https://github.com/wal-g/wal-g/blob/master/PostgreSQL.md)
PITR.
Google Summer of Code. , - Go «G», . , . , .
, . .
, WAL-G , . , , . pg_control . , - . , , . , .
offline-. , . . LSN, . LSN control . . . .
. . . - Go , bucket. pull request.
, , , , . . , , . , , PostgreSQL 9.4-9.5. , PostgreSQL, , . , , .
WAL-G , Database management. .
TODO. , , pull request, .
! ! , WAL-delta, , , full-page writes. , ? . , FPW ?
Full-page writes , . . . , , . , , , . FPW . , .
! . – ?
pull request. SSD NMVE . , . , . , , , . , pull request. ( 0.2.13)
. , WAL-G , . . ?
, , . , , . , . , , – Go.
! ! WAL-G GPG . , . - ? .
GitHub, .
. . ?
, WAL-G , , : « , ?». , . . . , , . GPG , . GPG, open PGP, WAL-G, . Libsodium ( 0.2.15). , , – , . - . (PGP encryption without external GPG — v0.2.9)
! ! . , pg_basebackup WAL log , . . . - ?
, .
, .
. . , issue GitHub?
, .
, Google Summer of Code, , , .
. issue GitHub. , -, . restore. , . issues. , . . , .
, - (CEPH) , . ? retry . . - , , . , , . retry — exponential backoff. , , . . . , . ? , , . , , , .
SLA? , ?
, , , . . . Amazon Google Cloud, Yandex Object Storage .
?
. - , , WAL-G. , . Object Storage, - listing . , folder. WAL-G – - , , , , . . . , , Object Storage - , .
Mail .
reproduce…
…
reproduce, , , retry , , . , , . , . . 16 , retry 8 , 4 , 2 . . - 7,5 , 7,5 . .
! workflow WAL-G? , , . , . , , . - delta- , . . - ?
API delta- . – max delta steps, . . , , backup-push, . , , 3, , backup-push, . , 3 .
. . , WAL-G, ?
, WAL-G, , .
, , pg_basebackup?
, , . Pg_basebackup . WAL-G , . warning, .
Pg_basebackup .
, . Pg_basebackup . , , , . pg_basebackup . , , .
, CephFS. Object Storage .
, , , . .
! . remote storage, , - -?
– . , , . . . , . 0.2. , . . . WAL-E, . delta-backup. . , , Google Cloud, S3- storage.
, . , WAL-G – - barmen? ?
, . WAL-G , , . , . , - , . , , . WAL-G . - , .
, storage. , , , , . , - . . , , . , - . . , , . - , , – , .
PS A new version 0.2.15 has been released , in which you can use the .walg.json configuration file, which by default is in the postgres home directory. You can opt out of bash scripts. An example of .walg.json is in this issue https://github.com/wal-g/wal-g/issues/545
Video: