Get best of the week

Data backup using a bunch of FreeFileSync and 7-zip

Anamnesis, so to speak:

Fujitsu rx300 s6 server, RAID6 of 6 1TB of disks, XenServer 6.2 is up, several servers are spinning, among them Ubunta with several balls, 3.5 million files, 1.5 TB of data, all this gradually grows and swells.

Task: configure backup of data from the file server, partially daily, partially weekly.
We have a Windows backup machine with RAID5 (in terms of poverty, a regular system unit with a RAID controller built into the mother) plus a separate 2TB disk for intermediate copying of the current state of files. You could use any Linux distribution, but this machine was already in stock with a raid array and a Windows license.

We install FreeFileSync on the backup server , configure the “mirror” of everything in a row from all the file server balls once a day in the evening after 18 hours by running through the scheduler.

An important point: when saving a batch task, it is obligatory to mark “Close the task window at completion”, otherwise the processes will multiply and multiply.

We throw temporary files into mask exceptions: * .dwl, * .dwl2, * .tmp.

FreeFileSync uses the network extremely well, copying takes several streams, the speed reaches 80 Mb / s when copying large files, no plugging was detected on small files.

We will carry out archiving already on the local backup server, instead of the TheCopier used earlierwith archiving on a network. By the way, TheCopier is great! But with such volumes, it just doesn’t have time to transfer everything, despite the 1GB \ s interface on the backup and 2GB \ s on the file interface (bond of two network cards). SyncToy was

also previously used , but with the number of files more than 1.5-2 million, it stopped working normally, just could not cope. To archive the necessary folders, we write a batch file for 7-zip : set now =% TIME: ~ 0, -3% set now =% now :: =.% Set now =% now: = 0% set now =% DATE: ~ - 4%.% DATE: ~ 3.2%.% DATE: ~ 0.2% _% now% C: \ "Program Files" \ 7-Zip \ 7z.exe a -tzip -mx = 1 -mmt = on -mtc = off -ssw D: \ backups \ All \% now% _10-04.zip E: \ 10-04








C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_35-110.zip E:\35-110
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_asu.zip E:\asu
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_director.zip E:\director
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_gpr.zip E:\gpr
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_otiz.zip E:\otiz
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_ps.zip E:\ps
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_pto.zip E:\pto
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_rza.zip E:\rza
C:\«Program Files»\7-Zip\7z.exe a -tzip -mx=1 -mmt=on -mtc=off -ssw D:\backups\All\%now%_smeta.zip E:\smeta

:: a —
:: -tzip -t7z — ( zip 1.5-2 )
:: -mx=1 — (1 , 9 x=[0 | 1 | 3 | 5 | 7 | 9 ])
:: -mmt=on —
:: -mtc=off — ( , ..)
:: -ssw — ,
:: -xr!.Sync* — BtSync ,


The construction from set now =% and so on allows you to save the time record format in the file name without the problems that occurred when the day or month was less than 10, that is, we substitute the zero.

Comment -xr! .Sync * is the vestigial remnant of the originally used BTSync .

Up to 500 GB and 700-800 thousand files, BTSync still worked nothing, synchronized on the fly, but at current volumes it was eating memory and processor resources both on the ubunt file server and on the Windows backup where it was launched by the service, and also simply raped a disk system with constant read-write.

Although the archiver is also 7-zip, we archive it in zip format instead of the native 7z, because it is much faster, and there is practically no difference in compression with mx = 1, it has been tested by many experiments.

Archives are executed in turn.

Cleaning the archive folder also occurs through the scheduled task using the fpurge utility, leaving the archives no older than a week.
As a result, we have a copy of the files for the previous day, as well as archives for the last week, FreeFileSync puts the deleted files in the trash.

Source: https://habr.com/ru/post/undefined/

More articles:


All Articles