Get best of the week

Remote incremental ZFS backups. Nowhere easier

Secure your brother, secure ...


ZFS 0.8 has the ability to send incremental snapshots of encrypted data without decrypting. That is, having an encrypted array, you can organize a mirror copy of it on a remote machine, not trusting the owner of the remote premises. The key and decrypted data do not leave the trusted premises.

None of my clients (microcontrollers) have more than one server room, if a fire comes there, for example, a star. Having agreed with your colleague in a dangerous admin business, you can mutually insure each other without fear that they might use the data.

Any service in the form of a virtual machine is installed on Proxmox, where the data is located on ZFS, and you will always have the opportunity to store a remote copy of the virtual machine. There is a chance that one day instead of falling into despondency, you will go to a remote room, pick up a piece of iron and run all the services in good health. I already transferred almost all of my services to Proxmox about 2 years ago and I have no regrets, everything works fine. On ZFS, I have a terminal server for 30 people, which is tied to thin clients (that is, 30 full-fledged jobs). In addition to the current copy, you will also have snapshots for the last x days both locally and remotely.

Yes, you will lose productivity from the very use of zfs and from encryption, but for myself I came to the conclusion that I was ready to pay this price for such functionality.

A spoon of tar


In tackle you immediately fly in the ointment. Yes, ZFS is slower. Essentially. But you cannot use it as a regular fs. It should work in an array of disks, where an increase in the number of disks is welcome, the disks should preferably be SAS, the controller should work in HBA mode (and not raid0 as single disks), there should be ECC RAM and it should be many, there must be one SSD (better PCI-E) as a cache for reading the second level (small, gigs from 64) and one SSD for ZIL can be very small, but preferably MLC.

Regarding the SSD, I put on these recommendations and bought 2 PCI-E SSDs of 128 and each divided into 16 gigs and the rest. I launched two 16 gigabyte partitions in ZIL as a mirror and launched the remaining parts under the second level cache as stripe. At the mercy of zfs gave 16 gigs of opera. (One slot on a mother bought with ebay turned out to be inoperative; if it worked, it would give 32)

zfskeeper.sh


From the script I needed automatic notification of problems, monitoring the number of snaps at both ends and speed control. Of the scripts that fell into my hands, there were no scripts that satisfied me. Below I will give some of the scripts that I reviewed before doing this. To receive notifications by e-mail, replace my data in your send_error_to_mail () function with yours

zfskeeper.sh help will show you examples of use
zfskeeper.sh prepare will tell you the procedure for the preparatory actions (recommended)
zfskeeper.sh bank / encrypted / subvol-100-disk-0 10 will take the current snapshot and leave the last 10 snapshots
zfskeeper.sh bank / encrypted / subvol-100-disk-0 10 Remotehost 22 10m replicator bank / rtl / subvol-100-disk-0 20Send unencrypted data to remotehost: 22 under the user replicator with a speed of up to 10MB / s and leave 10 images on the source and 20 on the remote host
zfskeeper.sh bank / encrypted / subvol-100-disk-0 10 Remotehost 22 10m replicator bank / rtl / subvol -100-disk-0 20 -w the same thing, but the data will be encrypted

#!/bin/bash

zpool=$1
lsnaps=$2

ip=$3
port=$4
speedl=$5
remoteuser=$6
backpool=$7
rsnaps=$8
sendargs=$9

ZFSLOGDIR=~/zfskeeplogs
if [ ! -d "$ZFSLOGDIR" ]; then
    mkdir -p $ZFSLOGDIR
fi

NAMESNAP=`date "+%Y_%m_%d_%T"|sed s/:/_/g`
logfile=$ZFSLOGDIR/$NAMESNAP.log

write_to_log ()
{
echo `date +"%T"`" $1" >> $logfile
}

log_last_command ()
{
if [ $? -ne 0 ]; then
	    write_to_log "$1 FAILED."
	    if [ $2 = true ]; then
		write_to_log "Script execution stops"
		send_error_to_mail "$1" "$?" "$*"
		exit 1
	    fi
	else
	    write_to_log "$1 COMPLETED."
	fi
}

send_error_to_mail ()
{

Subject=`hostname`" ZFS backup $zpool missing"
swaks --header "Subject: $Subject" --to admin@mydomain.ru --from "alertbot@mydomain.ru" -a LOGIN -au alertbot@mydomain.ru -ap ailPass --server mail.mydomain.ru -tls --body \
"$Subject
command: zfskeeper.sh $3

    $1
    $2" \
    --attach $logfile
#    --attach-type "$(get_mimetype $logfile)" --attach $logfile

}

case "$#" in
    1)
    case "$1" in
	prepare)
    echo "
    Sender mashine:
    apt install swaks mbuffer
    useradd -s /bin/bash replicator
    mkdir -p /home/replicator 
    chown replicator:replicator /home/replicator
    zfs allow -u replicator send,snapshot,destroy,mount bank/encrypted
    su replicator
    ssh-keygen

    Receiver machine:
    ln -s /usr/sbin/zfs /usr/local/bin/zfs
    useradd -s /bin/bash replicator
    mkdir -p /home/replicator 
    chown replicator:replicator /home/replicator
    passwd replicator

    Sender mashine:
    ssh-copy-id -i .ssh/id_rsa.pub Remotehost
    zfs allow -u replicator compression,create,receive,destroy,mount bank/companyname

    "
    ;;
	*)
    echo "
    Usage:

    To show the prepare instrutions:
    zfskeeper.sh prepare

    localkeeping:
    keep last 10 snapshots of bank/encrypted/subvol-100-disk-0

    zfskeeper.sh bank/encrypted/subvol-100-disk-0 10

    remotekeeping:

    keep last 10 snapshots bank/encrypted/subvol-100-disk-0 and send it by ssh to Remotehost:22 bank/rtl/subvol-100-disk-0
    and keep last 20 copies by replicator user and limit transferspeed 10Mb/s

    zfskeeper.sh bank/encrypted/subvol-100-disk-0 10 Remotehost 22 10m replicator bank/rtl/subvol-100-disk-0 20

    If you need to send encrypted data, then you need add -w to the end of the line

    zfskeeper bank/encrypted/subvol-100-disk-0 10 Remotehost 22 10m replicator bank/rtl/subvol-100-disk-0 20 -w
    "
    ;;
    esac
    exit 0
    ;;
    2)
    echo `date +"%T"`" Local keeping of $NAMESNAP being started" > $logfile
    ;;
    8|9)
    echo `date +"%T"`" Remote keeping of $NAMESNAP being started" > $logfile
    ;;
    *)
    echo "illegal number of parameters" >&2
    exit 1
    ;;
esac



####################################### Local part #######################################################

# Remove waste local snaps
numsnap=`zfs list -H -o name -t snapshot -s creation|grep "${zpool}@"|wc|awk '{ print $1 }'`
let MAXSNAP=numsnap-lsnaps+1 >/dev/null
if [ $MAXSNAP -gt 0 ] ; then
    for d in `zfs list -H -o name -t snapshot -s creation| grep "${zpool}@"|/usr/bin/head -n"$MAXSNAP"`; do
        zfs destroy ${d}
        log_last_command "Remove local snapshot $d"
    done
fi

# Create fresh local snap
zfs snapshot $zpool@backup_$NAMESNAP
log_last_command "Create local $zpool@backup_$NAMESNAP" true

####################################### Local part #######################################################

if [ -z "$backpool" ]; then
    write_to_log "Local keeping is complete"
    exit 0
fi

####################################### Remote part #######################################################


# Check remote address
if [ -n "$ip" ]; then
    if ping -c 1 $ip > null ; then
            sship="ssh -c aes128-ctr -p"$port" -l"$remoteuser" "$ip
    else
            echo "URL $ip is not accesiible, abort"
            send_error_to_mail "Try to ping remote host" "URL $ip is not accesiible, abort" "$*"
            exit 1
    fi
else
    # Remote backup is unnecessary
    exit 0
fi


# Remove waste local snaps
numsnap=`$sship zfs list -H -o name -t snapshot -s creation|grep "${backpool}@"|wc|awk '{ print $1 }'`
let MAXSNAP=numsnap-rsnaps+1 >/dev/null
if [ $MAXSNAP -gt 0 ] ; then
    for d in `$sship zfs list -H -o name -t snapshot -s creation| grep "${backpool}@"|/usr/bin/head -n"$MAXSNAP"`; do
        $sship zfs destroy ${d}
        log_last_command "Remove local snapshot $d"
    done
fi

# Receive last remote snap
MOST_SLAVE_SNAP_NAME=`$sship zfs list -H -o name -t snapshot -s creation|grep "$backpool@backup"|tail -n1|awk -F "@" '{print $2}'`

# If localbackup has same snap then send incremental copy. If has't then send full
if [ -n "`zfs list -H -o name -t snapshot -s creation|grep -w "$zpool@$MOST_SLAVE_SNAP_NAME"`" ]; then
    #zfs send $sendargs -i $MOST_SLAVE_SNAP_NAME $zpool@backup_$NAMESNAP | pv -L $speedl | $sship zfs receive -vF $backpool@backup_$NAMESNAP &>> $logfile
    zfs send $sendargs -i $MOST_SLAVE_SNAP_NAME $zpool@backup_$NAMESNAP | mbuffer -q -v 0 -s 128k -m 1G | pv -L $speedl | $sship "mbuffer -q -v 0 -s 128k -m 1G | zfs receive -vF $backpool@backup_$NAMESNAP" &>> $logfile
    log_last_command "Sending incremental backup to the remote machine" true
else
    #If backpool exist we remove it. You can rename it
    if [ -n "`$sship zfs list -H -o name -s creation|grep -w "$backpool"`" ]; then
	#$sship zfs rename -p $backpool `$sship zfs list -H -o name|grep -w "$backpool"|awk -F "/" '{print $1}'`/old_$NAMESNAP
	$sship zfs destroy -r $backpool 2>> $logfile
	log_last_command "Need to destroy remotepool for full sending $backpool" true
    fi
    #zfs send $sendargs $zpool@backup_$NAMESNAP | pv -L $speedl | $sship zfs receive -vF $backpool &>> $logfile
    zfs send $sendargs $zpool@backup_$NAMESNAP | mbuffer -q -v 0 -s 128k -m 1G | pv -L $speedl | $sship "mbuffer -q -v 0 -s 128k -m 1G | zfs receive -vF $backpool" &>> $logfile

    log_last_command "Sending full backup to the remote machine" true
fi


####################################### Remote part #######################################################

write_to_log "Remote keeping is complete"
exit 0


zfskeepall.sh


In order to keep toned several sections, you need a control script. Everyone can already make it to their taste and color, depending on the configuration. My test looks like this:

#!/bin/bash
scriptpath=/home/replicator/zfskeeper.sh
$scriptpath bank/encrypted/subvol-100-disk-0 5 RemoteHost 22 30m replicator bank/rtl/subvol-100-disk-0 5 -w

Existing solutions


I found it first, based on it I checked the overall operability of the mechanism. Thanks to the authors, but the writing style is rather dirty, it had to be rewritten unambiguously.

zfs-auto-snapshot.sh is just a block. There is something to borrow, beautiful, neat, but bulky and

low- functional zfsbackup.sh - also a neat, but

low- functional PS I bit the zfs from the third time, each time it nibbled for about a month. I could not get her to work quickly, and to the fullest failed. In general, the situation changed after installing SAS disks (at least 7200, at least NL).

Peace for everyone! Thank you for reading!

More articles:


All Articles