Backup und Restore

Server backup

(Local file system → ZFS storage →) ZFS snapshot → ZFS synchronization.

Rsync – local

To back up a local folder between two file systems we use Rsync + Crontab.

First we set up the target directory, e.g.:

sudo mkdir /storage/backup/local
sudo chown -R USERNAME: /storage/backup/local

We store the backup script under /data/scripts/backup/local_docker.sh, and it looks as follows:

#!/bin/sh

#README: https://niklas-stephan.de/wiki/tutorials/backup-und-restore/

# Set variables
SOURCE_DIR="/data/docker"
BACKUP_DIR="/storage/backup/local"
NTFY_URL="https://notify.handtrixxx.com/backup"

rsync -aX --delete --partial --inplace --timeout=60 \
    "$SOURCE_DIR" "$BACKUP_DIR"
RSYNC_EXIT="$?"
#RSYNC_EXIT="1"

if [ "$RSYNC_EXIT" -ne 0 ]; then
    if [ "$RSYNC_EXIT" -eq 23 ]; then
        MESSAGE="Backup of \"$SOURCE_DIR\" completed with partial errors (exit code 23 - some files could not be read)"
    else
        MESSAGE="Backup of \"$SOURCE_DIR\" failed with exit code $RSYNC_EXIT"
    fi
    curl --max-time 30 -s -o /dev/null\
        -H "Title: Backup failed on $(hostname)" \
        -H "Tags: warning" \
        -d "$MESSAGE" \
        "$NTFY_URL"
fi

We make this script executable via:

chmod +x /data/scripts/backup/local_docker.sh

To execute the script manually, the following command applies:

sudo /data/scripts/backup/local_docker.sh

For the automation we create a cron job that should run daily at 02:00.

sudo crontab -e

And add this line:

0 2 * * * /data/scripts/backup/local_docker.sh >> /var/log/backup_local_docker.log 2>&1

Explanation:

  • Runs daily at 02:00 (0 2 * * *)
  • Runs as root
  • Creates a log file for error analysis

ZFS – local

We use Sanoid for creating ZFS snapshots.

We need 2 files: first a configuration file for Sanoid in /data/scripts/backup/sanoid.conf with content like:

# Datasets to snapshot
# Add or remove [dataset] blocks to control which datasets are managed

[storage/backup/local]
    use_template = daily

[template_daily]
    frequently = 0
    hourly = 0
    daily = 14
    monthly = 0
    yearly = 0
    autosnap = yes
    autoprune = yes

Then our actual backup script /data/scripts/backup/local_zfs.sh:

#!/bin/sh

#README: https://niklas-stephan.de/wiki/tutorials/backup-und-restore/

# Set variables
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
NTFY_URL="https://notify.handtrixxx.com/backup"

sanoid --configdir="$SCRIPT_DIR" --cron
SANOID_EXIT="$?"

if [ "$SANOID_EXIT" -ne 0 ]; then
    MESSAGE="ZFS snapshot job failed with exit code $SANOID_EXIT"
    curl --max-time 30 -s -o /dev/null \
        -H "Title: ZFS snapshot failed on $(hostname)" \
        -H "Tags: warning" \
        -d "$MESSAGE" \
        "$NTFY_URL"
fi

We make this script executable via:

chmod +x /data/scripts/backup/local_zfs.sh

To execute the script manually, the following command applies:

sudo /data/scripts/backup/local_zfs.sh

For the automation we create a cron job that should run daily at 03:00.

sudo crontab -e

And add this line:

0 3 * * * /data/scripts/backup/local_zfs.sh >> /var/log/zfs_snapshot.log 2>&1

Explanation:

  • Runs daily at 03:00 (0 3 * * *)
  • Runs as root
  • Creates a log file for error analysis

ZFS – remote sync

We want to duplicate the snapshots existing on a remote system on another server, in order to be able to show a double backup at another physical location. That is the deployment scenario of Syncoid.

First we create a backup script /data/scripts/backup/remote_zfs.sh on the TARGET SYSTEM:

#!/bin/sh

#README: https://niklas-stephan.de/wiki/tutorials/backup-und-restore/

# Set variables
REMOTE_USER="USER"
REMOTE_HOST="HOST"
REMOTE_PORT="PORT"
SSH_CIPHER="aes128-gcm@openssh.com"
NTFY_URL="https://notify.handtrixxx.com/backup"

DATASETS="
zfsdata/sources:storage/backup/sources
"
FAILED=0
FAILED_DATASETS=""

for ENTRY in $DATASETS; do
    REMOTE_DS="${ENTRY%%:*}"
    LOCAL_DS="${ENTRY##*:}"

    syncoid \
        --sshport "$REMOTE_PORT" \
        --sshoption "Ciphers=$SSH_CIPHER" \
        "$REMOTE_USER@$REMOTE_HOST:$REMOTE_DS" \
        "$LOCAL_DS"

    EXIT="$?"
    if [ "$EXIT" -ne 0 ]; then
        FAILED=1
        FAILED_DATASETS="$FAILED_DATASETS $REMOTE_DS"
    fi
done

if [ "$FAILED" -ne 0 ]; then
    MESSAGE="Syncoid pull from $REMOTE_HOST failed for datasets:$FAILED_DATASETS"
    curl --max-time 30 -s -o /dev/null \
        -H "Title: ZFS sync failed on $(hostname)" \
        -H "Tags: warning" \
        -d "$MESSAGE" \
        "$NTFY_URL"
fi

We make this script executable via:

chmod +x /data/scripts/backup/remote_zfs.sh

To execute the script manually, the following command applies:

sudo /data/scripts/backup/remote_zfs.sh

For the automation we create a cron job that should run daily at 01:00.

sudo crontab -e

And add this line:

0 1 * * * /data/scripts/backup/remote_zfs.sh >> /var/log/zfs_remote_snapshot.log 2>&1

Explanation:

  • Runs daily at 01:00 (0 1 * * *)
  • Runs as root
  • Creates a log file for error analysis

If one wants to see the process (state of the data transfer), one can do that with the following command:

watch -n 1 'zfs list storage/backup/media'

That executes the command every second, so that one sees the current state.

:::warning The first remote synchronization can take long depending on the amount of data; one should watch out to deactivate the cron job for this period initially. :::

Server restore

ZFS – local

To see existing snapshots in the file system in the .zfs folder, this must first be activated:

sudo zfs set snapdir=visible zfsdata