Server backup
(Local file system → ZFS storage →) ZFS snapshot → ZFS synchronization.
Rsync – local
To back up a local folder between two file systems we use Rsync + Crontab.
First we set up the target directory, e.g.:
sudo mkdir /storage/backup/local
sudo chown -R USERNAME: /storage/backup/local
We store the backup script under /data/scripts/backup/local_docker.sh, and it looks as follows:
#!/bin/sh
#README: https://niklas-stephan.de/wiki/tutorials/backup-und-restore/
# Set variables
SOURCE_DIR="/data/docker"
BACKUP_DIR="/storage/backup/local"
NTFY_URL="https://notify.handtrixxx.com/backup"
rsync -aX --delete --partial --inplace --timeout=60 \
"$SOURCE_DIR" "$BACKUP_DIR"
RSYNC_EXIT="$?"
#RSYNC_EXIT="1"
if [ "$RSYNC_EXIT" -ne 0 ]; then
if [ "$RSYNC_EXIT" -eq 23 ]; then
MESSAGE="Backup of \"$SOURCE_DIR\" completed with partial errors (exit code 23 - some files could not be read)"
else
MESSAGE="Backup of \"$SOURCE_DIR\" failed with exit code $RSYNC_EXIT"
fi
curl --max-time 30 -s -o /dev/null\
-H "Title: Backup failed on $(hostname)" \
-H "Tags: warning" \
-d "$MESSAGE" \
"$NTFY_URL"
fi
We make this script executable via:
chmod +x /data/scripts/backup/local_docker.sh
To execute the script manually, the following command applies:
sudo /data/scripts/backup/local_docker.sh
For the automation we create a cron job that should run daily at 02:00.
sudo crontab -e
And add this line:
0 2 * * * /data/scripts/backup/local_docker.sh >> /var/log/backup_local_docker.log 2>&1
Explanation:
- Runs daily at 02:00 (
0 2 * * *) - Runs as root
- Creates a log file for error analysis
ZFS – local
We use Sanoid for creating ZFS snapshots.
We need 2 files: first a configuration file for Sanoid in /data/scripts/backup/sanoid.conf with content like:
# Datasets to snapshot
# Add or remove [dataset] blocks to control which datasets are managed
[storage/backup/local]
use_template = daily
[template_daily]
frequently = 0
hourly = 0
daily = 14
monthly = 0
yearly = 0
autosnap = yes
autoprune = yes
Then our actual backup script /data/scripts/backup/local_zfs.sh:
#!/bin/sh
#README: https://niklas-stephan.de/wiki/tutorials/backup-und-restore/
# Set variables
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
NTFY_URL="https://notify.handtrixxx.com/backup"
sanoid --configdir="$SCRIPT_DIR" --cron
SANOID_EXIT="$?"
if [ "$SANOID_EXIT" -ne 0 ]; then
MESSAGE="ZFS snapshot job failed with exit code $SANOID_EXIT"
curl --max-time 30 -s -o /dev/null \
-H "Title: ZFS snapshot failed on $(hostname)" \
-H "Tags: warning" \
-d "$MESSAGE" \
"$NTFY_URL"
fi
We make this script executable via:
chmod +x /data/scripts/backup/local_zfs.sh
To execute the script manually, the following command applies:
sudo /data/scripts/backup/local_zfs.sh
For the automation we create a cron job that should run daily at 03:00.
sudo crontab -e
And add this line:
0 3 * * * /data/scripts/backup/local_zfs.sh >> /var/log/zfs_snapshot.log 2>&1
Explanation:
- Runs daily at 03:00 (
0 3 * * *) - Runs as root
- Creates a log file for error analysis
ZFS – remote sync
We want to duplicate the snapshots existing on a remote system on another server, in order to be able to show a double backup at another physical location. That is the deployment scenario of Syncoid.
First we create a backup script /data/scripts/backup/remote_zfs.sh on the TARGET SYSTEM:
#!/bin/sh
#README: https://niklas-stephan.de/wiki/tutorials/backup-und-restore/
# Set variables
REMOTE_USER="USER"
REMOTE_HOST="HOST"
REMOTE_PORT="PORT"
SSH_CIPHER="aes128-gcm@openssh.com"
NTFY_URL="https://notify.handtrixxx.com/backup"
DATASETS="
zfsdata/sources:storage/backup/sources
"
FAILED=0
FAILED_DATASETS=""
for ENTRY in $DATASETS; do
REMOTE_DS="${ENTRY%%:*}"
LOCAL_DS="${ENTRY##*:}"
syncoid \
--sshport "$REMOTE_PORT" \
--sshoption "Ciphers=$SSH_CIPHER" \
"$REMOTE_USER@$REMOTE_HOST:$REMOTE_DS" \
"$LOCAL_DS"
EXIT="$?"
if [ "$EXIT" -ne 0 ]; then
FAILED=1
FAILED_DATASETS="$FAILED_DATASETS $REMOTE_DS"
fi
done
if [ "$FAILED" -ne 0 ]; then
MESSAGE="Syncoid pull from $REMOTE_HOST failed for datasets:$FAILED_DATASETS"
curl --max-time 30 -s -o /dev/null \
-H "Title: ZFS sync failed on $(hostname)" \
-H "Tags: warning" \
-d "$MESSAGE" \
"$NTFY_URL"
fi
We make this script executable via:
chmod +x /data/scripts/backup/remote_zfs.sh
To execute the script manually, the following command applies:
sudo /data/scripts/backup/remote_zfs.sh
For the automation we create a cron job that should run daily at 01:00.
sudo crontab -e
And add this line:
0 1 * * * /data/scripts/backup/remote_zfs.sh >> /var/log/zfs_remote_snapshot.log 2>&1
Explanation:
- Runs daily at 01:00 (
0 1 * * *) - Runs as root
- Creates a log file for error analysis
If one wants to see the process (state of the data transfer), one can do that with the following command:
watch -n 1 'zfs list storage/backup/media'
That executes the command every second, so that one sees the current state.
:::warning The first remote synchronization can take long depending on the amount of data; one should watch out to deactivate the cron job for this period initially. :::
Server restore
ZFS – local
To see existing snapshots in the file system in the .zfs folder, this must first be activated:
sudo zfs set snapdir=visible zfsdata