I had been thinking of automating my Raspberry Pi backup for a long time but never really got around in doing so. Until, one day the SD card failed and my only backup was from 1 year ago! It was terrible but I rebuilt it this time I rebuilt it onto an old external HDD. You would think that this incident would make me start having a strict backup regime but sadly it didn’t. 6 months later, that external HDD failed! But fortunately, I manage to painfully salvage the data.
Side note: If your pi is constantly booting into emergency mode, you should really consider doing a full backup and ready for it to fail! My pi was booting into emergency mode for a good 3 months before it finally gave up.
My original backup script was a very rudimentary script. As I am booting it off a USB, I was able to boot back into the original SD card and perform a dd and gzip the img of the USB. This works as the USB is small (say 16GB drive) and gzip compresses it down to a few GB of data (depending on how much data there is). But as I move towards external HDD (which gives you way better speeds), dd just doesn’t cut it anymore. That and the original script was very manual process. One had to manually reboot into SD card and run the backup script, come back again in an hour or so and reboot back into USB drive. Hence I started researching and finally came up with the following script.
What this script does is that it perform the following backup to the following locations:
- BackupLocal – This is a large partition in the external HDD (same drive as the rootfs). I have the following in BackupLocal. What it gives me is that now I have daily backups for the past week, then weekly backup for the past 4 weeks followed by 6 months worth of monthly backup. I don’t think I need anything more refined than this at the moment. Although I can possibly have daily backups for the past 6 months without incurring too much data storage space (via hard linking. More on that later).
- backup.daily.1 – backup.daily.6
- backup.weekly.1 – backup.weekly.4
- backup.monthly.1 – backup.monthly.6
- BackupHot – This is another USB flash drive. This drive contains most up to date data and is designed to allow me to quickly swap the boot point to this drive while I source myself a new HDD.
- BackupRemote – This is where I store .tar.gz files from BackupLocal to. This is a remote drive (a NAS unit with RAID 1 configuration) so that in the event the BackupLocal drive fails, I have some data stored safely somewhere else.
This is the backup script. It performs daily backup at 3am everyday. If it is not 3am, it performs a hot backup. If a parameter is added when calling the script (can be anything), it will also perform weekly/monthly rotation if it is Saturday or first day of the month. Daily and hot backups are always done if you call the script manually
#!/bin/bash
# script to synchronise Pi files to backup
# Parameter $1 allows for rotating weekly and monthly records. Used to just in case invoking manual backup
backupLocalDirAwk=backupLocal
backupHotDirAwk=backupHot
backupRemoteDirAwk=backupRemote
backupExcludeList=/home/pi/backup-rsync-exclude.txt
backupLocalDir=/mnt/$backupLocalDirAwk/
backupHotDir=/mnt/$backupHotDirAwk/
backupRemoteDir=/mnt/$backupRemoteDirAwk/
backupDaily="$backupLocalDir"backup.daily.
backupWeekly="$backupLocalDir"backup.weekly.
backupMonthly="$backupLocalDir"backup.monthly.
hourToCreateDailyBackup=03 # needs to be in 24 hour format (00-23)
# ======= CODE STARTS HERE =======
# make sure we're running as root
if (( `id -u` != 0 )); then { echo "Sorry, must be root. Exiting..."; exit; } fi
DOW=$(date +%u)
DOM=$(date +%d)
Hour=$(date +%H) # Hour in 24 hour format. (00-23)
dailyRotated=0
weeklyRotated=0
monthlyRotated=0
invokedByCron=$1
echo "Backup Started " $(date)
# Parameters:
# 1: Directory to tar (without the number suffix)
# 2: Number suffix for the directory to backup
function createTar {
backupTar="$1""$2".tar.gz
if [ -f "$backupTar" ]; then
rm "$backupTar"
fi
tarArgs="-zcp"
if [ -z "$invokedByCron" ]; then
tarArgs="$tarArgs""v"
fi
echo "Creating TAR: " $backupTar
tar "$tarArgs" --file="$backupTar" --directory="$1""$2" .
# Move TAR file to remote location
BACKUPRemote_MOUNTED=$(mount | awk "/$backupRemoteDirAwk/"'{print $6}' | grep "rw")
if [ "$BACKUPRemote_MOUNTED" ]; then
echo "Moving $backupTar to $backupRemoteDir"
mv -f "$backupTar" "$backupRemoteDir"
else
echo "$backupRemoteDir not found or not in rw mode"
fi
# Untar command: tar -zxvpf /filename.tar.gz
}
# Parameters:
# 1: Destination directory
# 2: Linked Directory (Optional. If not specified, no hard linking is done and no timestamp to be noted in backup)
function syncRoot {
rsyncArgs="-axH"
if [ -z "$invokedByCron" ]; then
rsyncArgs="$rsyncArgs""v"
fi
echo "Commencing Backup to " $1
if [ -z $2 ]; then
rsync "$rsyncArgs" --delete-during --delete-excluded --exclude-from="$backupExcludeList" / "$1"
else
rsync "$rsyncArgs" --delete-during --delete-excluded --link-dest="$2" --exclude-from="$backupExcludeList" / "$1"
echo $(date) > "$1"/backupDate.txt
fi
}
BACKUPLocal_MOUNTED=$(mount | awk "/$backupLocalDirAwk/"'{print $6}' | grep "rw")
BACKUPHot_MOUNTED=$(mount | awk "/$backupHotDirAwk/"'{print $6}' | grep "rw")
if [ "$BACKUPLocal_MOUNTED" ]; then
echo "Found $backupLocalDir in rw mode"
if [ ! -z "$invokedByCron" ] && [ "$Hour" = "$hourToCreateDailyBackup" ]; then
# Rotate Monthly backups
if [ "$DOM" = "01" ]; then
echo "Rotating Monthly"
rm -rf "$backupMonthly"6
mv "$backupMonthly"5 "$backupMonthly"6
mv "$backupMonthly"4 "$backupMonthly"5
mv "$backupMonthly"3 "$backupMonthly"4
mv "$backupMonthly"2 "$backupMonthly"3
mv "$backupMonthly"1 "$backupMonthly"2
mv "$backupWeekly"4 "$backupMonthly"1
monthlyRotated=1
fi
# Rotate Weekly backups
if [ "$DOW" = "6" ]; then
echo "Rotating Weekly"
if [ "$monthlyRotated" = 0 ]; then
rm -rf "$backupWeekly"4
fi
mv "$backupWeekly"3 "$backupWeekly"4
mv "$backupWeekly"2 "$backupWeekly"3
mv "$backupWeekly"1 "$backupWeekly"2
mv "$backupDaily"6 "$backupWeekly"1
weeklyRotated=1
fi
fi
# Rotate Daily backups
if [ "$Hour" = "$hourToCreateDailyBackup" ] || [ -z "$invokedByCron" ]; then
echo "Rotating Daily"
if [ "$weeklyRotated" = 0 ]; then
rm -rf "$backupDaily"6
fi
mv "$backupDaily"5 "$backupDaily"6
mv "$backupDaily"4 "$backupDaily"5
mv "$backupDaily"3 "$backupDaily"4
mv "$backupDaily"2 "$backupDaily"3
mv "$backupDaily"1 "$backupDaily"2
mkdir "$backupDaily"1
dailyRotated=1
fi
# Perform backup
if [ "$dailyRotated" = 1 ]; then
echo "Commencing Daily Backup"
syncRoot "$backupDaily"1 "$backupDaily"2
fi
if [ "$BACKUPHot_MOUNTED" ]; then
echo "Found $backupHotDir in rw mode"
echo "Commencing Hot Backup"
syncRoot "$backupHotDir"
fi
# Create TARs
# Create Daily TARs
if [ "$dailyRotated" = 1 ]; then
echo "Creating Daily TARs"
createTar "$backupDaily" 1
fi
# Create Weekly TARs
if [ "$weeklyRotated" = 1 ]; then
echo "Creating Weekly TARs"
createTar "$backupWeekly" 1
createTar "$backupWeekly" 3
fi
# Create Monthly TARs
if [ "$monthlyRotated" = 1 ]; then
echo "Creating Monthly TARs"
createTar "$backupMonthly" 1
createTar "$backupMonthly" 2
createTar "$backupMonthly" 4
createTar "$backupMonthly" 6
fi
echo "Backup Completed " $(date)
echo ""
else
echo "Backup drive not available or not writable" $(date)
fi
And this is the exclude list
#this is the backup-rsync-exclude.txt file /proc/* /sys/* /dev/* /boot/* /tmp/* /run/* /media/* .Trashes ._.Trashes .fseventsd .Spotlight-V100 .DS_Store .AppleDesktop .AppleDB Network Trash Folder Temporary Items .bash_history /etc/fake-hwclock.data /var/lib/rpimonitor/stat/
At the end of all these, you got to create a cron task so that it gets run hourly
# Executes at 3am, 9am, 12pm, 3pm, 6pm everyday to backup system # m h dom mon dow user command 0 3,9,12,15,18 * * * root bash /home/pi/backup.sh 1 >> /home/pi/backup_log.txt