Easy Automated Snapshot-Style Backups with Linux and Rsync

This document describes a method for generating automatic rotating "snapshot"-style backups on a Unix-based system, with specific examples drawn from the author's GNU/Linux experience. Snapshot backups are a feature of some high-end industrial file servers; they create the illusion of multiple, full backups per day without the space or processing overhead. All of the snapshots are read-only, and are accessible directly by users as special system directories. It is often possible to store several hours, days, and even weeks' worth of snapshots with slightly more than 2x storage. This method, while not as space-efficient as some of the proprietary technologies (which, using special copy-on-write filesystems, can operate on slightly more than 1x storage), makes use of only standard file utilities and the common rsync program, which is installed by default on most Linux distributions. Properly configured, the method can also protect against hard disk failure, root compromises, or even back up a network of heterogeneous desktops automatically.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

re:

Excellent. Thanks for sharing this informative articles. I found another good articles about linux manual. You can check it out at pdfph.com.

digitalcatalogue

regard, very usefull and thankyou ..

Incremental snapshot backups via rsync and ssh

I will be setting up a back-up of a remote web-host via rsync over ssh and creating the snapshot style backup as mentioned above on the local machine.

On the local backup machine:

  1. Generate the private/public key pair for passwordless login via ssh to remote host.
    # ssh-keygen -t dsa -b 2048 -f ~/.ssh/rsync_key
    
  2. Create an empty password by just hitting enter when prompted for password.
  3. Protect the private key, which should only be readable by the owner.
    # chmod 600 ~/.ssh/rsync_key
    
  4. Transfer the public key file to the remote host:
    # scp ~/.ssh/rsync_key.pub remote_host.tld:.
    
  5. The backups are on separately partitioned drive and auto-mounted read-only.

    In "/etc/auto.master"

    /data/bak   /etc/auto.bak
    

    In "/etc/auto.bak"

    .snapshots      -fstype=ext3,ro,nosuid,noexec   :/dev/hdb2-vg00/lv-bak
    
  6. Restart autofs:
    # service autofs restart
    
  7. Create the rsync backup script to run via cron. The script also rotates out the backups creating a snapshot style backup:
    #!/bin/bash
    # rsync_backup.sh
    
    NICE=/bin/nice
    CHOWN=/bin/chown
    CHMOD=/bin/chmod
    RSYNC=/usr/bin/rsync
    SSH=/usr/bin/ssh
    MOUNT=/bin/mount
    UMOUNT=/bin/umount
    PORT=22
    KEY=~/.ssh/rsync_key
    RUSER=root
    RHOST=remote_host.tld
    RPATH='/etc /var/lib/mysql /var/www'
    LPATH=/data/bak/.snapshots
    
    
    # Mount for writing
    $MOUNT -o remount,rw,nosuid,noexec,nodev $LPATH
    
    
    # rotate backups
    pushd $LPATH
    
    if [ -f daily.0/.rsync_bak_complete ] 
    then
    
            if [ "`date +%w`" -eq 0 ] 
            then 
                    [ -d weekly.3 ] && rm -rf weekly.3
                    [ -d weekly.2 ] && mv weekly.2 weekly.3
                    [ -d weekly.1 ] && mv weekly.1 weekly.2
                    [ -d weekly.0 ] && mv weekly.0 weekly.1
                    [ -d daily.3 ] && mv daily.3 weekly.0
            fi
    
            [ -d daily.3 ] && rm -rf daily.3
            [ -d daily.2 ] && mv daily.2 daily.3
            [ -d daily.1 ] && mv daily.1 daily.2
            [ -d daily.0 ] && mv daily.0 daily.1
    
            mkdir daily.0
    
            # Adjust permissions
            $CHMOD 050 daily.0
            $CHOWN root:remote_user daily.0
    
    fi
    
    # run the rsync
    $NICE -n 12 $RSYNC -azR -e "$SSH -i $KEY -p $PORT" --exclude=/var/www/*/log --delete \
          --timeout=1800 --link-dest=../daily.1 $RUSER@$RHOST:"$RPATH" $LPATH/daily.0
    
    # Confirm backup is complete so files are rotated on the backup side
    $RSYNC -a -e "$SSH -i $KEY -p $PORT" --timeout=10 $RUSER@$RHOST:/.rsync_bak_complete $LPATH/daily.0
    
    
    popd
    
    # Unmount
    $UMOUNT $LPATH
    
    exit 0
    
  8. Change the mode to be executable on the script.
    # chmod 750 /usr/local/bin/rsync_backup.sh
    
  9. Add the script to crontab for a daily incremental snapshot style backup.
    # Do rsync backup
    07 06 * * * /usr/local/bin/rsync_backup.sh > /dev/null 2>&1
    

On the remote host:

  1. Create script to validate the rsync command used.
    #!/bin/sh
    # rsync_validate.sh
    
    case "$SSH_ORIGINAL_COMMAND" in
    *\&*)
    echo "Rejected"
    ;;
    *\(*)
    echo "Rejected"
    ;;
    *\{*)
    echo "Rejected"
    ;;
    *\;*)
    echo "Rejected"
    ;;
    *\<*)
    echo "Rejected"
    ;;
    *\`*)
    echo "Rejected"
    ;;
    rsync\ --server*)
    $SSH_ORIGINAL_COMMAND
    ;;
    *)
    echo "Rejected"
    ;;
    esac 
    
  2. Change the mode to be executable on the script.
    # chmod 750 /usr/local/bin/rsync_validate.sh
    
  3. Modify the public key and prepend restricted host and command used, separated by just a comma and no spaces.
    from="backup_host.tld",command="/usr/local/bin/rsync_validate.sh" ssh-dss AAAAB3NzaC1kc3MAAA...
    
  4. Append the public key to authorized_keys2 file.
    # cat ~/rsync-key.pub >> ~/.ssh/authorized_keys2
    
  5. Modify "/etc/ssh/sshd_config" to accept forced-commands-only for root ssh login.
    PermitRootLogin forced-commands-only
    
  6. Restart ssh:
    # service sshd restart
    

That should do it... the result will be an incremental backup of the last 4 days plus a weekly of the month.

rsync backup shell script

I have since modified the rsync_backup.sh script and made it a bit modular with some amount of error checking:

#!/bin/bash
# do_rsync.sh
# usage: ./do_rsync.sh {client}

RM=/bin/rm
TOUCH=/bin/touch
NICE=/bin/nice
CHOWN=/bin/chown
CHMOD=/bin/chmod
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
MOUNT=/bin/mount
UMOUNT=/bin/umount
KEY=/root/.ssh/rsync_keys

CLIENT=$1
EXPECTED_ARGS=1
NUM_ARGS=$#
CONFIG_PATH=/root/scripts/snapshots/config
CONFIG_FILE=${CONFIG_PATH}/${CLIENT}/conf.txt
EXCLUDE_FILE=${CONFIG_PATH}/${CLIENT}/exclude.txt
LPATH=/home/virtual/${CLIENT}.DOMAIN.TLD/home/${CLIENT}/bak/.snapshots


# Check the number of args
badargs() {
        E_BADARGS=65

        if [ "$NUM_ARGS" -ne "$EXPECTED_ARGS" ]
        then
                echo "Usage: `basename $0` {client}"
                exit $E_BADARGS
        fi
}

# Include conf
include_conf() {
        if [ -f "${CONFIG_FILE}" ]; then
                # source the config file
                . ${CONFIG_FILE}
        else
                echo "ERROR: ${CONFIG_FILE}" does not exist.
                exit 1
        fi
}


# Mount for writing
mount_write() {
        $MOUNT -o remount,rw,nosuid,noexec,nodev $LPATH
        if (( e = $? )); then
                echo "ERROR: $e, Could not remount in ReadWrite mode. Exiting..."
                exit $e
        fi

}

# rotate backups
rotate_backups() {
        if [ -f daily.0/.rsync_bak_complete ]
        then

                if [ "`date +%w`" -eq 0 ]
                then
                        [ -d weekly.3 ] && rm -rf weekly.3
                        [ -d weekly.2 ] && mv weekly.2 weekly.3
                        [ -d weekly.1 ] && mv weekly.1 weekly.2
                        [ -d weekly.0 ] && mv weekly.0 weekly.1
                        [ -d daily.3 ] && mv daily.3 weekly.0
                fi

                [ -d daily.3 ] && rm -rf daily.3
                [ -d daily.2 ] && mv daily.2 daily.3
                [ -d daily.1 ] && mv daily.1 daily.2
                [ -d daily.0 ] && mv daily.0 daily.1
        fi

        if [ ! -d daily.0 ]; then

                mkdir daily.0

                # Adjust permissions
                $CHMOD 050 daily.0
                $CHOWN root:${GID} daily.0

        fi
}


# run the rsync
run_rsync() {
        $NICE -n 12 $RSYNC -azR -e "$SSH -i $KEY -p $PORT" --exclude-from=${EXCLUDE_FILE} --delete --timeout=1800 --link-dest=../daily.1 $RUSER@$RHOST:"$RPATH" $LPATH/daily.0

        # error 23 = partial transfer, probably temp files
        # error 24 = file vanished
        if (( e = $? )); then
                if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
                        echo "ERROR: $e, Rsync failed. Exiting..."
                        post_sync $e
                fi
        fi

        # Confirm backup is complete so files are rotated on the backup side
        $RSYNC -a -e "$SSH -i $KEY -p $PORT" --timeout=10 $RUSER@$RHOST:/.rsync_bak_complete $LPATH/daily.0

        if (( e = $? )); then
                if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
                        echo "ERROR: $e, Rsync CONFIRMATION failed."
                        echo "Check backup completion. Exiting..."
                        post_sync $e
                fi
        fi
}


# Unmount
unmount() {
        $UMOUNT $LPATH
        if (( e = $? )); then
                echo "ERROR: $e, Could not unmount ${LPATH}."
                echo "Check your mounts. Exiting..."
                exit $e
        fi
}

# Post Sync
post_sync() {
        $RM -rf .rsync_start
        popd
        sleep 10
        unmount
        exit $1
}

# Pre Sync
pre_sync() {
        pushd $LPATH
        sleep 10
        if [ -f .rsync_start ]; then
                echo "Rsync still running... Exiting!"
                exit 1;
        fi
        mount_write;
        rotate_backups;
        $TOUCH .rsync_start
}

# Main
badargs;
include_conf;
pre_sync;
run_rsync;
post_sync 0;

There is a configuration and an exclude file, conf.txt and exclude.txt .

"conf.txt" includes the details of the server that need to be backed up.

PORT=22
RUSER=root
RHOST=DOMAIN.TLD
RPATH='/etc /var/lib/mysql /var/www /var/spool/mail /var/spool/squirrelmail'
GID=admin138

"exclude.txt" file has the list of files to be excluded from being backed up.

# exclude patterns (one per line).
#/var/www/vhosts/*/statistics/logs/*
/var/www/*/log

Problem with crashed mountpoints

When the server crashed and can back up, the mountpoints would fail to automount as the filesystem needed manual repair via fsck.

So, also added the below line to "/etc/fstab" to auto fsck and recover journals on boot.

/dev/hdb2-vg00/lv-bak  /mnt/lv-bak     ext3    noauto,nodev,noexec,nosuid        0 2

This also allowed me to do quick read-write mounts when needed.

Comment