Incremental snapshot backups via rsync and ssh

In follow-up to the previous post, I am compiling this as a separate post as this solution is been running very stable for a while with quite a few updates and changes...

I will be setting up a back-up of a remote web-host via rsync over ssh and creating the snapshot style backup on the local machine.

The backups are done incremental, only the files that have changed are backed up so there is very less bandwidth used during the backup and also does not cause any load on the server.

These are sliced backups, meaning that you get a full backup of the last 4 days, and the last 4 weeks. So data can be restored for upto a month of back date.

Below is an example listing of backups you would see.

Mar 11 - daily.0
Mar 10 - daily.1
Mar 9 - daily.2
Mar 8 - daily.3
Mar 5 - weekly.0
Feb 27 - weekly.1
Feb 20 - weekly.2
Feb 13 - weekly.3

Each of those is a full snapshot for the particular day/week. The files are all hard-linked and would only require 2 to 3 times the space used on the server. The backups should consist of web, database, email and some of the important server configuration files.

On the local backup machine:

  1. Generate the private/public key pair for passwordless login via ssh to remote host.
    # ssh-keygen -t dsa -b 2048 -f ~/.ssh/rsync_key
  2. Create an empty password by just hitting enter when prompted for password.
  3. Protect the private key, which should only be readable by the owner.
    # chmod 600 ~/.ssh/rsync_key
  4. Transfer the public key file to the remote host:
    # scp ~/.ssh/rsync_key.pub remote_host.tld:.
  5. The backups are on separately partitioned drive and auto-mounted read-only.

    In "/etc/auto.master"

    /data/bak   /etc/auto.bak

    In "/etc/auto.bak"

    .snapshots      -fstype=ext3,ro,nosuid,noexec   :/dev/hdb2-vg00/lv-bak

  6. Restart autofs:
    # service autofs restart
  7. Create the rsync backup script to run via cron. The script also rotates out the backups creating a snapshot style backup:
    #!/bin/bash
    # do_rsync.sh
    # usage: ./do_rsync.sh {client}

    RM=/bin/rm
    TOUCH=/bin/touch
    NICE=/bin/nice
    CHOWN=/bin/chown
    CHMOD=/bin/chmod
    RSYNC=/usr/bin/rsync
    SSH=/usr/bin/ssh
    MOUNT=/bin/mount
    UMOUNT=/bin/umount
    KEY=/root/.ssh/rsync_keys

    CLIENT=$1
    EXPECTED_ARGS=1
    NUM_ARGS=$#
    CONFIG_PATH=/root/scripts/snapshots/config
    CONFIG_FILE=${CONFIG_PATH}/${CLIENT}/conf.txt
    EXCLUDE_FILE=${CONFIG_PATH}/${CLIENT}/exclude.txt
    LPATH=/home/virtual/${CLIENT}.DOMAIN.TLD/home/${CLIENT}/bak/.snapshots
    PID_FILE=/var/run/${CLIENT}_rsync.pid

    # Check the number of args
    badargs() {
      E_BADARGS=65
      if [ "$NUM_ARGS" -ne "$EXPECTED_ARGS" ]
      then
        echo "Usage: `basename $0` {client}"
        exit $E_BADARGS
      fi
    }

    # Include conf
    include_conf() {
      if [ -f "${CONFIG_FILE}" ]; then
        # source the config file
        . ${CONFIG_FILE}
      else
        echo "ERROR: ${CONFIG_FILE}" does not exist.
        exit 1
      fi
    }

    # Mount for writing
    mount_write() {
      $MOUNT -o remount,rw,nosuid,noexec,nodev $LPATH
      if (( e = $? )); then
        echo "ERROR: $e, Could not remount in ReadWrite mode. Exiting..."
        exit $e
      fi
    }

    # rotate backups
    rotate_backups() {
      if [ -f daily.0/.rsync_bak_complete ]
      then
        if [ "`date +%w`" -eq 0 ]
        then
          [ -d weekly.3 ] && rm -rf weekly.3
          [ -d weekly.2 ] && mv weekly.2 weekly.3
          [ -d weekly.1 ] && mv weekly.1 weekly.2
          [ -d weekly.0 ] && mv weekly.0 weekly.1
          [ -d daily.3 ] && mv daily.3 weekly.0
        fi
        [ -d daily.3 ] && rm -rf daily.3
        [ -d daily.2 ] && mv daily.2 daily.3
        [ -d daily.1 ] && mv daily.1 daily.2
        [ -d daily.0 ] && mv daily.0 daily.1
      fi
      if [ ! -d daily.0 ]; then
        mkdir daily.0
        # Adjust permissions
        $CHMOD 050 daily.0
        $CHOWN root:${GID} daily.0
      fi
    }


    # run the rsync
    run_rsync() {
      $NICE -n 12 $RSYNC -azR -e "$SSH -i $KEY -p $PORT" --exclude-from=${EXCLUDE_FILE} --delete --timeout=1800 --link-dest=../daily.1 $RUSER@$RHOST:"$RPATH" $LPATH/daily.0

      # error 23 = partial transfer, probably temp files
      # error 24 = file vanished
      if (( e = $? )); then
        if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
          echo "ERROR: $e, Rsync failed. Exiting..."
          post_sync $e
        fi
      fi

      # Confirm backup is complete so files are rotated on the backup side
      $RSYNC -a -e "$SSH -i $KEY -p $PORT" --timeout=10 $RUSER@$RHOST:/.rsync_bak_complete $LPATH/daily.0

      if (( e = $? )); then
        if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
          echo "ERROR: $e, Rsync CONFIRMATION failed."
          echo "Check backup completion. Exiting..."
          post_sync $e
        fi
      fi
    }


    # Remount Read Only
    mount_read() {
      $MOUNT -o remount,ro,nosuid,noexec $LPATH
      if (( e = $? )); then
        echo "ERROR: $e, Could not remount in ReadOnly mode."
        echo "Check your mounts. Exiting..."
        exit $e
      fi
    }

    # Unmount
    unmount() {
      $UMOUNT $LPATH
      if (( e = $? )); then
        echo "ERROR: $e, Could not unmount ${LPATH}."
        echo "Check your mounts. Exiting..."
        exit $e
      fi
    }

    # Check if the PID is running
    check_pid() {
      if [ -f "$PID_FILE" ]
      then
        MYPID=`head -n 1 "$PID_FILE"`
        TEST_RUNNING=`ps -p ${MYPID} | grep ${MYPID}`
        if [ -z "${TEST_RUNNING}" ]
        then
          echo "PID file exists but PID [$MYPID] is not running... creating new PID file [$PID_FILE]"
          echo $$ > "$PID_FILE"
        else
          echo "`basename $0` is already running [${MYPID}]... quitting"
          exit -1
        fi
      else
        echo "`basename $0` not running... creating new PID file [$PID_FILE]"
        echo $$ >> "$PID_FILE"
      fi
    }

    # Post Sync
    post_sync() {
      popd
      sleep 10
      unmount
      exit $1
    }

    # Pre Sync
    pre_sync() {
      check_pid
      pushd $LPATH
      sleep 10
      mount_write;
      rotate_backups;
    }

    # Main
    badargs;
    include_conf;
    pre_sync;
    run_rsync;
    post_sync 0;

    exit 0

    There is a configuration and an exclude file, conf.txt and exclude.txt .

    "conf.txt" includes the details of the server that need to be backed up.

    PORT=22
    RUSER=root
    RHOST=DOMAIN.TLD
    RPATH='/etc /var/lib/mysql /var/www /var/spool/mail /var/spool/squirrelmail'
    GID=admin138

    "exclude.txt" file has the list of files to be excluded from being backed up.

    # exclude patterns (one per line).
    #/var/www/vhosts/*/statistics/logs/*
    /var/www/*/log

  8. Change the mode to be executable on the script.
    # chmod 750 /usr/local/bin/rsync_backup.sh
  9. Add the script to crontab for a daily incremental snapshot style backup.
    # Do rsync backup
    07 06 * * * /usr/local/bin/rsync_backup.sh > /dev/null 2>&1
  10. Also add the below line to "/etc/fstab" to auto fsck and recover journals on boot.

    /dev/hdb2-vg00/lv-bak  /mnt/lv-bak     ext3    noauto,nodev,noexec,nosuid        0 2

    This also allowed me to do quick read-write mounts when needed.

On the remote host:

  1. Create script to validate the rsync command used.
    #!/bin/sh
    # rsync_validate.sh

    case "$SSH_ORIGINAL_COMMAND" in
    *\&*)
    echo "Rejected"
    ;;
    *\(*)
    echo "Rejected"
    ;;
    *\{*)
    echo "Rejected"
    ;;
    *\;*)
    echo "Rejected"
    ;;
    *\<*)
    echo "Rejected"
    ;;
    *\`*)
    echo "Rejected"
    ;;
    rsync\ --server*)
    $SSH_ORIGINAL_COMMAND
    ;;
    *)
    echo "Rejected"
    ;;
    esac
  2. Change the mode to be executable on the script.
    # chmod 750 /usr/local/bin/rsync_validate.sh
  3. Modify the public key and prepend restricted host and command used, separated by just a comma and no spaces.
    from="backup_host.tld",command="/usr/local/bin/rsync_validate.sh" ssh-dss AAAAB3NzaC1kc3MAAA...
  4. Append the public key to authorized_keys2 file.
    # cat ~/rsync-key.pub >> ~/.ssh/authorized_keys2
  5. Modify "/etc/ssh/sshd_config" to accept forced-commands-only for root ssh login.
    PermitRootLogin forced-commands-only
  6. Restart ssh:
    # service sshd restart

That should do it... the result will be an incremental backup of the last 4 days plus a weekly of the month.

Comment