I've looked for incremental backups that don't require root before, but I finally found it. Similar to rdiff-backup is a program called duplicity. It uses tar, gpg, and the rsync algorithm to store encrypted differential backups.

duplicity needs to be scripted a little bit to automate the backups. I came up with a script that will backup certain directories fully once a week and incrementally the rest of the week. I also disabled the encryption since the transfer method is secure (ssh in my case) and the end server is trusted. If you were using Amazon's S3 service or a public FTP share you might want to tweak this a bit.

/etc/cron.d/backup

32 3 * * 1-7 root /usr/local/sbin/backup.sh cmd=full
32 3 * * 0 root /usr/local/sbin/backup.sh

/usr/local/sbin/backup.sh

#!/bin/bash

[ "$v" = "" ] && v="-v0 --no-print-statistics"
[ "$cmd" = "" ] && cmd=""
duplicity="duplicity"
opt="--no-encryption $v"
dest="scp://taonas@thor.home.silfreed.net"
maxage="1M"

dcmd_backup="$duplicity $cmd $opt"
dcmd_age="$duplicity remove-older-than $maxage $opt"

# backups
$dcmd_backup /etc $dest/etc
$dcmd_backup /var/log $dest/var-log
$dcmd_backup /home \
        --exclude /home/silfreed/tmp \
        --exclude /home/silfreed/src/silfreednet/tmp \
        --exclude /home/silfreed/src/workspace \
        --exclude /home/silfreed/src/mozdev/workspace \
        $dest/home

mysqldump_dir=/tmp/mysqldump
mkdir $mysqldump_dir &&
        chmod 700 $mysqldump_dir &&
        mysqldump -u root -A > $mysqldump_dir/mysqldump.sql &&
        $dcmd_backup $mysqldump_dir $dest/mysql
rm -rf $mysqldump_dir

# age out paths
for path in /etc /var-log /home /mysql; do
        $dcmd_age $dest$path
done

I've been using rdiff-backup for several windows computers recently and it has worked out well. I started looking into using it for linux, but ran into a small problem.

Transfers of the backups between computers assumes root access on the destination.

This is a big problem for me; there's no reason to need root access on the destination server to save a backup of one system to another. The reason it's needed is to preserve UIDs and GIDs and set arbitrary attributes.

I don't have this problem with my tar-based backups.

Ideally, I'd like a solution that provides:

1. Differential backups - only backup what has changed
2. Incremental backups - backups of the same directories only records what's changed between runs
3. Preserves permissions in some kind of manifest - this way I don't need to setup root ssh keys between servers
4. (optional) compresses the backup

So lazyweb, do you know of this solution?

For some years now I've relied on a script written by Lindsay Snider while at PA.net for all my backup needs. A year or so ago a presentation was done at CPLug that described the tool, and it was agreed that the script should become public somewhere so people can use it.

It wasn't really clear whose job it was going to be to setup the project publically, so it kinda just was forgotten. I was recently looking into backup tools again and decided it was time to get the tool out there and integrate my local changes into a canonical copy.

So now we have the remotebackup project with a public SVN tree.

Feel free to send any questions/comments/ideas my way for now while we work out the structure of who will be working on the project.