Data backups with Duplicity

• Title: Data backups with Duplicity
• Author(s): Javier Alcazar
• Date: 27 Agosto 2014
• Version(s): 1.0
• Zentyal profiles: All

Duplicity incrementally backs up files and directory by encrypting tar-format volumes with GnuPG and uploading them to a remote (or local) file server. This is the general format of the command:

duplicity [full|incremental] [options] source_directory target_url 

You can installing just typing:

# apt-get install duplicity

In the following examples, we decided to exclude the same directories that where configured as excluded directories by default in Zentyal, but we also included, as example, the /home/user folder. Feel free to reconfigure them based on your needs.

File System Backups

Full copy example:

duplicity full --volsize 25  --exclude=/proc --exclude=/sys  
--exclude=/backups --exclude=/dev --exclude=/proc --exclude=/sys 
--exclude=/mnt --exclude=/media --exclude=/tmp --exclude=/var/spool 
--exclude=/var/cache --exclude=/var/tmp --include=/home/user 
--exclude=/  / file:///backups --no-encryption

/backups is excluded because is the designated destination folder

Incremental copy example:

duplicity incremental --volsize 25  --exclude=/proc --exclude=/sys  
--exclude=/backups --exclude=/dev --exclude=/proc --exclude=/sys 
--exclude=/mnt --exclude=/media --exclude=/tmp --exclude=/var/spool 
--exclude=/var/cache --exclude=/var/tmp --include=/home/user 
--exclude=/  / file:///backups --no-encryption

Checking the collection status:

duplicity collection-status file:///backups --no-encryption

Remove old backups, keeping last n:

duplicity remove-all-but-n-full 2 --force file:///backups --no-encryption

Restoring files from backup:

duplicity --no-encryption --file-to-restore home/user/documents/somefile.txt 
file:///backups /home/user/documents/somefile.txt

In this case, the original source and the destination have the same path, so we will be overwriting the file, but you may prefer to do something like:

duplicity --no-encryption --file-to-restore home/user/documents/somefile.txt 
file:///backups /restore/somefile.txt

Automating the backup

You can script the previous commands in order to easily manage them with cron. Just remember to add execution rights to the scripts (chmod u+x script_file) and include them to /etc/cron.d/backups (for daily use) like:

0 17 15 * * root nice -n 10 /root/script_full 

0 17 1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,19,21,20,22,23,24,25,26,27,28,29,
30,31 * * root nice -n 10 /root/script_incremental

0 23 15 * * root nice -n 10 /root/script_clean_old_backups

 1 .---------------- minutes (00 - 59)
 2 |  .------------- hours (00 - 23)
 3 |  |  .---------- day of month (01 - 31)
 4 |  |  |  .------- month (01 - 12)
 5 |  |  |  |  .---- day of week (0 - 6 where 0 = Sunday)
 6 |  |  |  |  |

   *  *  *  *  *  user  command

This cron configuration will create an incremental backup each day at 17:00, but only full backup on 15th of each month. Also, it will clean all previous backups on 15th at 23:00.

Backups over Rsync

First of all, make sure that the user has permissions to connect using SSH, to achieve this, you will generate a SSH key and then, copy the id to the destination remote machine:

Change to superuser in the CLI

$ sudo su -

Then you generate the keys for the *root* user:

# ssh-keygen

Finally you copy the SSH id to the remote machine:

# ssh-copy-id user@remote.domain.com

Now you can use duplicity with Rsync:

duplicity full --volsize 25 --exclude=/proc --exclude=/sys --exclude=/dev 
--exclude=/proc --exclude=/sys --exclude=/mnt --exclude=/media --exclude=/tmp 
--exclude=/var/spool --exclude=/var/cache --exclude=/var/tmp --include=/home/user 
--exclude=/  / rsync://user@remote.domain.com//path/to/folder --no-encryption

Backup over SCP

Here is an example of how to use duplicity with scp :

duplicity full --volsize 25  --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/proc 
--exclude=/sys --exclude=/mnt --exclude=/media --exclude=/tmp --exclude=/var/spool 
--exclude=/var/cache --exclude=/var/tmp --include=/home/user --exclude=/  
/  scp://user@remote.domain.com//path/to/folder --ssh-askpass --no-encryption

• Make sure that the destination folder has the proper rights in order to be managed by the configured user
• Remember to accept the fingerprint by manually launching the command in order to use scp as duplicity backup method:

The authenticity of host 'localhost' can't be established. 
SSH-RSA key fingerprint is 8a:f3:b0:3a:dd:c3:64:13:20:3a:1c:b6:54:80:18:0e. 
Are you sure you want to continue connecting (yes/no)? Please type 'yes' or 'no': yes 
Local and Remote metadata are synchronized, no sync needed. 
Last full backup date: none 
--------------[ Backup Statistics ]-------------- 
StartTime 1409052568.46 (Tue Aug 26 13:29:28 2014) 
EndTime 1409052571.85 (Tue Aug 26 13:29:31 2014) 
ElapsedTime 3.39 (3.39 seconds) 
SourceFiles 225 
SourceFileSize 54620986 (52.1 MB) 
NewFiles 225 
NewFileSize 54620986 (52.1 MB) 
DeletedFiles 0 
ChangedFiles 0 
ChangedFileSize 0 (0 bytes) 
ChangedDeltaSize 0 (0 bytes) 
DeltaEntries 225 
RawDeltaSize 54280963 (51.8 MB) 
TotalDestinationSizeChange 23693038 (22.6 MB) 
Errors 0 
-------------------------------------------------