Educational ICT Virtualisation Specialist

Twitter LinkedIn E-mail
Precedence Technologies Ltd
Technology House, 36a Union Lane
Cambridge, CB4 1QB, United Kingdom
T: +44 (0)8456 446 800 / +44 (0)1223 359900
F: +44 (0)8456 446 899 / +44 (0)1223 359459
E: enquiries@precedence.co.uk
Runrsync

Jump To: Support > KB > Backup > Runrsync

NetManager backup server overview

Jump to: Backup structure | Backup log file | Excluding files and folders

Multi-generation efficient file copies

NetManager Backup uses rsync to copy files from the source. rsync copies only files that have changed and, when possible, only the parts of the files that differ. Files are pulled by the NetManager backup at a scheduled time; the server being backed up does not push the data to the backup server. This means that servers to be backed up need to run a backup service. Communication between the backup server and the device being backed up can be compressed and certain files can be skipped. The backup server uses a username and password to identify itself to the device being backed up. The username and password should be the same throughout your network, but this is separate from all other users (i.e. this is not an Active Directory or NetManager user). If some of the servers to be backed up are outside of your administrative control, you can specify an alternative username and password for specific servers.

The runrsync command performs the backup and if run without any options will backup all configured shares on all servers. The syntax at the time of writing is:

$Revision: 1.59 $
Syntax: runrsync [-ehlLnvqsrRot] [-m <email address>] <service ...>
        -d = output log to stdout once backup completed (useful for email
             from cron)
        -e = update and edit configuration file, then quit
        -h = this help
        -l = list shares to be backed up, then quit
        -L = list all files, but don't copy
        -m = Email log file to specified or pre-configured address.
             This option requires the server to be able to email via an
             SMTP server.  You can specify this option more than once
             to send to multiple email addresses
        -n = do a dry-run, but don't actually copy or delete files
        -o = backup to remote server over ssh
             (uses /etc/netmanager/rsync.conf)
        -p = remove previous folders over a given number of days old
        -P = as -p but quit without running backup
        -r = restart transfer on data timeout
        -R = number of times to restart (default 20, 0 = for ever)
        -s = back up to local machine
             (uses /etc/netmanager/localbackup.conf)
        -S = copy symbolic links as files (only for settings share)
        -t = timeout setting in seconds
             (default 45 (300 if -o), 0 = no timeout)
        -v = lists all files (adding more increases diagnostics)

service names are in the format "share@server"

runrsync syncs to configured list of rsync shares

The runrsync can run in 3 distinct modes (backup from remote servers to this server, backup this server locally and backup this server to remote server) and the configuration file for each is different, but they share the same general syntax.

Backup from remote servers

The file /etc/netmanager/runrsync.conf is used to configure the operation when backing up remote servers (this is the default mode when neither of the -s and -o options are given). When runrsync is first run, a blank configuration file will be created ready for customisation. The file is well-commented (lines beginning with # are comments). An example file is below:

# Created by runrsync : 1.59 $
#
# Username to use to connect to rsync server
# This is username defined by the server just for rsync (i.e. not a local user)
user="backupuser"

# Password to use to connect to rsync server
# This is set in the rsyncd.secrets file
password="mypassword"

# Username and password can be overridden for certain servers
# Format is space-separated list of servername=user:password
# Any server not listed will use default user and password above
useroverride=""

# Shares and servers to backup
# Specify just the server name to backup all available rsync shares
# or use share@server to backup particular ones.
shares="netmanager fileserver"

# Local directory to backup to
dest="/data/backup"

# How to handle previous versions of files. If unset (not empty), a relative
# folder called 'previous' will used (for compatibility). Leave empty to
# generate no previous versions. Paths starting with / are relative to dest
# set above and have the server/share name added.
# "/%DAY" or "backup" are the recommended values.
# There are a number of substitutions that will be made:
# Use %DAY for the day of the week
# Use %MONTH for the month as two digits (e.g. 01)
# Use %YEAR for the year as four digits (e.g. 2012)
# Use %WEEK for the week of the year
# Use %WEEK2 for alternate weeks (i.e. 1 or 2 - allows fortnightly backups)
# Use %WEEK4 for a 4-week cycle (i.e. 1, 2, 3 or 4 - allows 28-day backups)
# e.g. /%DAY-%WEEK2 will give 14 backups with names Mon-1, Tue-1 through to
# Sun-2.
# Use %DATE for the actual date (2010-06-13)
# Substitutions can be combined (e.g. /previous/%YEAR/%DATE)
previous="/%DATE"

# Time to wait with no activity before aborting the backup in seconds.
# Used to spot a failed connection, but should not be too short as otherwise
# the scanning phase of the backup can cause a timeout.
# Default = 45, but should be increased if very large files or number of files
# on server being backed up
# Set to 0 to disable timeout
timeoutsecs=300

# Number of days to keep previous version of files for. Only makes sense if
# 'previous' setting above has %DATE in it. Age will be calculated from last
# successful backup rather than current date
# Set to 0 or leave empty to not expire at all
expire=90

# Delete files from the backup that no longer exist? (Default: yes)
delete="y"

# File extensions to skip (space separated)
skip=""

# Email address to email log file to
email=""

# Compress transfer?
compress="n"

This configuration backs up all configured shares (configured in the backup agent, not standard Windows shares) on the servers called netmanager and fileserver.

Correct operation can checked by using the -l flag:

backup 1# runrsync -l
home@netmanager
settings@netmanager
Users@fileserver
Common@fileserver
backup 2#

Here we can see that the 2 servers each have 2 shared resources being backed up (earlier version of runrsync will show this as home on netmanager, etc. This has been changed so that the lines show exactly what you can specify on the command line to backup just that share.

Backup structure

After the backup has been run a few times, the following structure will be seen:

The backup structure is split between the Live backup (which will reflect the state when the backup was last run, i.e. usually last night) and the Previous versions. For the Live backups, each server has its own folder named after that server. The Previous versions are all in dated folders underneath a folder called previous. To understand how the live and previous versions work, it is useful to know how the backup process works.

When the backup runs, all folders and files will be compared between the server being backed up and versions in the Live backup. If files have been changed or deleted, the old version will be moved to a folder named after the current date (the path within that dated folder will reflect the location the file exists at in the live version, so for example if the live file is fileserver/shared/Document.odt, the old version will be moved to previous/YYYY-MM-DD/fileserver/shared/Document.odt). The live version will then be updated from the current version of the file (or removed if the file has been deleted).

As files only get moved to the dated Previous folders when they change, files that do not change will not be in the Previous folders (and also therefore will take up no more space). The size of a dated Previous folder represents the size of the files changed within that day (assuming backups are run nightly).

Backup log file

Each dated previous folder contains a report of the status of the backup on that date called backuplog.txt. The lastrun.txt file at the top level is actually a link to the most recent backuplog.txt file. An example extract is:

Last backup run started: Tue Nov 30 08:41:58 GMT 2010

Backup    : home on netmanager
--------------------------------------------------
Started   : Tue Nov 30 08:41:58 GMT 2010
Completed : Tue Nov 30 08:41:59 GMT 2010
Backup was **successful**

Previous versions backed up to: /2010-11-30

Number of files: 9588
Number of files transferred: 0
Total file size: 1.35G bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 191.40K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 3.40K
Total bytes received: 202.14K

sent 3.40K bytes  received 202.14K bytes  137.02K bytes/sec
total size is 1.35G  speedup is 6563.23

Excluding files and paths

For a blanket block on certain file types, just add the file extensions to the skip line (e.g. skip="mp3 avi mpg").

For blocking paths and files use the following files:

  • /etc/netmanager/backup.exclude - global for all backup types
  • /etc/netmanager/localbackup.exclude - additional exclusions with -s option (backup to local storage)
  • /etc/netmanager/externalbackup.exclude - additional exclusions with -o option (backup to remote external system)

Each line contains a case-sensitive match against the root of the files in the share. Directory separator is /, not \. Wildcard can be specified with * (lines do not contain regular expressions).

So, for example, to block all .mp3 files you can use:
*/*.mpg
To block all directories with names containing Old use:
*/*Old*/*

Lines to act on a particular server should be prefixed with [server], lines to act on a particular share on a particular server should be prefixed with [share@server]. All other lines will act on all servers. For example, the following will skip all mp3 files, but only skip .avi files on the server called resources:

*/*.mp3
[resources]*/*.avi

NFS

Configuration of NFS storage will be specific to the site.

SMB/CIFS

Configuration of SMB/CIFS storage will be specific to the site.

© Copyright Precedence Technologies 1999-2017
Page last modified on March 06, 2017, at 11:32 AM by sborrill