A HPC account or HPC Data Service account provides a facility to save offsite copies of research data.

• HPC accounts will connect to an HPC system using their HPC/UQsignin account name.
• Data Service accounts will connect to "cirrus" using the Service account user name.
  

Note for security reasons, all HPC systems will detect a rapid sequence of connections will be as a potential hacking intrusion and result in the source address being temporarily blocked. Clients must use a single connection session or delay reconnections.

The HPC service may be used in several ways to either

• maintain a single up-to-date copy of data directories
  
• maintain several generation of file versions within directories
• backup data filesystems

 ITS provide several examples and scripts to assist clients.

Maintaining a single offsite copy

The "rsync" program provides a simple mechanism to create and update a remote copy. Versions of "rsync" are available for Windows as well as Linux and Mac.

rsync -a -e'ssh hpcoper@cirrus' /local/path/to/data cirrus:/backup/data

Maintaining several generations of file versions

Several client programs are available that can assist in maintaining multiple generations of files.

Linux - Simple Backup Suite (sbackup)

This program is relatively new, but provides many useful features.

The program may run as a system daemon/service for scheduled backups, or manually from the GUI configuration tool. It will save both full and incremental backups efficently to save set files and provide good options form managing old backups. A good file recovery tool.

Sbackup will connect to the HPC systems using SSH, but currently requires passwords within the configuration files, which is insecure and inappropriate on shared computers.

Linux - fwbackup

The "fwbackup" program, included in many Linux distributions, will connect to the HPC system using SSH/Sftp.

This GUI program is easy to use and reliable. It may schedule backups, and save these efficiently to compressed archive files and help remove older versions.

The efficient (compressed archive) save does not support incrementals.   

Rsync

The "rsync" program supports mechanisms to retain older versions of data.

The ITS server side script "cycle" supports these mechisms.

The example below creates directories with save date names and deletes versions older than 60 days.

ssh hpcoper@cirrus cycle -d -p 60 saveset_name

It will prompt with the corresponding rsync command to perform the save. Typically.

rsync -a -e'ssh hpcoper@cirrus' --delete --link-dest=saveset_name.last /local/path/to/data cirrus:saveset_name

Backing up a research data filesystem

Some examples are provided below.

ITS is developing a client side script to help manage this usage. Please contact us for the latest version.

Saving a filesystem dump (using xfsdump)

Example 1 - A regular dump

/usr/sbin/xfsdump -a -F -l 0 - \

| ssh hpcoper@cirrus "dd of=xfsdump_mnt_sw_20091008"

Creating file lists from an xfssdump

ssh hpcoper@cirrus "dd if=xfsdump_mnt_sw_20091008" | xfsrestore -t - > file_list

 ssh hpcoper@cirrus "dd if=xfsdump_mnt_sw_20091008" \

| xfsrestore -t - |ssh hpcoper@cirrus "dd of=filelist_mnt_sw_20091008"

Recovering files from an xfsdump

Note that xfsrestore file paths are relative to mount point. A restore will reconstruct the whole relatiive directory path, thus it is often preferable to restore files into a temporary directory. 

Example 1 - recovering a known file or subdirectory

ssh hpcoper@cirrus "dd if=xfsdump_mnt_sw_20091008" \

| xfsrestore -s apps/Matlab_2008a/sys/java/jre/glnxa64/jre.cfg - restore

Example 2 - Interactive browse and restore

ssh hpcoper@cirrus "dd if=xfsdump_mnt_sw_20091008" |/usr/sbin/xfsrestore -i - /tmp