Data Transfer

This page describes a number of data transfer options:

• methods, respectively tools
• file server addresses for those tools
• brief examples

For details of the file system and practical considerations please refer to the page. For an even more practical approach and more information please consider following an introductory course.

Addresses to connect with ssh/scp#

In order to contact our HPC systems with ssh (used by scp and rsync) you need to provide an address; these are identical to the ones described here:

• miil01.zdv.uni-mainz.de,
• miil02.zdv.uni-mainz.de or
• miil03.zdv.uni-mainz.de,
• resp. miil01, miil02 or miil03 from within the university net for MOGON II.

Copying files with scp#

scp stands for secure copy and works with ssh (the secure shell). To use scp for data transfer at this time, you must tunnel through the HPCGATE. The basic command would be:

scp -oProxyJump=<username>@hpcgate.zdv.uni-mainz.de <TheFileToCopy> <username>@<service-node>:~/<PathToDestination>

However, when using an ssh-configuration as shown in HowTo-connect section, the command simplifies to:

scp <TheFileToCopy> mogon:<PathToDestination>
# or in case of full directories:
scp -r <TheDirectoryToCopy> mogon:<PathToDestination>

See the section below for detailed instructions.

Basic Syntax#

To use scp you need to define the source and the destination. Here, a host needs to be given for either one or both, source and destination.

Conventions#

For the purpose of the following code examples below we will refer to

server=<some name of a server in your institutes basement>
mogon=<either mogon login access>

Example#

# to copy a <file> in the current location to a <remote> host into the home directory ('.'):
scp <file> <remote>:.

# if the username is different on the remote, you need to supply it:
scp <file> <user>@<remote>:.

# if you want the file to go to a different location on the remote, you need to supply it:
scp <file> <remote>:/path/to/location/on/remote/.

# you may rename the file, as with the cp program:
scp <file> <remote>:<new_name>

# an entire directory and subdirectory tree can be copied recursively with -r:
scp -r <directory> <remote>:/path/to/location/on/remote/.

These examples should illustrate how copy data from a host to a remote. In order to retrieve data from a remote, the remote server has to take the place of the source and, for instance, the destination becomes your host, e. g.:

scp -r <remote>:/path/to/location/on/remote/ <path/on/your/machine>

You can invoke scp on MOGON II as the host and treat your destination as the remote, too.

Important Flags#

Direct copy#

Direct copy means to copy from the machine you are logged on to and from MOGON.

Here, you can invoke scp for a single file:

scp <filename> <mogon>:/desired/path

or, using the -r flag an entire directory:

scp -r <directory> <mogon>:/desired/path

Copy from remote host to MOGON and vice versa#

The idea is to trigger a command on your desktop to transfer from a remote storage (e. g. an institutes server) directly to MOGON II without the need to transfer to your desktop and subsequently to MOGON II.

The command line will need an additional flag and the names of both servers:

scp $server:<path to copy from server> mogon:<destination path on mogon>
# example
scp $server:/data/scripts/script.sh mogon:./bin/.

In order to copy entire directories use the -r flag, see above.

Copying files with rsync#

rsync is a utility to transfer and synchronize files between computers (or external drives).

When to use rsync#

• The need to synchronize often arises in software development, when attempting to develop on a remote system and then transferring to an HPC in order to compile. This is an artificially imposed need. Consider to develop an HPC system and to use version management (e.g. the ZDV gitlab server ) for your development cycle. This means shorter turn-around times and better testing opportunities.
• Other needs might arise from project needs. In the case of big data which need to be in sync, please consider lftp.

Basics#

Using rsync is straight forward and best shown by a simple example:

Assuming you want to synchronize a <source> directory on your desktop with a known destination (a path) on <mogon> (here, <mogon> stands for the address to be used with ssh/scp as rsync uses ssh.

rsync -avzh <source> <mogon>:<destination>

Here,

• -a is for archive, meaning it will preserve all the permissions , links , dates etc.
• -v is for “verbosely report what you are doing” (can frequently be omitted)
• -z is used for compressing of data during the transfer (may or may not be faster, see scp-comment)
• -h ask to display all output in a human readable format.

Copying files with WinSCP#

If you want to use WinSCP on Windows, configure it according to the following screenshots.

Prev

1 2 3 4

Next

In order to contact MOGON II with WinSCP, use:

• miil01.zdv.uni-mainz.de,
• miil02.zdv.uni-mainz.de, or
• miil03.zdv.uni-mainz.de

Downloading data#

On our HPC systems, users can download scientific data from selected resources. Allowed resource URLs include a number of scientific databases and software repositories. It is not feasible to reflect the entire list here. If a certain resource is missing, just

If a URL is not included on our whitelist, you will receive the following message when trying to download something:

Resolving webproxy.zdv.uni-mainz.de (webproxy.zdv.uni-mainz.de)... 134.93.176.24
Connecting to webproxy.zdv.uni-mainz.de (webproxy.zdv.uni-mainz.de)|134.93.176.24|:8888... connected.
Proxy tunneling failed: Forbidden Unable to establish SSL connection.