File Transfer Guide¶
Several methods can be used to transfer files to the cluster. These include command line commands, Windows applications, or web protocols.
Method 1: RSYNC (Command Line)¶
rsync is a fast, file-copying tool that synchronizes files and directories between two locations over a network while only transferring the differences, preferred when you need efficient and resilient transfers of a large amount of data.
# Copy a single file to cluster
rsync -azrP myfile.txt username@orion-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
rsync -azrP myproject/ username@orion-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
rsync -azrP username@orion-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
rsync -azrP myfile.txt username@hercules-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
rsync -azrP myproject/ username@hercules-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
rsync -azrP username@hercules-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
rsync -azrP myfile.txt username@ptolemy-dtn.arc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
rsync -azrP myproject/ username@ptolemy-dtn.arc.msstate.edu:/path/to/destination/
# Copy from cluster to local
rsync -azrP username@ptolemy-dtn.arc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
rsync -azrP myfile.txt username@atlas-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
rsync -azrP myproject/ username@atlas-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
rsync -azrP username@atlas-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
rsync -azrP myfile.txt username@morrill-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
rsync -azrP myproject/ username@morrill-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
rsync -azrP username@morrill-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
You may omit the -P (equivalent to --progress + --partial) flags if you are only transferring a few small files, have a scripted transfer that would not benefit from progress being displayed, or if a partial file copy would be detrimental if the copy failed. You may use --progress without the -P to restore the progress display.
Important
It is suggested that you do not compress files prior to transfer if you are using rsync with the -z flag and are relying on the the protocol’s difference comparison feature. The -z flag will not further compress a compressed file and will simply waste CPU cycles. You should omit the -z if you are transferring a pre-compressed file.
Important
Because rsync does a file delta between the source and destination directories, it is suggested that you do not package files into an archive prior to transfer.
Trailing Slashes
Take careful note of trailing slashes when using commandline file transfer tools.
rsync /path/to/file remote:/new/path/
will not behave the same as
rsync /path/to/file/ remote:/new/path/
Method 2: SCP (Command Line)¶
scp copies files between hosts over an encrypted SSH connection and is preferred when you need a simple, secure, one-off transfer without the complexity of rsync.
# Copy a single file to cluster
scp myfile.txt username@orion-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
scp -r myproject/ username@orion-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
scp username@orion-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
scp myfile.txt username@hercules-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
scp -r myproject/ username@hercules-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
scp username@hercules-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
scp myfile.txt username@ptolemy-dtn.arc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
scp -r myproject/ username@ptolemy-dtn.arc.msstate.edu:/path/to/destination/
# Copy from cluster to local
scp username@ptolemy-dtn.arc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
scp myfile.txt username@atlas-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
scp -r myproject/ username@atlas-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
scp username@atlas-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
# Copy a single file to cluster
scp myfile.txt username@morrill-dtn.hpc.msstate.edu:/path/to/destination/
# Copy entire directory to cluster
scp -r myproject/ username@morrill-dtn.hpc.msstate.edu:/path/to/destination/
# Copy from cluster to local
scp username@morrill-dtn.hpc.msstate.edu:/path/to/myfile.txt /path/to/local/
You may want to include the -q flag if you are scripting file transfers, as it suppresses warnings and progress.
Method 3: Globus (Large Unattended Transfers)¶
Globus is used for large unattended automated data transfers, use Globus
Globus Endpoints:
Cluster |
Endpoint |
|---|---|
Orion |
msuhpc2#Orion-dtn |
Hercules |
msuhpc2#Hercules-dtn |
Ptolemy |
MsStateHPC#Ptolemy-dtn |
Atlas |
SCINet-Atlas |
Morrill |
MsStateHPC#Morrill-dtn |
Method 4: SFTP (Command Line Interactive)¶
sftp provides an interactive, encrypted file transfer session over SSH.
sftp username@orion-dtn.hpc.msstate.edu
sftp username@hercules-dtn.hpc.msstate.edu
sftp username@ptolemy-dtn.arc.msstate.edu
sftp username@atlas-dtn.hpc.msstate.edu
sftp username@morrill-dtn.hpc.msstate.edu
Once connected:
# List files in remote directory
ls
# Change remote directory
cd /path/to/dir
# Current working remote directory
pwd
# List files in local directory
lls
# Change local directory
lcd /path/to/dir
# Current working local directory
lpwd
# Copy a single file to remote
put myfile.txt /path/to/destination/
# Copy an entire directory to remote (requires -r)
put -r myproject/ /path/to/destination/
# Copy from remote to local
get /path/to/remote/file.txt /path/to/local/
# Exit
exit
Method 5: Open OnDemand (Web)¶
Login to http://orion-ood.hpc.msstate.edu
Click Files
Navigate to destination directory
Click Upload to add files
Select files and click Download to retrieve
Login to http://hercules-ood.hpc.msstate.edu
Click Files
Navigate to destination directory
Click Upload to add files
Select files and click Download to retrieve
Login to http://ptolemy-ood.arc.msstate.edu
Click Files
Navigate to destination directory
Click Upload to add files
Select files and click Download to retrieve
Login to http://atlas-ood.hpc.msstate.edu
Click Files
Navigate to destination directory
Click Upload to add files
Select files and click Download to retrieve
Login to http://morrill-ood.hpc.msstate.edu
Click Files
Navigate to destination directory
Click Upload to add files
Select files and click Download to retrieve
Open OnDemand works for all clusters - just select the desired cluster when logging in.
Method 6: WinSCP (Windows GUI)¶
Create new session:
Protocol: SFTP
Host: Use your cluster’s data transfer node (e.g.,
orion-dtn.hpc.msstate.edu)Username/Password: Your credentials
Drag and drop files between panels
Best Practices¶
Compress Before Transfer¶
# Create archive locally
tar -czf project.tar.gz myproject/
# Transfer compressed file
scp project.tar.gz username@cluster-dtn.hpc.msstate.edu:/path/to/destination
# Extract on cluster
tar -xzf project.tar.gz
Where to Store Files¶
See Storage for where files should go as well as details on quotas and retention policies for each filesystem, which differs by cluster.
Transfer Speed Tips¶
Compress files before transferring
Avoid transferring many small files (use tar)
Transfer during off-peak hours for better bandwidth
Important
Do not pre-compress files when using rsync -z or when relying on rsync’s incremental sync.
Troubleshooting¶
Connection Timeout¶
Check network connection
Verify hostname is correct
Try a different login node
Check if firewall is blocking port 22
Permission Denied¶
Verify username/password
Check SSH key permissions (
chmod 600 ~/.ssh/id_rsa)Ensure you have write permissions in destination directory
Contact support if account issues
File Too Large¶
Check quota in target directory (
df -h)Use a different storage location
Compress the file before transfer