Transfer a 1GB file in 17 seconds using rsync

How to use rsync to transfer a 1GB file in 17 seconds.

September 24, 2023

rsync stands for "Remote Sync" and is a fast file transfer utility. It only transfers changed parts of files, making it incredibly efficient. Now, let's get into how you can use rsync to transfer a 1GB file in just 17 seconds.

The mkfile command comes bundled on macOS. It creates a file on your computer of any size you choose. It can be used to create temporary files for testing.

The mkfile command takes 2 arguments, size and filename:

mkfile -n size[b|k|m|g] filename

To create a 1GB file called 'useless_file' type the following in the terminal:

mkfile -n 1g useless_file

A new 1GB file is created in the current working directory. It will be made up of all 0's (this ties into using the rsync command below).

The rsync command also comes bundled on macOS. It can transfer files locally or remotely. It supports a number of features including compression and SSH to copy files over a network.

Here is the command to copy a file within the same directory:

rsync useless_file useless_file_copied

This makes a duplicate of useless_file called useless_file_copied

On my SSD this took about 10 seconds.

To copy this over a network use this command:

rsync -e ssh useless_file <username>@<destination-ip>:~

  • -e and ssh are used together to enable a secure ssh connection.

  • The <username> and <destination-ip> are your username credential and address of the destination computer.

  • Directly after the : is the save directory on the destination computer. I currently have useless_file saving to the users ~ home directory.

This took over 2 minutes and 26 seconds to transfer on a local network. To speed this up use the -z flag which enables compression. Because useless_file is all 0's it will compress more than a typical file.

Here is the updated command with compression enabled:

rsync -z -e ssh useless_file <username>@<destination-ip>:~

This time it took 17 seconds to transfer a 1GB file to another computer.

Pretty neat, eh?

coffee break

But wait… what about using rsync in the 'real world'?

Because this 1GB file is made up of all 0's, how does this compare to sending something useful?

I took a 234MB .dmg and transfered it with and without compression, here are the results:

  • 25 seconds without compression
  • 23 seconds with compression

This .dmg file was already heavily compressed by Oracle. It is the Java installer .dmg from their website.

Okay, how about a 730MB .wav file (people use those):

  • 1 minute and 26 seconds without compression
  • 41 seconds with compression

That's less than half the time needed to transfer the same file. Pretty sweet, isn't it?

So to conclude, the rsync command was used to transfer 3 different files to a remote computer on a local network. The first file was 1GB and made using the mkfile command. It was composed of all 0's and took 17 seconds to transfer.

The second and third transfer showed results using more useful files. A file that is already compressed such as a .dmg (or .zip, etc) will probably not benefit from adding the -z compression flag.

But uncompressed files such a .wav transferred in less than half the time.

rsync is also the backbone of the popular Carbon Copy Cloner software for macOS. It is open source and credited on their website.

Next time you reach for the popular scp command to copy a file, check out rsync. It might speed up your file transfers.

Originally published at blog.jdwilson.ca on March 30, 2017.