Best Open Source Alternative to Cloudsfer, Mover.io and MultCloud
Cloud storage providers are well-known for providing promotional free storage and teaser prices to attract new users to upload data to their service. Companies like Dropbox, Google Drive, and Microsoft OneDrive know how time-consuming it can be to download all your data and re-upload it to a different cloud storage service. A typical home or small business Internet connection might have a top download speed of 30Mbps (3.75MB/s) so downloading 200GB of data you stored in the cloud would take 15 hours. Unfortunately since most cable or DSL plans are asymmetrical, the upload speed for the same connection might be limited to 5Mbps (0.625MB/s). In the best case scenario, the same 200GB of data would take 89 hours (almost 4 days) to upload from your computer to a different cloud storage service. Any shrewd business knows that if the switching costs are high, they can easily “lock in” customers to their service and dictate terms such as pricing and policies.
Web-based tools such as Cloudsfer, Mover.io, and MultCloud have emerged to transfer data from cloud-to-cloud, but they don’t support many of the cloud storage services out there, and charge for every gigabyte of data that is moved ($1/GB after the first 20GB for Mover.io). If you have terabytes of data to move, your costs paid to Cloudsfer, Mover.io, or MultiCloud alone would be thousands of dollars. Given how cheap bandwidth is, this is a ridiculous markup over wholesale prices which you pay for the graphical, point-and-click tools that Cloudsfer, Mover.io, or MultCloud provide to move your data.
Perhaps your reason for moving data from a cloud storage service like Dropbox or Google Drive has nothing to do with pricing at all. Following the PRISM revelations by Edward Snowden, many customers simply don’t wish to store data with the major US technology companies anymore for privacy reasons. If you set up a NextCloud instance and want to migrate your data to it, the data must also be downloaded from the existing storage service and re-uploaded to NextCloud. If this is one of the reasons discouraging you from switching to a self-hosted file sync & share solution, the solution we outline below for moving files from cloud-to-cloud will be equally interesting to you.
Move Data from Dropbox, Google Drive, OneDrive to Amazon S3, BackBlaze B2, and other cloud storage services.
How we moved hundreds of GBs of data between cloud storages in hours, not days.
rclone is an open-source tool that connects to the APIs for 38 different cloud and object storage services to retrieve, sync, and upload data. The biggest bottleneck for switching cloud storage providers is typically the slow Internet connection at your home or office. rclone is a command-line based utility which can run on a VPS within a datacenter. A datacenter connection clocks in many times faster than consumer cable or DSL (100Mbps to 1Gbps upload and download speed), which means moving between cloud storage providers can take hours instead of days. Using a utility such as rclone to transfer your data from cloud-to-cloud also means the transfer process won’t be disrupted when you shutdown your workstation or do other bandwidth-intensive tasks.
If you choose a VPS with plenty of included bandwidth such as DigitalOcean, you could move terabytes of data with no per-gigabyte cost unlike services such as Cloudsfer, Mover.io, or MultCloud. With most cloud computing providers, only egress bandwidth is counted towards your quota – so a 1TB data transfer means you could actually move that volume of data using rclone, at no additional cost. The round-trip bandwidth used to move 1TB of data would actually be 2TB (1TB to download from service A + 1TB to upload to service B) but since only egress bandwidth is metered, you would only be deemed to have consumed 1TB of data transfer.
The most inexpensive droplet on DigitalOcean includes 1TB of data transfer, and overage is billed at only $0.01/GB (1 TB = $10). We do not recommend using rclone or similar utilities with a cloud provider such AWS, Azure, or Google Cloud, as bandwidth prices are 9-10X more expensive than with DigitalOcean.
Linode, ScaleWay, or Vultr are other fantastic options for spinning up an rclone node to transfer data between cloud storage services, with similarly cheap (or unlimited) bandwidth to DigitalOcean. It doesn’t really matter which service you use, but if you are moving terabytes of data, it might be worth doing a speed test to see which provider’s datacenters has the best transfer speeds from/to the services you are pulling & pushing data. Generally, cloud servers in major US-based locations like New York City or San Francisco will have the best transfer speeds with Dropbox, Google Drive, and OneDrive. If your data is destined for a self-hosted NextCloud instance, it would be advisable to choose a VPS geographically close to your NextCloud primary storage — even in the same datacenter if possible.
rclone can be installed on any major Linux distro, including CentOS, Debian, or Ubuntu using a one-line command which invokes an installer script.
curl https://rclone.org/install.sh | sudo bash
Once installed, run the rclone config command to provide your credentials for the cloud storage providers you plan to retrieve and upload data.
It is best to use a command such as nohup or screen to run any rclone jobs in the background so they do not depend on your SSH session to stay alive for the duration of the data transfer.
The rclone sync command is one of the most common options for moving data from one cloud storage provider to another. According to the sync documentation, it makes “source and dest identical, modifying destination only.”
The syntax is as follows:
rclone sync source:path dest:path [flags]
However if you have existing files at the destination (which do not exist at the source) that you do not wish to be deleted, use rclone copy instead. The copy operation “[copies] files from source to dest, skipping already copied.” The --progress flag is helpful for monitoring the transfer rate and ETA of the transfer.
rclone copy source:path dest:path [flags]
When you added the credentials for each storage provider with rclone config you specified an alias for each provider. When “source” and “dest” appear above, it should be replaced with the corresponding alias for the provider you would like to copy from/to. The path for each provider should be specified in the required format for each provider.
You can create the containing directory for the files on the destination using the web-based interface or desktop sync client for the respective cloud storage service, or by using the rclone mkdir command.
If you created the parent folders for the data you plan to transfer using an interface other than rclone, use the rclone ls (list directories) and rclone lsd (list buckets) commands to ensure the correct bucket and directory name you are copying to. In addition, if not transferring the entire contents of an account or bucket from the source provider, doing a ls first is helpful to ascertain the correct directory or directories to copy from.
Using the --dry-run flag to simulate a transfer before actually performing it is always a wise idea to prevent inadvertently overwriting files, or unintended changes to the directory structure at the destination.
Because there are 38 different providers supported by rclone, it is outside the scope of this article to list all the possible examples of the syntax you would use to move files from provider X to provider Y. Anybody familiar with text-based tools such as rsync or the AWS CLI should find rclone pretty easy to grasp, but it can be daunting for the average user.
That’s why Autoize’s team of specialists is available to help you or your organization migrate data between cloud storage services using rclone from the list below. We handle the move from end-to-end for a fixed price (bandwidth included), including for services that are not supported by Cloudsfer, Mover.io, or MultCloud. All you have to do is provide us your credentials for the source and destination services, and tell us how much data you plan to move — and we would be pleased to handle the process from start-to-finish, and verify the data has been successfully copied. No fuss, no muss.
- Alibaba Cloud (Aliyun) Object Storage System (OSS)
- Amazon Drive
- Amazon S3
- Backblaze B2
- Box
- Ceph
- DigitalOcean Spaces
- Dreamhost
- Dropbox
- FTP
- Google Cloud Storage
- Google Drive
- HTTP
- Hubic
- Jottacloud
- IBM COS S3
- Koofr
- Memset Memstore
- Mega
- Microsoft Azure Blob Storage
- Microsoft OneDrive
- Minio
- Nextcloud
- OVH
- OpenDrive
- Openstack Swift
- Oracle Cloud Storage
- ownCloud
- pCloud
- put.io
- QingStor
- Rackspace Cloud Files
- Scaleway
- SFTP
- Wasabi
- WebDAV
- Yandex Disk
- The local filesystem
