Migrating NextCloud from S3 Backend to S3FS Object Storage

Autoize recently migrated a NextCloud instance hosted on AWS configured with the S3 primary storage backend to S3FS on behalf of a major restaurant franchise operator in the southwestern US. The client had stored 2.7 TB of contract and financial documents, marketing collateral, and video surveillance footage on NextCloud for 23 users for the IT, Marketing, and Accounting departments of their company.

Challenge

The frontend of the instance had become too slow to reasonably view or upload any documents or perform any administrative tasks, such as managing users and groups, from NextCloud’s web interface. Also, using the default S3 object storage backend of NextCloud meant that the uploaded files were stored as urn:oid objects instead of preserving the file/folder names and hierarchy in the bucket.

First, the Autoize team checked whether the instance was properly resourced, as it was running a T2-series EC2 instance on Amazon’s cloud. CPU throttling caused by running down an instance’s CPU credits through persistent high CPU utilization is a common problem that can simply be solved by resizing an EC2 to a non-T2-or-T3 instance type.

Based on our findings, CPU or memory usage did not appear to be the bottleneck causing the slow performance and web frontend timeouts on the client’s NextCloud instance. Instead, the large number of small files and subfolders within the directory structure were resulting in an I/O bottleneck on the S3 bucket using NextCloud’s default object storage driver.

The problem turned out to be that the images and CSS for the NextCloud web interface were also persisted on the same S3 bucket using NextCloud’s bespoke object format, leading to broken styles/images when accessing the web frontend while any other users were syncing files from their NextCloud share on the server.

Using Amazon’s S3 object storage instead of a regular filesystem as the primary storage for their NextCloud was a requirement for the client, as their IT department did not want to have to resize or add additional EBS disks to a volume group – as the amount of data they stored in NextCloud continued to grow.

Recommendation

The client engaged Autoize to implement the recommendation of modifying the NextCloud primary storage from the default object storage backend to a S3FS storage solution for NextCloud developed by our team.

Using S3FS-Fuse for NextCloud primary storage means that the storage is presented as if it’s a regular file system to NextCloud, completely bypassing the object storage backend. Any data uploaded to a S3FS-based instance would be stored with its original name, containing folder and sub-folder names so that it’s browsable in the AWS Management Console or using a S3 client such as CyberDuck. It would also allow us to map the styles/images for the web frontend to the EC2 instance’s EBS disk, ensuring that the interface loads up from SSD block storage without experiencing timeouts.

Modifying the primary storage backend always requires a reinstallation of NextCloud, so it was decided that the project would be handled as a rebuild and migration. Autoize would spin up a new EC2 instance on AWS, deploy an optimized NextCloud instance, and migrate the data over Rclone and WebDAV to the new instance.

When a deployment uses NextCloud’s default S3 backend, it is impossible to back up the data stored in the S3 bucket by simply copying the urn:oid objects to a new bucket. Part of the reason for that is because the metadata mapping the urn:oid objects to the original filename (and name of the containing folder) is stored in the MySQL or Postgres database. There is currently no migration tool to convert the list of S3 objects generated by NextCloud back into a human-readable directory structure resembling a regular filesystem.

Using the custom scripts developed by Autoize, we temporarily reset the passwords for the users to a secret which would be shared by both the existing and rebuilt NextCloud instance. The migration script would recursively cycle through the list of users and copy the data owned by the user between the source and the destination, while recreating the metadata on the new instance that is connected to a new S3 bucket over S3FS.

Result

Over the course of approximately 48 hours after temporarily scaling up the EC2 instances to a more powerful M5 instance type (then scaling back down), the 2.7 TB of data was converted from the urn:oid format in the old S3 bucket to a human-readable format in the new S3 bucket. This change would allow the client to directly view and backup the data residing on their NextCloud from the new S3 bucket.

As well, the performance of the web frontend was significantly improved as the styles/image data that is normally stored under the NextCloud data directory had been bind mounted to the relatively high-performance SSD root volume attached to the new NextCloud EC2 instance. This eliminated the timeouts that the client’s users were previously encountering whenever they attempted to access the NextCloud interface through a browser.

With our solution, the restaurant franchise operator client would be able to continue leveraging Amazon S3 for NextCloud storage on a “pay as you go” pricing model, at a much lower cost of ownership and greater privacy than with EFSS competitors like Box.

If you have an existing NextCloud instance and you are not entirely satisfied with its performance, contact our dedicated NextCloud consultants. We will likely be able to identify optimizations such as the above to make NextCloud a more productive collaboration and enterprise file sync & share tool for your on-site and remote workforce.