Implementing S3 Compatible Primary Storage for NextCloud

NextCloud, an open source cloud storage solution, supports using S3 compatible object storage not only as external storage, but also as the primary (internal) storage for your NextCloud deployment. First, why is it “S3 compatible” instead of S3? The Amazon S3 (Simple Storage Service) API has become the de-facto standard for create, read, update, and delete (CRUD) operations for object storage. Many of Amazon’s cloud competitors support the S3 protocol for accessing their respective object storage services, including DigitalOcean Spaces, Linode Object Storage, Wasabi Hot Storage, and OpenStack Swift. Therefore, any third-party software integrations such as NextCloud’s External Storage App and OC\Files\ObjectStore\S3 class can completely interoperate with S3 compatible services.

Why would one be interested in using an object store for NextCloud storage?

  • Inexpensive – Amazon S3 storage is 2.3 cents per GB/month, and S3 compatible services generally settle around 2 cents per GB/month. Wasabi, the latest entrant with cut-rate prices comes to 0.59 cents per GB/month.
  • Infinitely scalable – Each object storage bucket can store an unlimited number of objects up to a ridiculously large file size (5TB for Amazon S3). Gone are the worries of running out of storage space, or resizing block volumes.
  • Durability built in – At a minimum, most object storage services triple replicate your data across multiple hard drives to guard against hardware failure. Some services use “erasure coding” (similar to RAID striping) to be able to rebuild the data with no interruption to availability.

Furthermore, open-source applications such as MinIO can be optionally used as an API gateway to translate NextCloud’s S3-compatible API calls into API calls that non-S3 services such as Azure Blob Storage, Google Cloud Storage, or BackBlaze B2 can understand.

Until now, using S3 compatible object storage as the backend for NextCloud has been shrouded in mystery. OwnCloud (the predecessor to NextCloud) and NextCloud have long advertised “S3 as primary storage” as an “enterprise feature.”

Many administrators of the community edition of NextCloud have settled for mounting S3 into a NextCloud folder using the External Storage app. Compared to using S3 as the primary storage, this approach comes with considerable drawbacks.

External Storage Support in NextCloud (Recommended only for single users)

External Storage support ships as a default app with NextCloud; it simply needs to be enabled by an admin user by browsing to “Apps” from the dropdown menu. Once it’s enabled in Settings you will have the option to mount “External Storages” under Administration.

If “/” is chosen as the Folder name, any NextCloud users who are granted access to the External Storage will upload to the S3 compatible backend by default. If this isn’t the desired behavior, the bucket should be mounted with a specific folder name – so that users will need to access the mount from the “External Storages” area of their NextCloud dashboard.

Administration – Mounting an S3 Bucket as External Storage

NextCloud “External storages” view from user dashboard

File and directory names are visible from within the DigitalOcean Spaces dashboard

The main drawbacks of mounting an S3 bucket (or in this case, DigitalOcean Space) as external storage in NextCloud include the following:

  • Inadequate access controls – One storage bucket must be created for each NextCloud user (if each user is to have their own “private” cloud storage), considerably increasing the setup time for administrators. Using one bucket for all users has the same effect as sharing a DropBox Pro or Google Drive account between the entire team/company.
  • Storage quotas are not applicable – Storage quotas do not apply to external storage in NextCloud, leaving the administrator without control over how much data each user may keep.
  • NextCloud sharing doesn’t work – Files on external storage cannot be shared with other users (who don’t have access to the share) via the normal workflow – using Name, Federated Cloud ID, or Email Address. Access to external storage may only be shared as an entire folder, making NextCloud much less useful as a collaborative tool.
S3 as Primary Storage in NextCloud (Scalable, Enterprise Solution)

To overcome all of the drawbacks as mentioned above, S3-compatible object stores can be configured as primary storage for your NextCloud instance. This can only be setup during the initial NextCloud install process, so if you already have a NextCloud instance, you will need to create a brand new instance (and migrate any data over via WebDAV).

Create a new bucket or space in your object storage service of choice, and note down the bucket name. If you haven’t generated the API credentials (or IAM role) needed for your NextCloud instance to access the S3-compatible storage, you should do that as well. Be sure that you do not create any files or folders in the bucket through the S3 interface, as the data will be completely managed through NextCloud.

If you are using the same object storage service as your cloud provider, you should choose the same region as the datacenter where your NextCloud deployment is hosted. This will reduce latency (improving performance) and negate egress bandwidth charges from your NextCloud server to the object storage endpoint.

In the example below, our NextCloud instances uses the FRA1 (Frankfurt) DigitalOcean Spaces endpoint as our NextCloud deployment is also hosted in the FRA1 region.

Wasabi, or other standalone object storage services can be used as well, but unless the cloud provider where you are hosting NextCloud offers a large amount of free outgoing Internet bandwidth, it can end up increasing your costs. If you host your NextCloud deployment with DigitalOcean, Linode, or any cloud provider who is part of the Bandwidth Alliance of which Wasabi is a member, then you can transfer large amounts of data to Wasabi at a free or heavily reduced cost. Currently (as of August 2019), DigitalOcean and Linode have announced plans to waive bandwidth charges to other Bandwidth Alliance members, but have not yet begun doing so. Until then, the regular bandwidth allotments for their plans apply (1TB for every $5/month of the cost of the plan).

We recommend using a S3-compatible storage service such as DigitalOcean Spaces or Wasabi that does not bill for API requests, as a large volume of API requests can be generated by NextCloud as your users interact with the NextCloud dashboard or sync client.

Prior to installing NextCloud for the first time, browse to the config folder under your NextCloud application directory (for example if your NextCloud directory is /var/www/nextcloud/, it would be /var/www/nextcloud/config/). Then create a file named storage.config.php.

<?php
$CONFIG = [
'objectstore' => [
'class' => 'OC\Files\ObjectStore\S3',
'arguments' => [
'bucket' => 'bucketname',
'autocreate' => true,
'key' => 'AAAAAAAAAAAAAAAAAAA',
'secret' => 'AaAAaaaAAA11aAAAAAaa11aA+aAaAA1aAaaAAaAAAAA',
'hostname' => 'fra1.digitaloceanspaces.com',
'port' => 443,
'use_ssl' => true,
'region' => 'fra1',
// required for some non Amazon S3 implementations
'use_path_style'=> false, ],
],
];

Once you’ve completed the NextCloud install through the web wizard, the skeleton of the home directory for the initial (admin) user account will be populated into the object store. From this point onward, data uploaded by any NextCloud user (to any folder) will be automatically stored in the S3 bucket and endpoint specified in the config file.

Note that when using S3-compatible storage as NextCloud’s primary backend, the files are stored separately from the metadata (file and folder names) meaning you won’t be able to browse or interact with the data through S3 directly (outside of NextCloud).

When using an object store as primary storage, transactional file locking should be disabled by adding this line to config.php above );

'filelocking.enabled' => false,

Although you will see a warning message appear in the Administration > Overview section of the NextCloud dashboard, it is safe to ignore this message – as the object storage service’s API should gracefully handle the situation if two PHP workers attempt to write to the same file simultaneously. If locking is handled by both NextCloud (at the application level) and by the storage backend, a conflicting situation will occur where files cannot be updated or deleted due to file locks.

If your NextCloud deployment uses an online office suite such as Collabora or ONLYOFFICE – and you want to implement file locking to prevent users from inadvertently overwriting each other’s changes at the same time, you can consider adding a NextCloud app such as W2G2.

It is imperative that you safeguard the NextCloud MySQL or Postgres database cautiously as the metadata stored there is required to reconstruct the file and folder structure of the data that any of your users store within NextCloud. Consider setting up master/slave database replication to ensure you always have a working copy of your NextCloud database – in case your primary database server fails for any reason.

File and directory names are not visible from within the DigitalOcean Spaces dashboard

NextCloud’s server-side encryption feature is supported in conjunction with S3 object storage as primary storage, but it increases the size of data stored by about 35%, and an additional performance overhead of about the same. If you require your users’ data to be encrypted by their user-specific key (so it cannot be trivially read by the cloud provider), then you can turn this feature on. Otherwise, we recommend to keep server-side encryption switched off for performance and compatibility with all NextCloud apps.

If your NextCloud users will simultaneously upload large quantities of data to the instance, you will need a /tmp directory on the application server that can accomodate the total size of the buffered data, before it’s streamed to the object storage.

This instance uses an S3 based object store as primary storage. The uploaded files are stored temporarily on the server and thus it is recommended to have 50 GB of free space available in the temp directory of PHP. Check the logs for full details about the path and the available space. To improve this please change the temporary directory in the php.ini or make more space available in that path.

Thinking of using S3 as primary storage with your NextCloud deployment? Our NextCloud administrators can assist with selecting the most cost-optimized, S3-compatible object storage, migrating data from an existing NextCloud instance, backing up data between multiple clouds, and any other NextCloud configuration needs you may encounter – please don’t hesitate to contact the Autoize team.