Using Filespooler over rclone and S3, rsync.net, etc.

You can use Filespooler with a number of other filesystems and storage options. s3fs, for instance, lets you mount S3 filesystems locally. I can’t possibly write about every such option, so I’ll write about one: rclone.

rclone is nice because it is a unified interface to dozens of storage backends. Examples include S3, Dropbox, Nextcloud, OneDrive, Amazon Drive, SFTP, and many more. rclone can interact with these storage backends via the command line, or can mount them as with s3fs. Let’s dive in.

Preparation

You will need to create a configuration with rclone config. See the rclone documentation for details; this is out of scope for this manual.

Basics & Setup

There are several ways we can do this:

  1. Mounting a remote directory directly into the jobs directory within the queue

  2. Using commands like rclone move to download remote files into there.

Because option 1 is generally simpler and safer, we’ll use it here.

Use with rclone mount

Because Filespooler does indeed only read and write files sequentially, it is fully compatible with the limitations on rclone mount. You do not need to use any of the --vfs-cache-mode options with Filespooler. (Though of course you may wish to if they provide useful performance benefits in some cases)

You should mount only the jobs subdirectory. That will mean creating an append-only queue on the sending side. The page Filespooler Append-Only Queues discusses doing this. Once you have mounted the remote end with rclone mount, the rest of the Using Filespooler over Syncthing tutorial is applicable.

Additional notes

Some rclone backends dislike empty directories. You could easily touch jobs/keep-me or something to make sure that directory is never empty. Since it doesn’t match a Filespooler pattern, Filespooler will just ignore it.

Some rclone backends have eventual consistency; that is, writes may not immediately be visible. This is perfectly fine with Filespooler, so long as only jobs is on the remote. The strong sequencing will make sure that a new job is processed only when it becomes visible.

Things to avoid

There are some hazards in that some rclone commands with some backends may write in a proper way, while others don’t. I strongly recommend you test your setup with the Guidelines for Writing To Filespooler Queues Without Using Filespooler approach.

The rclone bisync command comments that “When using Local, FTP or SFTP remotes rclone does not create temporary files at the destination when copying, and thus if the connection is lost the created file may be corrupt, which will likely propagate back to the original path on the next sync, resulting in data loss.” This violates the Guidelines for Writing To Filespooler Queues Without Using Filespooler and makes it unsuitable. Note that you will almost always have Local as one end of the bisync. These comments apply to some other tools as well.

Filespooler provides the fspl queue-write command to easily add files to a queue. However, the design of Filespooler intentionally makes it easy to add files to the queue by some other command. For instance, Using Filespooler over Syncthing has Syncthing do the final write, the nncp-file (but not the nncp-exec) method in Using Filespooler over NNCP had NNCP do it, and so forth.

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication.