Feeding Filespooler Queues from Other Queues

Sometimes with Filespooler, you may wish for your queue processing to effectively re-queue your jobs into other queues. Examples may be:

  • Parallel processing
  • Distributing jobs to various workers

There are two approaches to this:

  1. Directly reusing the job file in the destination. To do this, you would hard link the job file into the destination queue’s jobs directory. See Parallel Processing of Filespooler Queues for an example of this approach.
  2. Alternatively, you could pipe data to fspl prepare | fspl queue-write as usual. This would have the effect of re-generating the sequence numbers, which may be desirable in some cases.

Depending on the specifcs of your situation, you may have to process jobs out of order; see the conversation in Many-To-One with Filespooler for more on this.

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication.