Verifying Filespooler Job Integrity

Sometimes, one wants to verify the integrity and authenticity of a Filespooler job file before processing it.

This poses a particular challenge, because most of these verification tools (including GnuPG (GPG) have a weakness: even though some of them can work in a pipe, most of them are not able to establish whether the data was authentic or not until they have processed the entire file. So even when Filespooler may only want to read 100 bytes to get to the header, the verifier may emit that data without having validated it.

So the usual Filespooler trick of using a --decoder isn’t applicable here.

Instead, we really need to pre-process the data. Since one can Process Filespooler Queues Without Filespooler, we can have a two-step approach:

  1. Incoming data is written to an “incoming” queue, which will never actually be processed by Filespooler.
  2. When it is verified, it is moved to the real queue to be processed.

Cryptographic signatures (or, really, simple hashes) come in two flavors: ones that embed the signature in the file itself, and ones that put it in a separate file. This example focuses on the former, but the latter is easy to handle as well; you can just reference the signature file and delete it once verified.

Dealing with GPG signatures

GPG is a particularly interesting problem, because it has several options:

  • --verify can be used to verify a signature on a plaintext
  • But if you are verifying a signature on something encrypted, you have to use --decrypt and that implies a verification, which will be done at the end.

This example will assume data was encrypted and signed. We will put it into the incoming queue, process from there without using Filespooler, and then move it to the “verified” queue.

Prepare the queues

First, let’s create the queues:

fspl queue-init -q incoming
fspl queue-init -q verified

We’re never going to actually use Filespooler to read the incoming queue, but by setting it up this way, we can use fspl queue-write to add things to it.

Creating the signed and encrypted packets

For more detail, please refer to Encrypting Filespooler Jobs with GPG. This assumes a file synchronization transport as in Using Filespooler over Syncthing; of course, you can adapt it to whatever transport you like.

You can create a signed and encrypted packet with something like this:

echo hi | \
  fspl prepare -s state -i - | \
  gpg -e -r 0x1234567891123456789212345678931234567894 -u 0x1234567891123456789212345678931234567894 --sign - | \
  fspl queue-write -q incoming

I always like to give the full 40-digit hex ID for a key to GPG for maximum security. Here, we:

  • Use echo hi to supply some input for the packet.
  • Use fspl prepare like usual to generate it.
  • Pipe that through gpg. Its options are:
    • -e says to encrypt the data
    • -r sets the recipient for the encryption
    • -u sets the key ID to sign with
    • --sign says to sign it
    • - says to process stdin
  • Then we use fspl queue-write to add it to incoming. Notice that fsp queue-write has no access to the plaintext of the packet, nor does it need it.

Verify the signed packets

Now, we need to verify the signed packets. As we’ve established before, using Filespooler to do this directly has some obstacles, but it is easy to do ourselves. First, we write a script to verify it:

#!/usr/bin/env bash

set -euo pipefail

if gpg --status-fd 1 -o /dev/null -d < "$2" | \
        grep -q 'VALIDSIG 1234567891123456789212345678931234567894'; then
   echo ln "$2" "$1/$2"
   echo rm "$2"
   echo "Failed validation of $1"
   exit 1

Save this as queue-verify somewhere on your path and mark it executable.

Let’s walk through that script and how it works:

  • We assume it is called with two parameters:
    • $1 is the path to the verified queue
    • $2 is the file to verify
  • We call gpg to process it.
    • --status-fd 1 tells gpg to write status information in a machine-parsable way to stdout.
    • -o /dev/null tells gpg to discard the decrypted data.
  • Then we grep for an indication that we had a valid signature from the key we expected.
  • If so, we use the method recommended in Guidelines for Writing To Filespooler Queues Without Using Filespooler to safely inject it into the verified queue:
    • First, create a hardlink to add it there
    • Then, delete it from the incoming queue.

We will call this script from find. Due to the way it handles paths, the directory it’s called from is important:

cd incoming
find jobs -name 'fspl-*.fspl' -exec queue-verify ../verified {} \;

Handle the verified packets

This is the same as in Encrypting Filespooler Jobs with GPG. For instance:

fspl queue-ls -d 'gpg -q -d - 2>/dev/null' -q verified

Of course, you could use queue-process or whatever here also.

Other verifiers

Here are some other ideas for verification tools:

  • sha256sum doesn’t verify a signature, but if all you need is detection of integrity, that could be an easy option.
  • signify is another tool for making signatures.
  • GnuPG itself has a command gpgv that may be useful in some situations.
  • Sequoia PGP is an alternative to GnuPG that is broadly compatible with its on-disk formats.

GnuPG (also known by its command name, gpg) is a tool primarily for public key Encryption and cryptographic authentication.

Thanks to Filespooler’s support for decoders, data for filespooler can be Encrypted at rest and only decrypted when Filespooler needs to scan or process a queue.

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication.