Easily Using SSH with FIDO2/U2F Hardware Security Keys

A lot of new hardware security keys (Yubikey, Nitrokey, Titan, etc.) now support FIDO2 (aka U2F aka Webauthn aka Passkey; yes it’s a mess).

So does OpenSSH.

This spells good news for us, because it is far easier to use than previous hardware security types (eg, PKCS#11 and OpenPGP) with ssh.

A key benefit of all this, if done correctly, is that it is actually impossible to access the raw SSH private key, and impossible to use it without the presence of the SK and a human touching it.

Also, ssh agent forwarding becomes safer again, and what’s more, it can be used to let you tap your local key to authenticate even when sshing from remote machine A to remote machine B.

I’m going to call these hardware security keys “SKs” within this article.

I’ve been annoyed at the material out there, which often doesn’t explain what’s happening and suggests insecure practices.

So, I’m going to introduce SKs and FIDO2, show how to use the keys with SSH, explain the role of ssh-agent with all of this, and walk you through all of the steps.

The SK landscape

To use this, you need a SK that’s compatible with FIDO2. Personally, I prefer the YubiKey 5 series. They make some very small ones that can stay in a USB-A or USB-C port permanently, and just barely stick out over the edge far enough for you to tap them. There are also larger ones that you can put on your keychain.

Many others also make FIDO2 keys; for instance, Nitrokey 3 series and Google Titan.

I use the YubiKeys because they are easy to obtain around the world, have the best Linux support, and also have other “applications” (TOTP to replace authenticator apps, OpenPGP, etc.) that you can use. For this article, I tested with a YubiKey 5C NFC, a YubiKey 5C Nano, and a Titan. But really, any SK supporting FIDO2 should work.

WARNING!

I wrote this page because I saw a lot of guides online, that seem to have sort of copied-and-pasted from each other and don’t seem to consider what they’re doing.

In particular, if you find another SK guide online that recommends using ssh-keygen with -O resident, then you should run away fast and not use it. (If it adds -O verify-required to -O resident then you should probably run away slowly and not use it.)

There are reasons for -O resident, which I will get into later, but for now, just understand that it likely weakens your security in most scenarios.

FIDO2 key basics

FIDO2 keys are part of a whole ecosystem designed for mainly web authentication. You’ll see names like U2F, Webauthn, and Passkey used. These have certain technical distinctions but basically they are referring to broadly the same ecosystem.

This article gives a good background on the whole system. SSH uses this same mechanism to improve the handling of SSH private keys.

Non-resident keys with SSH

Non-resident means that the SSH private key doesn’t reside within the SK. It also doesn’t reside on your PC!

When you generate a “sk” keypair using ssh-keygen, what’s written to your drive is:

  1. A public key portion like usual
  2. A private key file that holds a “key handle”. This also can be encrypted with a passphrase like usual.

So what is this key handle?

Effectively, it is a SSH private key that is encrypted. There is no way for your computer to ever access the decrypted private key.

It can only be decrypted within the SK using a secret key unique to each SK (which you can’t access). The SK won’t reveal the decrypted private key; rather, it performs the crypto operations using it internally, and only sends back the result of those operations. Generally the SK will require you to physically touch it before performing these operations. For more, see OpenSSH’s PROTOCOL.u2f documentation. Interestingly, nothing is stored within the SK’s non-volatile memory for a non-resident key; you can have as many as you like.

So effectively you create a key that, in order to be used, requires:

  1. The content of the SSH “private key” file (really a key handle)
  2. Optionally, a SSH passphrase to decrypt the key handle in the usual fashion
  3. The security key it was created with
  4. A human to touch that security key at the appropriate time (optionally, this can be augmented by a PIN/passphrase at the SK level)

The really important benefit of this is that the actual SSH private key can never be accessed. There is simply no way to extract it from the SK or the PC.

Further, it is virtually impossible, even if your computer is compromised, for an attacker to use your account to ssh to somewhere else. They would somehow have to convince you to touch the SK at the right time.

Put another way, in order to impersonate you, an attacker would have to obtain the SK itself or convince you to touch it, the content of the private key file, AND your ssh passphrase (if you enabled it). So a simple compromise of your machine (even with a keylogger) doesn’t automatically compromise everything you can log in to. Neither does a stolen SK. Nice, right?

You can also see that securing the key handle with a passphrase is less important with SKs than typical, but I still do so as a good practice.

Generating your keypair

This is how you generate a SK keypair:

ssh-keygen -t ed25519-sk

If you get an error, your key may not support ed25519. In that case, use:

ssh-keygen -t ecdsa-sk

(ed25519-sk is generally preferred, but ecdsa-sk is fine also, and supported by more SKs.)

And you’re done. Using this key will require you to touch the SK. You have a ~/.ssh/id_*_sk.pub and a corresponding ~/.ssh/id_*_sk file. They can be used exactly as any other keypair; add the .pub to ~/.ssh/authorized_keys on a remote like usual.

When you ssh to a remote system, your local ssh will prompt you:

Confirm user presence for key ECDSA-SK...

That’s when you touch your SK.

Use with ssh-agent

You can ssh-add your SK keys just like any other key. If you encrypted the key handle with a password, this will, like any other use of ssh-agent, prevent you from having to enter the password on every connection.

Now, think about what we’re adding to the agent here: it’s the key handle, not the actual private key, because we have no way to access the actual private key. When you need to use the private key, you will still have to touch the SK.

Now this brings some useful properties. Agent forwarding with ssh -A becomes much safer; the main risk for it in traditional setups is that an attacker on a remote host could gain access to the forwarded agent socket and use it to authenticate with your credentials. Not so easy now, since you have to touch the SK to use it. (And can optionally configure it to require inputting a PIN as well)

Another interesting aspect is that no matter how many hosts down a chain of forwarded agents you go, the request to the SK will always come back to the local machine you’re sitting at and you will have to touch the SK on the local machine. Very slick!

ssh-askpass weirdness

You might notice that when you load your key into ssh-agent, you no longer get a “Confirm user presence for key” message. You just have to know to touch your SK. This is harmless unless you want to use -O verify-required to require the entry of a PIN; more on that below. (Some SKs will light up to prompt you to touch them as well)

If you really want to see a prompt, you need to use some flavor of ssh-askpass. That’s because ssh-agent doesn’t have direct access to your terminal and can’t prompt you there itself.

On Debian, for instance, you can install packages like ssh-askpass, ssh-askpass-gnome, ksshaskpass, etc. This will let ssh-agent pop up a window prompting you to confirm presence. Note that it may look like you need to enter a passphrase in that window; don’t do that unless you’re using PIN mode; just touch your SK.

Usually logging out and back in will enable ssh-askpass after installation.

ssh-agent on Wayland

Wayland can be trickier for both ssh-agent and ssh-askpass, since session scripts are less likely to start ssh-agent there. I found that for KDE Plasma on Wayland, I can solve the issue by creating this file at ~/.config/plasma-workspace/env/local-ssh-agent.sh:

if [ -z "$SSH_AUTH_SOCK" ] ; then
        eval `DISPLAY=:0 SSH_ASKPASS_REQUIRE=prefer ssh-agent -s`
fi

Plasma often hasn’t set DISPLAY yet at this point, and these tools generally require it to be set.

Using multiple SKs with one host

It’s a good idea to have a backup SK or two around; that way, if you lose or break your main SK, you can use another.

The ssh keypair that you generated with the ssh-keygen command above only works with the one SK it was generated with. So the simple solution is: just use multiple keypairs. For each SK, run ssh-keygen, saving the key into its own file.

When running ssh-add, you can specify all of them on the ssh-add command line. If you used the same ssh passphrase for all of them, then you only have to type it in once. Note that the SK doesn’t have to be present to load the key handle into ssh-agent; it only has to be present when it is used.

Now, add all the public keys to the authorized_keys file on the remote.

When you try to connect to the remote, only one SK need to present. You may see a message like:

sign_and_send_pubkey: signing failed for ED25519-SK "jgoerzen" from agent: agent refused operation

This is simply indicating an attempt to use a SK that wasn’t plugged in. You can ignore it, because ssh will skip (with that warning) the SKs that aren’t present and just use the one that is. You’ll touch it to proceed like usual.

Using multiple hosts with one SK

This is not a problem at all. With non-resident keys, nothing is actually stored on the SK itself. Use it with as many machines as you like and it’ll be fine.

Requiring a passphrase

You can require a passphrase (“PIN” in the SK world) in addition to a touch to access your key. This can work with both non-resident keys and the resident keys I’ll describe below.

If you generate a key adding -O verify-required to your ssh-keygen command line, ssh will require a PIN entry with every use.

Notably, the PIN entry happens on your local machine via a local ssh-askpass process, even if you are using agent forwarding with ssh -A. You can also add verify-required to your authorized_keys file for the corresponding key to force PIN input.

You can generate multiple keys, some with verification required and some without. Then you choose which connections merit the extra hassle.

I should note that if you use multiple different keys with a single machine as shown above, then this can be a bit annoying; you will be prompted for a PIN for each key before the system checks to see if that key is present.

Note that the PIN is a property of the SK, not of individual keys. The SK PIN applies to all verify-required keys, and changing the SK PIN changes the pin for all verify-required keys.

Setting the SK PIN

If you’re going to use a PIN and haven’t already set a FIDO2 PIN on your SK, you need to do that first. I’ll show you how to do it with the fido2-tools package. YubiKey owners can use this method, or you can also use ykman or ykman-gui to do so.

First, install fido2-tools.

Then, run fido2-token -L. Note the /dev/hidraw device number listed.

Now, run fido2-token -S /dev/hidrawXX (substituting in the /dev/hidraw number you noted before).

You can also use this process if you want to use a PIN with a non-resident key.

Resident vs. non-resident keys

Thus far, everything I’ve described has been about non-resident keys. I gave a warning up top – as ssh itself does – not to use resident keys. Compared to non-resident keys, these are the differences with resident keys:

  1. The ssh private key handle is stored within the SK
  2. It is possible to extract the key handle from the SK
  3. You can delete your local private key file and load resident keys with ssh-add -K. You can also regenerate the local public and private key files based on the data in the SK using ssh-keygen -K.
  4. You should generate the key with -O application=ssh:hostname
  5. There is a limit to the number of resident keys a SK can hold

The primary benefit of a resident key is that you can use it on multiple machines without needing to have the private key handle file on each machine. This weakens security, because an attacker would only need the SK to be able to authenticate as you. By adding -O verify-required, you can force the SK to prompt for a “PIN” (really a passphrase) before using it, if supported by the SK. This is strongly encouraged, because otherwise someone could simply steal your SK to use your private key.

Generating a resident key

If you’re really going to generate a resident key against my advice, use a command like:

ssh-keygen -t ed25519-sk -O resident -O verify-required -O application=ssh:hostname

As before, use ecdsa-sk instead of ed25519-sk if you get an error.

Because you are storing the key material in the SK itself, you need to give each key a unique name (“application”). These must start with ssh: for ssh. I recommend using the hostname, but you can use whatever you want here, so long as it is unique to each use of the SK.

You can see what keys are resident on your SK with fido2-token -L -r /dev/hidrawXX.

If you haven’t already set a FIDO2 PIN, read the section on that above before running these commands.

Using a resident key

You can use a resident key just like any other. You can also load it directly from the SK using ssh-add -K even if it isn’t on your local machine.

ssh-agent with resident keys

The ssh-askpass issue I described is more important now, because you really really should be using -O verify-required with your resident keys. This means that you can’t just ignore a missing ssh-agent prompt, since you have to actually give it a PIN. Make sure you have ssh-askpass configured and working before you try to use resident keys with ssh-agent.

Remote sudo

Let’s say that you have connected to a remote machine using your SK and now want to sudo to root. Well, normal sudo is a lot less secure than the setup here; after all, all it requires is a password, and we can do a lot better, right?

The best way is to add your SK public key to root’s ~/.ssh/authorized_keys. You can even, if you want, restrict it to localhost. Then you can use ssh agent forwarding and ssh root@localhost to achieve a protected local elevation to root, and delete sudo. Or just ssh directly into the root account.

Other fun tricks

Here are some other things you can do with SKs.

Headless servers

Let’s say that you have a server that uses SSH to connect to other systems to download backups. You probably store the SSH private key in a passwordless file. Maybe you already lock it down in those other systems’ authorized_keys file with a command= clause. But still, if an attacker obtains the private key file, they can access all your data from anywhere.

You can achieve some increase in security (though not as much as outlined above) by generating a touchless keypair. Plug a SK into the backup server. Now, when you run ssh-keygen, pass -O no-touch-required. You will also need to add no-touch-required to the authorized_keys on the target, because the ssh server rejects no-touch-required keys by default.

With this setup, if an attacker compromises the backup server, they can still use its credentials to access remotes – but only from that server, since they can’t actually steal the ssh private key. This is less secure than the standard touch-required methods, but more secure than just having the private key in plaintext on the drive.

Cryptographic signatures

You can use ssh to create signatures over data, and to verify them.

Troubleshooting

If you are using PINs and can’t get your PIN to work, it is possible you’ve locked the key with too many bad passphrase entries. With YubiKeys at least, you can reset the lock by just unplugging it and plugging it back in.

fido2-token has a lot of commands to interrogate what’s on the key, but this is mainly of use with resident keys.

You can always use ssh -v to see what ssh is doing. You may also consider specifying a certain key with ssh -i to make sure the correct key is being used. If you used to use a more standard ssh key and aren’t being required to touch the SK when logging in, you may still be using your prior key.

Other uses

You can also use SKs as primary or secondary factors in local logins, including with your graphical or text console logins, via PAM. You can also use them with LUKS to protect a disk. See Debian’s U2F page for more details (not specific to Debian).

You can also use a plugin for Age (Encryption) to perform encryption using your FIDO2 SK.

Many SKs include an OpenPGP app; there are a lot of guides out there on how to do this, such as this and this from Yubikey and this from Nitrokey. This lets you perform more versatile encrypting and signing using the SK, but doesn’t use FIDO2.

These are all beyond the scope of this article; I haven’t tested any of them.

Mobile apps

On Android, Termius is the only maintained program I could find that has support for FIDO2 ssh keys.

On iOS, Prompt and Blink seem to.

I haven’t tried any of these. It is unfortunate that none of them are Free Software.

Additional references

Here are some (potentially) interesting topics you can find here: