SSH certificates

7 min readOct 31, 2019

Update Nov 29, 2020: If your ssh client tells you check_host_cert: certificate signature algorithm ssh-rsa: signature algorithm not supported, see the end of this article.

For years I have been using SSH keys for password-less access to my computers at home. This method is superior to passwords, but has its flaws, especially in a datacenter with many actors. SSH certificates are a much better solution, but they are not that well documented.

So I decided to explore them. Due to sloppy reading and my ignorance in security matters, it took me a bit of trial and error before I successfully implemented SSH certificates. My mistakes were of two categories: Incorrect key signing, which leads to invalid certificates, and performing steps on the wrong computer. Since three computers (Client, Server, Trusted Server) are involved, it was easy to get confused.

Public key authentication without certificates

Let’s review how authentication works without certificates. A computer has one of two roles: Client or Server. A Client initiates an SSH connection to the Server. The sshd process runs on the Server.

First, the Server proves its identity by presenting its public host key. The Client may contain a copy of the Server’s key in its known_hosts file, in which case it trusts the Server and accepts authentication. If the Server’s key is not in known_hosts, the Client asks the user whether it should trust the Server:

The authenticity of host 'xxx' can't be established.
RSA key fingerprint is SHA256:kfcwi9X8T4nMRw1OM0xDXETGcqjU26/zbM+KqNB6CKw.
Are you sure you want to continue connecting (yes/no)?

Humans being fundamentally lazy, the usual user reaction is to type “yes” without checking the fingerprint. Unconditionally accepting the Server’s authentication request is one of the weaknesses of the non-certified public key protocol.

Next, the Client authenticates the user by sending the user’s public key to the Server. The Server accepts the key if it is contained in its authorized_keys file. This highlights two other weaknesses: How can users add their keys to the Server’s authorized_keys if they have no password access, and how will they manage the hundreds of copies of their key accumulating in the datacenter?

Another weakness: Keys never expire. Neither users nor hosts are forced to refresh their keys.

How certificates change the process

When certificates are used, user keys are not stored in the Server’s authorized_keys, and host keys are not stored in the Client’s known_hosts. So how are keys validated?

Here is where a third role comes in, the Trusted Server. Located on the Trusted Server is a private key named Certificate Authority (CA), which is used to digitally sign host and user keys. The result of the signing process is a so-called certificate. To authenticate, Client and Server now exchange certificates instead of keys. A certificate is then validated by the CA’s public key.

Certificates solve the above-mentioned weaknesses inherent in non-certified public key authentication: Certificates have an expiration date; there is no need to copy user keys to Servers and manage key copies, because user keys are not stored on Servers.

There is one inconvenience: Users can’t sign their public keys themselves, since they have no access to the private CA key. A process must be in place to get keys signed by a privileged operator or an application.

Note that SSH certification follows the OpenPGP standard, not SSL/TLS, and that the certificate format is OpenPGP, not X509.

Certificate authentication in practice

A few misunderstandings and my generally shallow knowledge of IT security caused me a bit of trial and error to get the process working. My steps for creating and using host and user certificates are below. To avoid the pitfalls that I stumbled into, pay attention to where the steps are executed (Client, Server, Trusted Server) and how certificates are signed.

Prequisite

On the Trusted Server, create a key pair for signing. I will call it CA:

ssh-keygen -f CA

This generates two files, private key CA and public key CA.pub. You can store them anywhere; I put then in /etc/ssh. Feel free to enter a passphrase.

Obviously, the private CA key must be kept secret. Consequently, non-privileged users are unable to sign their own keys.

Host certificates

Host certificates step 1: Sign host keys and create host certificates

On the Trusted Server, use private key CA to sign the public host key of each Server in the datacenter. The command is:

ssh-keygen -h -s CA -n LIST-OF-PRINCIPALS -I ID -V +52w KEYFILE.pub

The -h option is required for signing host keys. -s specifies the signature key (must be a private key). ID is meant to be a short, human-readable description of the certificate. -V is the expiration time; in the example, 52 weeks from now. KEYFILE.pub is the host key to be signed.

LIST-OF-PRINCIPALS is a comma-separated list of the domain names by which the Server is accessed, for example -n ceres,ceres.example.com. IP addresses don’t seem to work. I made my first mistake here: Instead of the respective domain names of each server, I entered the Trusted Server’s names in all certificates.

The result of this command is a new file named KEYFILE-cert.pub. This is the certificate the Server will use to prove its identity from now on. Store it under /etc/ssh on the Server.

Host certificates step 2: Configure sshd

Servers’ ssh daemons need to know that they have to present their host certificates to Clients. Add this clause to sshd_config on each Server:

HostCertificate /etc/ssh/KEYFILE-cert.pub

sshd must be restarted after changing its configuration.

Host certificates step 3: Tell Clients to trust the Trusted Server

When using non-certified public keys, Clients keep host keys that they trust in known_hosts. For example, a line like this:

ceres.example.com ssh-rsa AAAAB3Nza...

tells the Client that Server ceres.example.com should present the host key starting with ssh-rsa. If the Server presents a different host key, the Client will refuse to connect.

Instead of unsigned host keys, you want Clients to trust the CA from now on.

On all Clients, remove such host key lines from known_hosts. Don’t forget that there is a global /etc/ssh/ssh_known_hosts, and each user account may have its own ~/.ssh/known_hosts.

Instead of the host keys, include the CA in known_hosts on each Client:

@cert-authority LIST-OF-SERVERS ssh-rsa AAAAB3Nza...

LIST-OF-SERVERS is a comma-separated list of those Servers that signed their host key. Wildcards are permitted, for example *.example.com. The string starting with ssh-rsa is the public CA key.

Now, when establishing a connection to a Server that corresponds to the LIST-OF-SERVERS, this Client uses the public CA key to validate the host key.

Host certificate test

On a Client that trusts the CA (step 3), ssh to one of the Servers with signed host key. If you see the familiar message

The authenticity of host 'odroid (192.168.1.16)' can't be established.
RSA key fingerprint is SHA256:kfcwi9X8T4nMRw1OM0xDXETGcqjU26/zbM+KqNB6CKw.
Are you sure you want to continue connecting (yes/no)?

the test failed. Host certificates don’t require the user to confirm that this is the correct Server. ssh -vvv helps to understand what’s wrong.

User certificates

User certificate step 1: Sign the user’s public key

Since normal users don’t have access to the private CA key, they can’t sign their keys themselves. An administrator or a special application are required. The command to sign a user key is

ssh-keygen -s CA -I ID -n USERNAME -V +52w KEYFILE.pub

Compared to signing host keys, there are two differences: There is no -h option, and more importantly the username is used as the sole principal. Again, the command generates a certificate named KEYFILE-cert.pub. It needs to be stored on the client, for example in ~/.ssh.

Here is where I made my second major mistake. At first, I used the computer’s domain name as principal. ssh-keygen happily obliges and creates an invalid user certificate. It took me a while to find out that the principal of a user certificate is the user, not the host. Makes sense, doesn’t it! Unfortunately, the only error message I got was “invalid certificate”.

User certificate step 2: Install the CA’s public key

To allow a Server to validate a user certificate, it must have the CA’s public key. If you followed the Prerequisite instructions, this key is /etc/ssh/CA.pub on the Trusted Server. Copy it to /etc/ssh on each Server.

User certificate step 3: Tell sshd where to find the CA’s public key

On each Server, add this clause to sshd_config and restart sshd:

TrustedUserCAKeys /etc/ssh/CA.pub

User certificate test

Remove or rename authorized_keys on one of the Servers. From a Client that trusts the CA, ssh into the Server. Even though authorized_keys doesn’t exist anymore, you should be able to log on without password. The servers authentication log (probably /var/log/secure or /var/log/auth.log) should contain a message like this:

Accepted publickey for bbausch from 192.168.1.51 port 39498 ssh2: RSA-CERT ID odin-bbausch (serial 0) CA RSA SHA256:BmLWnPGoPg2Edyk2NsZGQ62lm7Cae6j5bOj3uKvXzcs

Remarks and tips

While this article deals with a single CA, there is no problem having several CAs in an environment.

Update Nov 29, 2020: RSA key signing deprecated

OpenSSH 8.2, released in February 2020, deprecates key signing using the ssh-rsa algorithm (see release notes). Consequently, newer ssh clients may issue the error check_host_cert: certificate signature algorithm ssh-rsa: signature algorithm not supported.

The secure solution is to update OpenSSH on the servers and generate new certificates. If that is too much hassle right now, you can tell your ssh client to accept RSA signing by adding this line to the global /etc/ssh/ssh_config or your account’s $HOME/.ssh/config:

CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2–512,rsa-sha2–256,ssh-rsa

Or by adding it to the command line:

ssh -oCASignatureAlgorithms=ssh-rsa user@server

I discovered this solution at https://ibug.io/blog/2020/04/ssh-8.2-rsa-ca.