By

Graham Land

·

SSH Certificate based Authentication – A Quick Guide

During the last decade we moved from telnet and clear text passwords to ssh and encrypted key pairs. This decade, with the rapid adoption of public clouds and micro-service architectures we need a more robust, scalable and manageable solution. SSH and key pairs are great for small deployments, however with the introduction of public clouds…

During the last decade we moved from telnet and clear text passwords to ssh and encrypted key pairs. This decade, with the rapid adoption of public clouds and micro-service architectures we need a more robust, scalable and manageable solution.

SSH and key pairs are great for small deployments, however with the introduction of public clouds and continuous deployment pipelines it becomes very challenging to manage your users’ public keys. When was the last time you audited your authorised_keys files across your estate to ensure all public keys were still valid? That contractor that needed access for a week….who remembered to remove their key?…and so on.

The big players in the cloud space today, like Netflix and Facebook, have shown us how they have tackled this challenge whilst still using SSH.

They use CA signed certificates to provide both Host and User Authenticity with SSH. By leveraging certificates we get the ability to create ephemeral, short lived Credentials. Your team could automatically get a newly signed certificate when they login in the mornings and have this expire after 8 hours. If a user’s certificate is compromised it can be revoked if it’s still valid.

Also, as you’ll see below, it’s much easier to manage SSH configuration for a fleet of host servers.

This guide is intended to help folks understand the mechanics involved in setting up certificate based ssh. For production situations typically an API based solution would be leveraged such as HashiCorp’s Vault – I’ll cover this in a future blogpost.

Certificate Authority (CA) Server Host Server(s) Client(s)
Host Server Certificate Configuration
This is the server typically managed by a security team. The root CA private keys are held on this server and should be protected. If these keys are compromised it will be necessary to Revoke & Rotate/Recreate ALL Certificates!! These are the servers that are being built or reprovisioned. The Host CA Signed Certificate is used to prove Host Authenticity to clients. It is sent to the ssh client during the initial handshake when a ssh client attempts to login. The user laptop or server that’s runing the ssh client. The Client CA Signed Certificate is used to prove Client Authenticity to the Host Server
Step 1. Create HOST CA signing keys : Example ssh-keygen -t rsa -N '' -C HOST-CA -b 4096 -f host-ca Step 2. Let’s generate a fresh set of ssh RSA HOST keys with 4096 bits. Typically the keys are generated by default when openssh-server is installed but it uses 2048 bits. You need to do this when cloning VMs too if you need unique authenticity : Example sudo ssh-keygen -N '' -C HOST-KEY -t rsa -b 4096 -h -f /etc/ssh/ssh_host_rsa_key
Step 3. Copy the PUBLIC key, user@target-host:/etc/ssh/ssh_host_rsa_key.pub, created in Step 2. on the host server to the CA server: Example scp root@192.168.9.200:/etc/ssh/ssh_host_rsa_key.pub .
Step 4. Create the CA signed Host Certificate for the target host using the CA-HOST private key, host-ca, created in Step 1., and the host server’s public key, ssh_host_rsa_key.pub, retrieved in Step 3 : Example ssh-keygen -s ../host-ca -I dev_host_server -h -V -5m:+52w ssh_host_rsa_key.pub
Step 5. Copy the HOST Certificate, ssh_host_rsa_key-cert.pub, created in Step 4., back onto the host server : Example scp ssh_host_rsa_key-cert.pub root@192.168.9.200:/etc/ssh/ssh_host_rsa_key-cert.pub
Step 6. Remove the now obsolete host public key and host cert from the CA server: Example rm ssh_host_rsa_key-cert.pub ssh_host_rsa_key.pub
Step 7. Configure the Host Server to use the new certificate file,/etc/ssh/ssh_host_rsa_key-cert.pub, within ssh server conf, /etc/ssh/sshd_config, by adding the following line HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub. Now restart the ssh service. Example grep -qxF 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' /etc/ssh/sshd_config || echo 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' | sudo tee -a /etc/ssh/sshd_config followed by sudo systemctl restart ssh
Step 8. Capture the contents of the CA-HOST PUBLIC key, host-ca.pub, as this will be needed to configure the ssh clients. Example cat host-ca.pub Step 9. Now we need to configure the ssh clients to be able to validate the Host Certificates using the CA-HOST PUBLIC key, host-ca.pub , created in Step 1. by adding it to the individual user’s ~/.ssh/known_hosts Example grep -qxF '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' ~/.ssh/known_hosts || echo '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' | tee -a ~/.ssh/known_hosts
Client Certificate Configuration
Step 10. Create Client CA signing keys : Example ssh-keygen -t rsa -N '' -C CLIENT-CA -b 4096 -f client-ca
Step 11. Copy the public Client CA signing key, client-ca.pub, created in Step 10. to the target host servers (NOT the client servers) Example scp client-ca.pub root@host:/etc/ssh/client-ca.pub Step 12. Configure the Host Server to use the new Client CA file, client-ca.pub, within ssh server conf, /etc/ssh/sshd_config, by adding the following line TrustedUserCAKeys /etc/ssh/client-ca.pub. Then restart the ssh service. Example grep -qxF 'TrustedUserCAKeys /etc/ssh/client-ca.pub' /etc/ssh/sshd_config || echo 'TrustedUserCAKeys /etc/ssh/client-ca.pub' | sudo tee -a /etc/ssh/sshd_config followed by sudo systemctl restart ssh
Step 13. Copy, fax, email or however the client(s) public ssh key, /home/someuser/.ssh/id_rsa.pub, to the CA server and sign the key as follows: Example ssh-keygen -s client-ca -I graham-dev -n root,vagrant,graham,pi -V -5:+52w -z 1 ~/.ssh/id_rsa.pub Step 14. Copy, fax, email or however the client(s) new ssh certificate, id_rsa-cert.pub, back to the clients /home/someuser/.ssh directory and test as follows: ssh -v root@192.168.9.200

Host Server Certificate Configuration

Step 1 on CA server – Create Host Signing Key

graham@graz-baz:~/.WiP $ ssh-keygen -t rsa -N '' -C HOST-CA -b 4096 -f host-ca
Generating public/private rsa key pair.
Your identification has been saved in host-ca.
Your public key has been saved in host-ca.pub.
The key fingerprint is:
SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 HOST-CA
The keys randomart image is:
+---[RSA 4096]----+
|.. .o. |
|.. .+. . |
|. . . .o ... |
| o . .. .o. . +|
|+ . +. S .o.. E.|
|+. o +. .= + .|
|+o ..... .. = . |
|B.o.o.. o . o |
|B=.o .o.. . |
+----[SHA256]-----+
graham@graz-baz:~/.WiP $ ls -al
total 16
drwx------ 2 graham graham 4096 Jan 3 12:35 .
drwxr-xr-x 19 graham graham 4096 Jan 3 12:14 ..
-rw------- 1 graham graham 3247 Jan 3 12:35 host-ca
-rw-r--r-- 1 graham graham 733 Jan 3 12:35 host-ca.pub
graham@graz-baz:~/.WiP $

Notes:

  • omit -N if you want to include a passphrase with the key generation
  • -t type can be dsa, rsa, ecdsa or ed25519. I choose rsa as it’s widely accepted everywhere though less secure.
  • -b key size of 4096 bits to delay brute force attacks – all bets are off when we have qbit mobile phones 😉

Step 2 on target host server(s) – Create a new host ssh key pair (optional, if keys exist)

root@redis01:# sudo ssh-keygen -N '' -C HOST-KEY -t rsa -b 4096 -h -f /etc/ssh/ssh_host_rsa_key
Generating public/private rsa key pair.
/etc/ssh/ssh_host_rsa_key already exists.
Overwrite (y/n)? y
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc HOST-KEY
The keys randomart image is:
+---[RSA 4096]----+
| .+O=.. |
| ==*== |
| . *o*.o |
| ...Eo . o|
| .+S O . ..|
| .=o* = =.. |
| .+ + o =. |
| .. o .. |
| .o .. |
+----[SHA256]-----+
root@redis01:~# ls -al /etc/ssh/
total 596
drwxr-xr-x 2 root root 4096 Jan 2 22:51 .
drwxr-xr-x 89 root root 4096 Jan 2 23:57 ..
-rw-r--r-- 1 root root 384 Jan 2 22:51 hashistack-ca.pub
-rw-r--r-- 1 root root 553122 Mar 4 2019 moduli
-rw-r--r-- 1 root root 1580 Mar 4 2019 ssh_config
-rw-r--r-- 1 root root 3362 Jan 2 22:51 sshd_config
-rw------- 1 root root 227 Jan 2 22:45 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 1083 Jan 2 22:51 ssh_host_ecdsa_key-cert.pub
-rw-r--r-- 1 root root 174 Jan 2 22:45 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 Jan 2 22:45 ssh_host_ed25519_key
-rw-r--r-- 1 root root 94 Jan 2 22:45 ssh_host_ed25519_key.pub
-rw------- 1 root root 3243 Jan 3 13:38 ssh_host_rsa_key
-rw-r--r-- 1 root root 734 Jan 3 13:38 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 338 Jan 2 22:45 ssh_import_id
root@redis01:~# date
Fri Jan 3 13:38:52 UTC 2020
root@redis01:~#

Notes:

  • For my dev environment I use the same host keys and certificates on all host servers (cloned from the same template). Possibly not a good idea for production but better than what I've got today.
  • HashiCorp's Vault provides a neat API based PKI CA signing solution when you need to scale and maintain certificate management and auditability. This will be implemented next…

Step 3 on CA server – Copy over public host key(s)

graham@graz-baz:~/.WiP $ mkdir tmp-keys
graham@graz-baz:~/.WiP $ cd tmp-keys/
graham@graz-baz:~/.WiP/tmp-keys $ scp root@192.168.9.200:/etc/ssh/ssh_host_rsa_key.pub .
ssh_host_rsa_key.pub 100% 734 416.9KB/s 00:00
graham@graz-baz:~/.WiP/tmp-keys $ ls -al
total 12
drwxr-xr-x 2 graham graham 4096 Jan 3 13:44 .
drwx------ 3 graham graham 4096 Jan 3 13:42 ..
-rw-r--r-- 1 graham graham 734 Jan 3 13:44 ssh_host_rsa_key.pub
graham@graz-baz:~/.WiP/tmp-keys $

Step 4 on CA server – Create signed ssh host certificate

graham@graz-baz:~/.WiP/tmp-keys $ ssh-keygen -s ../host-ca -I dev_host_server -h -V -5m:+52w ssh_host_rsa_key.pub
Signed host key ssh_host_rsa_key-cert.pub: id "dev_host_server" serial 0 valid from 2020-01-03T16:51:35 to 2021-01-01T16:56:35
...
graham@graz-baz:~/.WiP/tmp-keys $ date
Fri 3 Jan 14:15:47 GMT 2020
graham@graz-baz:~/.WiP/tmp-keys $ ls -al
total 16
drwxr-xr-x 2 graham graham 4096 Jan 3 14:15 .
drwx------ 3 graham graham 4096 Jan 3 13:42 ..
-rw-r--r-- 1 graham graham 2359 Jan 3 14:15 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 graham graham 734 Jan 3 13:44 ssh_host_rsa_key.pub
graham@graz-baz:~/.WiP/tmp-keys $

Step 5 on CA server – copy new ssh host certificate back to host

On CA Server

graham@graz-baz:~/.WiP/tmp-keys $ scp ssh_host_rsa_key-cert.pub root@192.168.9.200:/etc/ssh/ssh_host_rsa_key-cert.pub
ssh_host_rsa_key-cert.pub 100% 2359 973.0KB/s 00:00
graham@graz-baz:~/.WiP/tmp-keys $

On Host Server

root@redis01:~# ls -al /etc/ssh/
total 600
drwxr-xr-x 2 root root 4096 Jan 3 14:38 .
drwxr-xr-x 89 root root 4096 Jan 2 23:57 ..
-rw-r--r-- 1 root root 384 Jan 2 22:51 hashistack-ca.pub
-rw-r--r-- 1 root root 553122 Mar 4 2019 moduli
-rw-r--r-- 1 root root 1580 Mar 4 2019 ssh_config
-rw-r--r-- 1 root root 3362 Jan 2 22:51 sshd_config
-rw------- 1 root root 227 Jan 2 22:45 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 1083 Jan 2 22:51 ssh_host_ecdsa_key-cert.pub
-rw-r--r-- 1 root root 174 Jan 2 22:45 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 Jan 2 22:45 ssh_host_ed25519_key
-rw-r--r-- 1 root root 94 Jan 2 22:45 ssh_host_ed25519_key.pub
-rw------- 1 root root 3243 Jan 3 13:38 ssh_host_rsa_key
-rw-r--r-- 1 root root 2359 Jan 3 14:38 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 root root 734 Jan 3 13:38 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 338 Jan 2 22:45 ssh_import_id
root@redis01:~#

Step 6 on CA server – tidy up

graham@graz-baz:~/.WiP/tmp-keys $ ls -al
total 16
drwxr-xr-x 2 graham graham 4096 Jan 3 14:15 .
drwx------ 3 graham graham 4096 Jan 3 13:42 ..
-rw-r--r-- 1 graham graham 2359 Jan 3 14:15 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 graham graham 734 Jan 3 13:44 ssh_host_rsa_key.pub
graham@graz-baz:~/.WiP/tmp-keys $ rm ssh_host_rsa_key.pub ssh_host_rsa_key-cert.pub
graham@graz-baz:~/.WiP/tmp-keys $ ls -al
total 8
drwxr-xr-x 2 graham graham 4096 Jan 3 15:41 .
drwx------ 3 graham graham 4096 Jan 3 13:42 ..
graham@graz-baz:~/.WiP/tmp-keys $

Step 7 on Host server – Configure ssh to use new host certificate

root@redis01:~# grep -qxF 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' /etc/ssh/sshd_config || echo 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' | sudo tee -a /etc/ssh/sshd_config
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
root@redis01:~# grep HostCertificate /etc/ssh/sshd_config
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
root@redis01:~# sudo systemctl restart ssh
root@redis01:~# sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-01-03 15:33:29 UTC; 6s ago
Process: 31111 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 31112 (sshd)
Tasks: 1 (limit: 1112)
CGroup: /system.slice/ssh.service
└─31112 /usr/sbin/sshd -D

Jan 03 15:33:29 redis01 systemd[1]: Stopping OpenBSD Secure Shell server...
Jan 03 15:33:29 redis01 systemd[1]: Stopped OpenBSD Secure Shell server.
Jan 03 15:33:29 redis01 systemd[1]: Starting OpenBSD Secure Shell server...
Jan 03 15:33:29 redis01 sshd[31112]: Server listening on 0.0.0.0 port 22.
Jan 03 15:33:29 redis01 sshd[31112]: Server listening on :: port 22.
Jan 03 15:33:29 redis01 systemd[1]: Started OpenBSD Secure Shell server.
root@redis01:~#

Step 8 on CA server – Capture details of Host CA public key

graham@graz-baz:~/.WiP $ ls
host-ca host-ca.pub tmp-keys
graham@graz-baz:~/.WiP $ cat host-ca.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA

Step 9 on Client(s) – Configure Host CA public key

Before the host-ca public key is added:

graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ ssh -v root@192.168.9.200
OpenSSH_7.4p1 Raspbian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.9.200 [192.168.9.200] port 22.
debug1: Connection established.
debug1: identity file /home/graham/.ssh/id_rsa type 1
debug1: identity file /home/graham/.ssh/id_rsa-cert type 5
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.9.200:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc, serial 0 ID "dev_host_server" CA ssh-rsa SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 valid from 2020-01-03T14:10:32 to 2021-01-01T14:15:32

debug1: No matching CA found. Retry with plain key

The authenticity of host '192.168.9.200 (192.168.9.200)' cant be established.

RSA key fingerprint is SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc.

Are you sure you want to continue connecting (yes/no)?

Adding the host-ca public key :

graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ grep -qxF 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' ~/.ssh/known_hosts || echo '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' | tee -a ~/.ssh/known_hosts
@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA
graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ cat ~/.ssh/known_hosts
@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA
@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA
graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $

And the big test…

graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ ssh -v root@192.168.9.200
OpenSSH_7.4p1 Raspbian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.9.200 [192.168.9.200] port 22.
debug1: Connection established.
debug1: identity file /home/graham/.ssh/id_rsa type 1
debug1: identity file /home/graham/.ssh/id_rsa-cert type 5
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/graham/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.9.200:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc, serial 0 ID "dev_host_server" CA ssh-rsa SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 valid from 2020-01-03T16:51:35 to 2021-01-01T16:56:35

debug1: Host '192.168.9.200' is known and matches the RSA-CERT host certificate.

debug1: Found CA key in /home/graham/.ssh/known_hosts:1

debug1: rekey after 134217728 blocks

Client Certificate Configuration

Step 10 on CA server – Create a client CA signing key

$ ssh-keygen -t rsa -N '' -C CLIENT-CA -b 4096 -f client-ca
Generating public/private rsa key pair.
Your identification has been saved in client-ca.
Your public key has been saved in client-ca.pub.
The key fingerprint is:
SHA256:bDOWewefRFdvXDGQixnVgVjnHHPSjOLeO381BiAPF/Q CLIENT-CA
The keys randomart image is:
+---[RSA 4096]----+
| .o=+*@*|
| o =oo==X|
| =.*Eoo+|
| . . =.+ . |
| S ..... |
| o + +...o.|
| . . + ..o|
| . . o .|
| oo|
+----[SHA256]-----+
~/hashistack-ca
$ ls -al
total 48
drwx------ 8 grazzer staff 256 3 Jan 21:30 .
drwxr-xr-x+ 106 grazzer staff 3392 3 Jan 17:21 ..
-rw-r--r--@ 1 grazzer staff 6148 3 Jan 21:00 .DS_Store
-rw------- 1 grazzer staff 3369 3 Jan 21:30 client-ca
-rw------- 1 grazzer staff 735 3 Jan 21:30 client-ca.pub
-rw------- 1 grazzer staff 3247 3 Jan 20:46 host-ca
-rw------- 1 grazzer staff 733 3 Jan 20:46 host-ca.pub
drwx------ 5 grazzer staff 160 3 Jan 20:50 tmp-keys
~/hashistack-ca
$

Step 11 on CA server – Copy public client CA signing key to ALL target hosts

$ scp client-ca.pub root@192.168.9.200:/etc/ssh/client-ca.pub
client-ca.pub
$

Step 12 on target Hosts – Configure to use new public client CA key

vagrant@redis01:~$ sudo mv .ssh/client-ca.pub /etc/ssh/.
vagrant@redis01:~$ chmod 644 /etc/ssh/client-ca.pub
vagrant@redis01:~$ grep -qxF 'TrustedUserCAKeys /etc/ssh/client-ca.pub' /etc/ssh/sshd_config || echo 'TrustedUserCAKeys /etc/ssh/client-ca.pub' | sudo tee -a /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/client-ca.pub
vagrant@redis01:~$ sudo systemctl restart ssh
vagrant@redis01:~$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-01-03 22:18:58 UTC; 30s ago
Process: 29741 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 29743 (sshd)
Tasks: 1 (limit: 1112)
CGroup: /system.slice/ssh.service
└─29743 /usr/sbin/sshd -D

Jan 03 22:18:58 redis01 systemd[1]: Starting OpenBSD Secure Shell server...
Jan 03 22:18:58 redis01 sshd[29743]: Server listening on 0.0.0.0 port 22.
Jan 03 22:18:58 redis01 sshd[29743]: Server listening on :: port 22.
Jan 03 22:18:58 redis01 systemd[1]: Started OpenBSD Secure Shell server.
vagrant@redis01:~$

Step 13 on CA server – Create individual client signed certificates

$ ssh-keygen -s ../client-ca -I graham-dev -n root,vagrant,graham,pi -V -5:+52w -z 1 id_rsa.pub
Signed user key id_rsa-cert.pub: id "graham-dev" serial 1 for root,vagrant,graham,pi valid from 2020-01-03T23:05:07 to 2021-01-01T23:05:12
~/hashistack-ca/client-key
$ scp id_rsa-cert.pub graham@192.168.1.199:/home/graham/.ssh/id_rsa-cert.pub
id_rsa-cert.pub 100% 2562 136.2KB/s 00:00
~/hashistack-ca/client-key
$

Step 14 on client(s) – Test new ssh certificate access

$ ssh -v root@192.168.9.200
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to 192.168.9.200 [192.168.9.200] port 22.
debug1: Connection established.
debug1: identity file /Users/grazzer/.ssh/id_rsa type 0
debug1: identity file /Users/grazzer/.ssh/id_rsa-cert type 4
debug1: identity file /Users/grazzer/.ssh/id_dsa type -1
debug1: identity file /Users/grazzer/.ssh/id_dsa-cert type -1
debug1: identity file /Users/grazzer/.ssh/id_ecdsa type -1
debug1: identity file /Users/grazzer/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/grazzer/.ssh/id_ed25519 type -1
debug1: identity file /Users/grazzer/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/grazzer/.ssh/id_xmss type -1
debug1: identity file /Users/grazzer/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to 192.168.9.200:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc, serial 0 ID "dev_host_server" CA ssh-rsa SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 valid from 2020-01-03T16:51:35 to 2021-01-01T16:56:35

debug1: Host '192.168.9.200' is known and matches the RSA-CERT host certificate.

debug1: Found CA key in /Users/grazzer/.ssh/known_hosts:1

debug1: rekey after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: rekey after 134217728 blocks

debug1: Will attempt key: /Users/grazzer/.ssh/id_rsa RSA SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc

debug1: Will attempt key: /Users/grazzer/.ssh/id_rsa RSA-CERT SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc

debug1: Will attempt key: /Users/grazzer/.ssh/id_dsa

debug1: Will attempt key: /Users/grazzer/.ssh/id_ecdsa

debug1: Will attempt key: /Users/grazzer/.ssh/id_ed25519

debug1: Will attempt key: /Users/grazzer/.ssh/id_xmss

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=ssh-ed25519

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: publickey

debug1: Offering public key: /Users/grazzer/.ssh/id_rsa RSA SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc

debug1: Authentications that can continue: publickey,password

debug1: Offering public key: /Users/grazzer/.ssh/id_rsa RSA-CERT SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc

debug1: Server accepts key: /Users/grazzer/.ssh/id_rsa RSA-CERT SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc

debug1: Authentication succeeded (publickey).

Authenticated to 192.168.9.200 ([192.168.9.200]:22).

debug1: channel 0: new [client-session]

debug1: Requesting no-more-sessions@openssh.com

debug1: Entering interactive session.

debug1: pledge: network

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

debug1: Sending environment.

debug1: Sending env LANG = en_GB.UTF-8

debug1: Sending env LC_TERMINAL_VERSION = 3.3.6

debug1: Sending env LC_TERMINAL = iTerm2

Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

* Overheard at KubeCon: "microk8s.status just blew my mind".

https://microk8s.io/docs/commands#microk8s.status

* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
Last login: Fri Jan 3 22:46:44 2020 from 192.168.2.101
root@redis01:~#

Note:

The certificate generated above is Valid -V from 5 minutes ago for the next 30 days -5m:+30d. Ideally keep this as short as possible – for my dev/play environment security is not a real concern – 30 days is fine.

HTH

Graham

Tags: