During the last decade we moved from telnet and clear text passwords to ssh and encrypted key pairs. This decade, with the rapid adoption of public clouds and micro-service architectures we need a more robust, scalable and manageable solution.
SSH and key pairs are great for small deployments, however with the introduction of public clouds and continuous deployment pipelines it becomes very challenging to manage your users’ public keys. When was the last time you audited your authorised_keys files across your estate to ensure all public keys were still valid? That contractor that needed access for a week….who remembered to remove their key?…and so on.
The big players in the cloud space today, like Netflix and Facebook, have shown us how they have tackled this challenge whilst still using SSH.
They use CA signed certificates to provide both Host and User Authenticity with SSH. By leveraging certificates we get the ability to create ephemeral, short lived Credentials. Your team could automatically get a newly signed certificate when they login in the mornings and have this expire after 8 hours. If a user’s certificate is compromised it can be revoked if it’s still valid.
Also, as you’ll see below, it’s much easier to manage SSH configuration for a fleet of host servers.
This guide is intended to help folks understand the mechanics involved in setting up certificate based ssh. For production situations typically an API based solution would be leveraged such as HashiCorp’s Vault – I’ll cover this in a future blogpost.
| Certificate Authority (CA) Server | Host Server(s) | Client(s) |
|---|---|---|
| Host Server Certificate Configuration | ||
| This is the server typically managed by a security team. The root CA private keys are held on this server and should be protected. If these keys are compromised it will be necessary to Revoke & Rotate/Recreate ALL Certificates!! | These are the servers that are being built or reprovisioned. The Host CA Signed Certificate is used to prove Host Authenticity to clients. It is sent to the ssh client during the initial handshake when a ssh client attempts to login. | The user laptop or server that’s runing the ssh client. The Client CA Signed Certificate is used to prove Client Authenticity to the Host Server |
Step 1. Create HOST CA signing keys : Example ssh-keygen -t rsa -N '' -C HOST-CA -b 4096 -f host-ca |
Step 2. Let’s generate a fresh set of ssh RSA HOST keys with 4096 bits. Typically the keys are generated by default when openssh-server is installed but it uses 2048 bits. You need to do this when cloning VMs too if you need unique authenticity : Example sudo ssh-keygen -N '' -C HOST-KEY -t rsa -b 4096 -h -f /etc/ssh/ssh_host_rsa_key |
|
Step 3. Copy the PUBLIC key, user@target-host:/etc/ssh/ssh_host_rsa_key.pub, created in Step 2. on the host server to the CA server: Example scp root@192.168.9.200:/etc/ssh/ssh_host_rsa_key.pub . |
||
Step 4. Create the CA signed Host Certificate for the target host using the CA-HOST private key, host-ca, created in Step 1., and the host server’s public key, ssh_host_rsa_key.pub, retrieved in Step 3 : Example ssh-keygen -s ../host-ca -I dev_host_server -h -V -5m:+52w ssh_host_rsa_key.pub |
||
Step 5. Copy the HOST Certificate, ssh_host_rsa_key-cert.pub, created in Step 4., back onto the host server : Example scp ssh_host_rsa_key-cert.pub root@192.168.9.200:/etc/ssh/ssh_host_rsa_key-cert.pub |
||
Step 6. Remove the now obsolete host public key and host cert from the CA server: Example rm ssh_host_rsa_key-cert.pub ssh_host_rsa_key.pub |
||
Step 7. Configure the Host Server to use the new certificate file,/etc/ssh/ssh_host_rsa_key-cert.pub, within ssh server conf, /etc/ssh/sshd_config, by adding the following line HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub. Now restart the ssh service. Example grep -qxF 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' /etc/ssh/sshd_config || echo 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' | sudo tee -a /etc/ssh/sshd_config followed by sudo systemctl restart ssh |
||
Step 8. Capture the contents of the CA-HOST PUBLIC key, host-ca.pub, as this will be needed to configure the ssh clients. Example cat host-ca.pub |
Step 9. Now we need to configure the ssh clients to be able to validate the Host Certificates using the CA-HOST PUBLIC key, host-ca.pub , created in Step 1. by adding it to the individual user’s ~/.ssh/known_hosts Example grep -qxF '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' ~/.ssh/known_hosts || echo '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' | tee -a ~/.ssh/known_hosts |
|
| Client Certificate Configuration | ||
Step 10. Create Client CA signing keys : Example ssh-keygen -t rsa -N '' -C CLIENT-CA -b 4096 -f client-ca |
||
Step 11. Copy the public Client CA signing key, client-ca.pub, created in Step 10. to the target host servers (NOT the client servers) Example scp client-ca.pub root@host:/etc/ssh/client-ca.pub |
Step 12. Configure the Host Server to use the new Client CA file, client-ca.pub, within ssh server conf, /etc/ssh/sshd_config, by adding the following line TrustedUserCAKeys /etc/ssh/client-ca.pub. Then restart the ssh service. Example grep -qxF 'TrustedUserCAKeys /etc/ssh/client-ca.pub' /etc/ssh/sshd_config || echo 'TrustedUserCAKeys /etc/ssh/client-ca.pub' | sudo tee -a /etc/ssh/sshd_config followed by sudo systemctl restart ssh |
|
Step 13. Copy, fax, email or however the client(s) public ssh key, /home/someuser/.ssh/id_rsa.pub, to the CA server and sign the key as follows: Example ssh-keygen -s client-ca -I graham-dev -n root,vagrant,graham,pi -V -5:+52w -z 1 ~/.ssh/id_rsa.pub |
Step 14. Copy, fax, email or however the client(s) new ssh certificate, id_rsa-cert.pub, back to the clients /home/someuser/.ssh directory and test as follows: ssh -v root@192.168.9.200 |
Host Server Certificate Configuration
Step 1 on CA server – Create Host Signing Key
graham@graz-baz:~/.WiP $ ssh-keygen -t rsa -N '' -C HOST-CA -b 4096 -f host-ca Generating public/private rsa key pair. Your identification has been saved in host-ca. Your public key has been saved in host-ca.pub. The key fingerprint is: SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 HOST-CA The keys randomart image is: +---[RSA 4096]----+ |.. .o. | |.. .+. . | |. . . .o ... | | o . .. .o. . +| |+ . +. S .o.. E.| |+. o +. .= + .| |+o ..... .. = . | |B.o.o.. o . o | |B=.o .o.. . | +----[SHA256]-----+ graham@graz-baz:~/.WiP $ ls -al total 16 drwx------ 2 graham graham 4096 Jan 3 12:35 . drwxr-xr-x 19 graham graham 4096 Jan 3 12:14 .. -rw------- 1 graham graham 3247 Jan 3 12:35 host-ca -rw-r--r-- 1 graham graham 733 Jan 3 12:35 host-ca.pub graham@graz-baz:~/.WiP $
Notes:
- omit
-Nif you want to include a passphrase with the key generation -ttype can bedsa,rsa,ecdsaored25519. I choose rsa as it’s widely accepted everywhere though less secure.-bkey size of 4096 bits to delay brute force attacks – all bets are off when we have qbit mobile phones 😉
Step 2 on target host server(s) – Create a new host ssh key pair (optional, if keys exist)
root@redis01:# sudo ssh-keygen -N '' -C HOST-KEY -t rsa -b 4096 -h -f /etc/ssh/ssh_host_rsa_key Generating public/private rsa key pair. /etc/ssh/ssh_host_rsa_key already exists. Overwrite (y/n)? y Your identification has been saved in /etc/ssh/ssh_host_rsa_key. Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. The key fingerprint is: SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc HOST-KEY The keys randomart image is: +---[RSA 4096]----+ | .+O=.. | | ==*== | | . *o*.o | | ...Eo . o| | .+S O . ..| | .=o* = =.. | | .+ + o =. | | .. o .. | | .o .. | +----[SHA256]-----+ root@redis01:~# ls -al /etc/ssh/ total 596 drwxr-xr-x 2 root root 4096 Jan 2 22:51 . drwxr-xr-x 89 root root 4096 Jan 2 23:57 .. -rw-r--r-- 1 root root 384 Jan 2 22:51 hashistack-ca.pub -rw-r--r-- 1 root root 553122 Mar 4 2019 moduli -rw-r--r-- 1 root root 1580 Mar 4 2019 ssh_config -rw-r--r-- 1 root root 3362 Jan 2 22:51 sshd_config -rw------- 1 root root 227 Jan 2 22:45 ssh_host_ecdsa_key -rw-r--r-- 1 root root 1083 Jan 2 22:51 ssh_host_ecdsa_key-cert.pub -rw-r--r-- 1 root root 174 Jan 2 22:45 ssh_host_ecdsa_key.pub -rw------- 1 root root 399 Jan 2 22:45 ssh_host_ed25519_key -rw-r--r-- 1 root root 94 Jan 2 22:45 ssh_host_ed25519_key.pub -rw------- 1 root root 3243 Jan 3 13:38 ssh_host_rsa_key -rw-r--r-- 1 root root 734 Jan 3 13:38 ssh_host_rsa_key.pub -rw-r--r-- 1 root root 338 Jan 2 22:45 ssh_import_id root@redis01:~# date Fri Jan 3 13:38:52 UTC 2020 root@redis01:~#
Notes:
- For my dev environment I use the same host keys and certificates on all host servers (cloned from the same template). Possibly not a good idea for production but better than what I've got today.
- HashiCorp's Vault provides a neat API based PKI CA signing solution when you need to scale and maintain certificate management and auditability. This will be implemented next…
Step 3 on CA server – Copy over public host key(s)
graham@graz-baz:~/.WiP $ mkdir tmp-keys graham@graz-baz:~/.WiP $ cd tmp-keys/ graham@graz-baz:~/.WiP/tmp-keys $ scp root@192.168.9.200:/etc/ssh/ssh_host_rsa_key.pub . ssh_host_rsa_key.pub 100% 734 416.9KB/s 00:00 graham@graz-baz:~/.WiP/tmp-keys $ ls -al total 12 drwxr-xr-x 2 graham graham 4096 Jan 3 13:44 . drwx------ 3 graham graham 4096 Jan 3 13:42 .. -rw-r--r-- 1 graham graham 734 Jan 3 13:44 ssh_host_rsa_key.pub graham@graz-baz:~/.WiP/tmp-keys $
Step 4 on CA server – Create signed ssh host certificate
graham@graz-baz:~/.WiP/tmp-keys $ ssh-keygen -s ../host-ca -I dev_host_server -h -V -5m:+52w ssh_host_rsa_key.pub Signed host key ssh_host_rsa_key-cert.pub: id "dev_host_server" serial 0 valid from 2020-01-03T16:51:35 to 2021-01-01T16:56:35 ... graham@graz-baz:~/.WiP/tmp-keys $ date Fri 3 Jan 14:15:47 GMT 2020 graham@graz-baz:~/.WiP/tmp-keys $ ls -al total 16 drwxr-xr-x 2 graham graham 4096 Jan 3 14:15 . drwx------ 3 graham graham 4096 Jan 3 13:42 .. -rw-r--r-- 1 graham graham 2359 Jan 3 14:15 ssh_host_rsa_key-cert.pub -rw-r--r-- 1 graham graham 734 Jan 3 13:44 ssh_host_rsa_key.pub graham@graz-baz:~/.WiP/tmp-keys $
Step 5 on CA server – copy new ssh host certificate back to host
On CA Server
graham@graz-baz:~/.WiP/tmp-keys $ scp ssh_host_rsa_key-cert.pub root@192.168.9.200:/etc/ssh/ssh_host_rsa_key-cert.pub ssh_host_rsa_key-cert.pub 100% 2359 973.0KB/s 00:00 graham@graz-baz:~/.WiP/tmp-keys $
On Host Server
root@redis01:~# ls -al /etc/ssh/ total 600 drwxr-xr-x 2 root root 4096 Jan 3 14:38 . drwxr-xr-x 89 root root 4096 Jan 2 23:57 .. -rw-r--r-- 1 root root 384 Jan 2 22:51 hashistack-ca.pub -rw-r--r-- 1 root root 553122 Mar 4 2019 moduli -rw-r--r-- 1 root root 1580 Mar 4 2019 ssh_config -rw-r--r-- 1 root root 3362 Jan 2 22:51 sshd_config -rw------- 1 root root 227 Jan 2 22:45 ssh_host_ecdsa_key -rw-r--r-- 1 root root 1083 Jan 2 22:51 ssh_host_ecdsa_key-cert.pub -rw-r--r-- 1 root root 174 Jan 2 22:45 ssh_host_ecdsa_key.pub -rw------- 1 root root 399 Jan 2 22:45 ssh_host_ed25519_key -rw-r--r-- 1 root root 94 Jan 2 22:45 ssh_host_ed25519_key.pub -rw------- 1 root root 3243 Jan 3 13:38 ssh_host_rsa_key -rw-r--r-- 1 root root 2359 Jan 3 14:38 ssh_host_rsa_key-cert.pub -rw-r--r-- 1 root root 734 Jan 3 13:38 ssh_host_rsa_key.pub -rw-r--r-- 1 root root 338 Jan 2 22:45 ssh_import_id root@redis01:~#
Step 6 on CA server – tidy up
graham@graz-baz:~/.WiP/tmp-keys $ ls -al total 16 drwxr-xr-x 2 graham graham 4096 Jan 3 14:15 . drwx------ 3 graham graham 4096 Jan 3 13:42 .. -rw-r--r-- 1 graham graham 2359 Jan 3 14:15 ssh_host_rsa_key-cert.pub -rw-r--r-- 1 graham graham 734 Jan 3 13:44 ssh_host_rsa_key.pub graham@graz-baz:~/.WiP/tmp-keys $ rm ssh_host_rsa_key.pub ssh_host_rsa_key-cert.pub graham@graz-baz:~/.WiP/tmp-keys $ ls -al total 8 drwxr-xr-x 2 graham graham 4096 Jan 3 15:41 . drwx------ 3 graham graham 4096 Jan 3 13:42 .. graham@graz-baz:~/.WiP/tmp-keys $
Step 7 on Host server – Configure ssh to use new host certificate
root@redis01:~# grep -qxF 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' /etc/ssh/sshd_config || echo 'HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub' | sudo tee -a /etc/ssh/sshd_config HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub root@redis01:~# grep HostCertificate /etc/ssh/sshd_config HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub root@redis01:~# sudo systemctl restart ssh root@redis01:~# sudo systemctl status ssh ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2020-01-03 15:33:29 UTC; 6s ago Process: 31111 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) Main PID: 31112 (sshd) Tasks: 1 (limit: 1112) CGroup: /system.slice/ssh.service └─31112 /usr/sbin/sshd -D Jan 03 15:33:29 redis01 systemd[1]: Stopping OpenBSD Secure Shell server... Jan 03 15:33:29 redis01 systemd[1]: Stopped OpenBSD Secure Shell server. Jan 03 15:33:29 redis01 systemd[1]: Starting OpenBSD Secure Shell server... Jan 03 15:33:29 redis01 sshd[31112]: Server listening on 0.0.0.0 port 22. Jan 03 15:33:29 redis01 sshd[31112]: Server listening on :: port 22. Jan 03 15:33:29 redis01 systemd[1]: Started OpenBSD Secure Shell server. root@redis01:~#
Step 8 on CA server – Capture details of Host CA public key
graham@graz-baz:~/.WiP $ ls host-ca host-ca.pub tmp-keys graham@graz-baz:~/.WiP $ cat host-ca.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA
Step 9 on Client(s) – Configure Host CA public key
Before the host-ca public key is added:
graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ ssh -v root@192.168.9.200 OpenSSH_7.4p1 Raspbian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to 192.168.9.200 [192.168.9.200] port 22. debug1: Connection established. debug1: identity file /home/graham/.ssh/id_rsa type 1 debug1: identity file /home/graham/.ssh/id_rsa-cert type 5 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u7 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000 debug1: Authenticating to 192.168.9.200:22 as 'root' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc, serial 0 ID "dev_host_server" CA ssh-rsa SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 valid from 2020-01-03T14:10:32 to 2021-01-01T14:15:32 debug1: No matching CA found. Retry with plain key The authenticity of host '192.168.9.200 (192.168.9.200)' cant be established. RSA key fingerprint is SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc. Are you sure you want to continue connecting (yes/no)?
Adding the host-ca public key :
graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ grep -qxF 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' ~/.ssh/known_hosts || echo '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA' | tee -a ~/.ssh/known_hosts @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ cat ~/.ssh/known_hosts @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmZo/bkvhmUEjx3erXC+rZ1R3htLHtz0VzZNpgQD2sT2KZLW3yBiKYIKgxICM04MQsVHY1k5y4ek/tgnw05m5KOO5KTHxxKjcBKf2EyvwG0o8vnzo6UgweqXEePigAzSGQfcsGp75tVu3qmeLKXtJOo1WaWmTSNH4Qoq89xRiPslCVDi1i2VEPxJi3+eeFL5WO+nBK9Xt28DaXY4B43sgC1KC6DSRUR2JhlgPGMKP2eTE5+UaEldyPVzdIl2j3tLsaURfr+cZ6ryPEE9phT1bjcOSC3A88NrROZH1FvpZpG6NQPXusTWjre/NIz2TdG44AopbFRKAEpMVFw67AJ6oDWHPTrh2TGh3SQEIIZTdhudZIHnwiSBuKUOqyV65KH/mmy5gr8X2miHbM+qh6ISjqwPN6TjAhUPgkjxtwa7K+tDseBoFsrRIgP65hHAIlEFodHUI8Lu3P5HswH39z8ouEDR+qU54z9JO/E0Mw9YQPk19A6jr7o9/06wqSXfkVmS1VwvyZI90Zqrtg4+lZ3Zq/GLDqpxTlakfEAddOd9Ns01GgeSab4mKDwB6r2VTsunXQ4DDJkzxm9ioJmX7Ctv9J50Hqqcv+kiM8jJHrsB4IIrc0Cc/qb08YAo//i44JTuPxs2+FS2ifDmQA+TK5fJxwUIQJ6KDQ+0wB+T6yeYMJw== HOST-CA graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $
And the big test…
graham@graz-baz:~/web_page_counter/terraform/VMware/Dev/Monolith $ ssh -v root@192.168.9.200 OpenSSH_7.4p1 Raspbian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to 192.168.9.200 [192.168.9.200] port 22. debug1: Connection established. debug1: identity file /home/graham/.ssh/id_rsa type 1 debug1: identity file /home/graham/.ssh/id_rsa-cert type 5 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/graham/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u7 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000 debug1: Authenticating to 192.168.9.200:22 as 'root' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc, serial 0 ID "dev_host_server" CA ssh-rsa SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 valid from 2020-01-03T16:51:35 to 2021-01-01T16:56:35 debug1: Host '192.168.9.200' is known and matches the RSA-CERT host certificate. debug1: Found CA key in /home/graham/.ssh/known_hosts:1 debug1: rekey after 134217728 blocks
Client Certificate Configuration
Step 10 on CA server – Create a client CA signing key
$ ssh-keygen -t rsa -N '' -C CLIENT-CA -b 4096 -f client-ca Generating public/private rsa key pair. Your identification has been saved in client-ca. Your public key has been saved in client-ca.pub. The key fingerprint is: SHA256:bDOWewefRFdvXDGQixnVgVjnHHPSjOLeO381BiAPF/Q CLIENT-CA The keys randomart image is: +---[RSA 4096]----+ | .o=+*@*| | o =oo==X| | =.*Eoo+| | . . =.+ . | | S ..... | | o + +...o.| | . . + ..o| | . . o .| | oo| +----[SHA256]-----+ ~/hashistack-ca $ ls -al total 48 drwx------ 8 grazzer staff 256 3 Jan 21:30 . drwxr-xr-x+ 106 grazzer staff 3392 3 Jan 17:21 .. -rw-r--r--@ 1 grazzer staff 6148 3 Jan 21:00 .DS_Store -rw------- 1 grazzer staff 3369 3 Jan 21:30 client-ca -rw------- 1 grazzer staff 735 3 Jan 21:30 client-ca.pub -rw------- 1 grazzer staff 3247 3 Jan 20:46 host-ca -rw------- 1 grazzer staff 733 3 Jan 20:46 host-ca.pub drwx------ 5 grazzer staff 160 3 Jan 20:50 tmp-keys ~/hashistack-ca $
Step 11 on CA server – Copy public client CA signing key to ALL target hosts
$ scp client-ca.pub root@192.168.9.200:/etc/ssh/client-ca.pub client-ca.pub $
Step 12 on target Hosts – Configure to use new public client CA key
vagrant@redis01:~$ sudo mv .ssh/client-ca.pub /etc/ssh/. vagrant@redis01:~$ chmod 644 /etc/ssh/client-ca.pub vagrant@redis01:~$ grep -qxF 'TrustedUserCAKeys /etc/ssh/client-ca.pub' /etc/ssh/sshd_config || echo 'TrustedUserCAKeys /etc/ssh/client-ca.pub' | sudo tee -a /etc/ssh/sshd_config TrustedUserCAKeys /etc/ssh/client-ca.pub vagrant@redis01:~$ sudo systemctl restart ssh vagrant@redis01:~$ sudo systemctl status ssh ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2020-01-03 22:18:58 UTC; 30s ago Process: 29741 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) Main PID: 29743 (sshd) Tasks: 1 (limit: 1112) CGroup: /system.slice/ssh.service └─29743 /usr/sbin/sshd -D Jan 03 22:18:58 redis01 systemd[1]: Starting OpenBSD Secure Shell server... Jan 03 22:18:58 redis01 sshd[29743]: Server listening on 0.0.0.0 port 22. Jan 03 22:18:58 redis01 sshd[29743]: Server listening on :: port 22. Jan 03 22:18:58 redis01 systemd[1]: Started OpenBSD Secure Shell server. vagrant@redis01:~$
Step 13 on CA server – Create individual client signed certificates
$ ssh-keygen -s ../client-ca -I graham-dev -n root,vagrant,graham,pi -V -5:+52w -z 1 id_rsa.pub Signed user key id_rsa-cert.pub: id "graham-dev" serial 1 for root,vagrant,graham,pi valid from 2020-01-03T23:05:07 to 2021-01-01T23:05:12 ~/hashistack-ca/client-key $ scp id_rsa-cert.pub graham@192.168.1.199:/home/graham/.ssh/id_rsa-cert.pub id_rsa-cert.pub 100% 2562 136.2KB/s 00:00 ~/hashistack-ca/client-key $
Step 14 on client(s) – Test new ssh certificate access
$ ssh -v root@192.168.9.200 OpenSSH_7.9p1, LibreSSL 2.7.3 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug1: Connecting to 192.168.9.200 [192.168.9.200] port 22. debug1: Connection established. debug1: identity file /Users/grazzer/.ssh/id_rsa type 0 debug1: identity file /Users/grazzer/.ssh/id_rsa-cert type 4 debug1: identity file /Users/grazzer/.ssh/id_dsa type -1 debug1: identity file /Users/grazzer/.ssh/id_dsa-cert type -1 debug1: identity file /Users/grazzer/.ssh/id_ecdsa type -1 debug1: identity file /Users/grazzer/.ssh/id_ecdsa-cert type -1 debug1: identity file /Users/grazzer/.ssh/id_ed25519 type -1 debug1: identity file /Users/grazzer/.ssh/id_ed25519-cert type -1 debug1: identity file /Users/grazzer/.ssh/id_xmss type -1 debug1: identity file /Users/grazzer/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.9 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002 debug1: Authenticating to 192.168.9.200:22 as 'root' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:3lpUNJcP8GDhkHmozrhP4XA99s16AaV0kLI1fysjpEc, serial 0 ID "dev_host_server" CA ssh-rsa SHA256:uKJqQBCvoukiJ6n0GRX6Me8/VmHUB6bps81ekpUjdJ8 valid from 2020-01-03T16:51:35 to 2021-01-01T16:56:35 debug1: Host '192.168.9.200' is known and matches the RSA-CERT host certificate. debug1: Found CA key in /Users/grazzer/.ssh/known_hosts:1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 134217728 blocks debug1: Will attempt key: /Users/grazzer/.ssh/id_rsa RSA SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc debug1: Will attempt key: /Users/grazzer/.ssh/id_rsa RSA-CERT SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc debug1: Will attempt key: /Users/grazzer/.ssh/id_dsa debug1: Will attempt key: /Users/grazzer/.ssh/id_ecdsa debug1: Will attempt key: /Users/grazzer/.ssh/id_ed25519 debug1: Will attempt key: /Users/grazzer/.ssh/id_xmss debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=ssh-ed25519 debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,password debug1: Next authentication method: publickey debug1: Offering public key: /Users/grazzer/.ssh/id_rsa RSA SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc debug1: Authentications that can continue: publickey,password debug1: Offering public key: /Users/grazzer/.ssh/id_rsa RSA-CERT SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc debug1: Server accepts key: /Users/grazzer/.ssh/id_rsa RSA-CERT SHA256:ZjZMqCQywrvljKloim0haaUEx7Io0NrVHO5QxXWaZdc debug1: Authentication succeeded (publickey). Authenticated to 192.168.9.200 ([192.168.9.200]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions@openssh.com debug1: Entering interactive session. debug1: pledge: network debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0 debug1: Sending environment. debug1: Sending env LANG = en_GB.UTF-8 debug1: Sending env LC_TERMINAL_VERSION = 3.3.6 debug1: Sending env LC_TERMINAL = iTerm2 Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage * Overheard at KubeCon: "microk8s.status just blew my mind". https://microk8s.io/docs/commands#microk8s.status * Canonical Livepatch is available for installation. - Reduce system reboots and improve kernel security. Activate at: https://ubuntu.com/livepatch Last login: Fri Jan 3 22:46:44 2020 from 192.168.2.101 root@redis01:~#
Note:
The certificate generated above is Valid -V from 5 minutes ago for the next 30 days -5m:+30d. Ideally keep this as short as possible – for my dev/play environment security is not a real concern – 30 days is fine.
HTH
Graham
allthingscloud.eu 