SSH Host Keys are they Public / Private keys that identify a server when connecting to it via SSH.
Most people don’t understand very well how these work, and just quickly click, or type ‘yes’ to approve the Key Fingerprint
when you connect via SSH to a server.

The first time you connect to a server, you will see something like this:

The authenticity of host '[myremoteserver.com]:22 ([12.34.56.78]:22)' can't be established.
ED25519 key fingerprint is SHA256:Vqfv339yJU/zRADJ4SlgF8DcZ0d7Cy1zWX69C33d3e4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

This means that it is the first time your computer has connected to the remote SSH. It is asking if the Key Fingerprint is what you expected. Since we don’t tend to communicate key fingerprints in advance, we usually trust that this is correct and just type ‘yes’.

But this is an important part of the Authentication process. There are a number of possible ways that the remote server may NOT be the server you intend. You could have simply typed the hostname wrong. More nefarious examples might include DNS hijacking or rerouting of your traffic.

When you answer ‘yes’ to that question, the host key fingerprint is saved to a file on your machine in ~/.ssh/known_hosts. If you connect to the same host again, it won’t ask that question again, since you’ve already approved it.

Note that SSH Host Keys (sometimes called SSH Instance Keys) are in the same format, but have a different purpose than SSH User Keys with which most people are familiar. The Host Keys are intended to identify the MACHINE, while your user key is meant to identify YOU.

The SSH Host Key is usually created when an instance is turned on for the first time. When the SSH Server Starts, if it doesn’t find existing host keys, it creates them using a pseudo-random number generator. It kindof just magically happens without anyone having to think about it.

I happen to connect to a lot of servers that are turned on by AWS Auto Scaling Groups. Whenever a new server is launched, that instance creates new SSH Host Keys. If a server has been recreated since I last connected to it, I get this nasty error message:

user@my-machine ~ % ssh ubuntu@myremotemachine
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:Vqfv339yJU/zRADJ4SlgF8DcZ0d7Cy1zWX69C33d3e4.
Please contact your system administrator.
Add correct host key in /Users/myusername/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /Users/myusername/.ssh/known_hosts:16
Host key for [myremotemachine]:22 has changed and you have requested strict checking.
Host key verification failed.

This error message explains that the SSH Host Key of the machine to which I’ve attempted to connect doesn’t match what it used to be. This could be due to a man-in-the-middle attack, or it could be that the host key legitimately changed, as is what happens when my Auto-Scaling group creates a new instance.

You can “fix” this error by editing your ~/.ssh/known_hosts file and removing the offending line that is mentioned. In this example, it is line 16.

I’ve recently gotten tired of fixing my known_hosts file and have started changing my Auto-Scaling groups so that they use the same Host Key each time that the instance starts. That means I don’t get the error message, and it saves me ~10 seconds (and doesn’t break my train-of-thought) when connecting to an instance that has been replaced.

This is an example of what I enter into the UserData section of my CloudFormation template inside the LaunchTemplate section. It specifies two pre-generated SSH Keys so that each time the instances launches, it will have the same host key.

In order to generate these, I usually just launch an instance the first time without it, then grab the four files mentioned. The files are contained in:

  • /etc/ssh/ssh_host_ecdsa_key
  • /etc/ssh/ssh_host_ecdsa_key.pub
  • /etc/ssh/ssh_host_ed25519_key
  • /etc/ssh/ssh_host_ed25519_key.pub

You could also create these files in advance using ssh-keygen.
My example below uses the newer ecdsa and ed25519 keys, and avoids using the older rsa and dsa keys. This should work fine for most modern distributions and SSH Clients.

UserData: !Base64 |
  #cloud-config
  write_files:
    - path: /etc/motd
      owner: root:root
      permissions: '0644'
      content: |
        You are connected to my-hostname

  ssh_keys:
    ecdsa_private:
      -----BEGIN OPENSSH PRIVATE KEY-----
      put-your-private
      key-contents
      here
      -----END OPENSSH PRIVATE KEY-----
   ecdsa_public:
      ssh-ed25519 AAAAyour-public-key-contents-here ecdsa-my-hostname

    ed25519_private:
      -----BEGIN OPENSSH PRIVATE KEY-----
      put-your-private
      key-contents
      here
      -----END OPENSSH PRIVATE KEY-----
   ed25519_public:
      ssh-ed25519 AAAAyour-public-key-contents-here ed25519-my-hostname

There is one downside, that the host keys are now stored in my CloudFormation template, so I need to make sure and keep that secure. Anybody that has access to these keys could impersonate the server on which it is used.