Web Programming, Linux System Administation, and Entrepreneurship in Athens Georgia

Category: Encryption

SSH Key Best Practices for 2025 – Using ed25519, key rotation, and other best practices

Apparently Google thinks I’m an expert at SSH Keys, so I’m providing an update to my previous post two years ago with some slight updates.

You can tell quite a bit about other IT professionals from their Public SSH Key! I often work with others and ask for their key when granting access to a machine I control. Its a negative sign when they ask how to create one. If they provide one in the PuttyGen format, I know they’ve been asked for their key exactly once. A 2048 bit or smaller RSA key means they haven’t generated one in a long time. If they send me an ed25519 key with a comment other than their machine name, I feel confident that they know what they are doing.

For reference, a 4096-bit RSA key will be in this format:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDowuIZFbN2EWbVwK9TG+O0S85yqr7EYc8Odv76H8+K6I7prrdS23c3rIYsKl2mU0PjOKGyyRET0g/BpnU8WZtDGH0lKuRaNUT5tpKvZ1iKgshdYlS5dy25RxpiVC3LrspjmKDY/NkkflKQba2WAF3a5M4AaHxmnOMydk+edBboZhklIUPqUginLglw7CRg/ck99M9kFWPn5PiITIrpSy2y2+dt9xh6eNKI6Ax8GQ4GPHTziGrxFrPWRkyLKtYlYZr6G259E0EsDPtccO5nXR431zLSR7se0svamjhskwWhfhCEAjqEjNUyIXpT76pBX/c7zsVTBc7aY4B1onrtFIfURdJ9jduYwn/qEJem9pETli+Vwu8xOiHv0ekXWiKO9FcON6U7aYPeiTUEkSDjNTQPUEHVxpa7ilwLZa+2hLiTIFYHkgALcrWv/clNszmgifdfJ06c7pOGeEN69S08RKZR+EkiLuV+dH4chU5LWbrAj/1eiRWzHc2HGv92hvS9s/c= someuser@brandonsLaptop

And for comparison, an ed25519 key looks like this:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBLEURucCueNvq4hPRklEMHdt5tj/bSbirlC0BkXrPDI someuser@ip-172-31-74-201

You’ll notice in both of these, the first characters contain the key type. The middle section with all of the random characters contain the base-64 encoded public key. And at the end is a comment that is intended to identify the user to whom it belongs.

The ed25519 key is much shorter than an RSA keys, so if you’ve never seen one before, you might think it is less secure. But this key type is newer, and uses a totally different, more complex algorithm. Although the 256-bit ed25519 key has fewer characters, it is, for all practical purposes, as secure as the 4096-bit RSA key above. The ed25519 algorithm is more computationally complex, so it requires fewer bits for a similar level of security.

The ed25519 algorithm is based on a specific formula for an ellipse instead of prime numbers like the RSA algorithm. It has been in wide use for ~10 years, is supported by all modern software, and as such is the current standard for most professional users. Creating a key is simple with the ssh-keygen command. But before jumping to the actual command, I wanted to also explain a few other tips that I use, and think others should adopt as well.

Keys should created by individuals, not issued to groups

You should never share your private key with anybody. Ever. If a key is ever shared, you have to assume that the other party can impersonate you on any system in which it is used.

I’ve been a part of some teams which create a new server and create a new key to access that server, and share they new key with everybody who needs to accss the machine. I think this practice stems from AWS or other providers who create an SSH key for you, along with a new machine, and the user just continuing the practice. I wish they’d change that.

That’s the backwards way of thinking about it. Individuals should own their own keys. They should be private. And you can add multiple public keys to resources where multiple people need access. Again, I wish AWS and others will allow this more easily instead of allowing only a single key. You then revoke access by removing the public key, instead of having to re-issue a new key whenever the group changes. (Or worse, not changing the key at all!)

Rotating your SSH keys

You should rotate your SSH keys regularly. The thought process here is that if you have used the same key for a long time, and then your laptop with your private key gets lost, or your key compromised, every machine that you’ve been granted access to over that time is potentially at risk, because administrators are notoriously bad about revoking access. By changing out your key regularly, you limit the potential access in the case of a compromised key. Generating a new SSH key also ensures that you are using more modern algorithms and key sizes.

I like to create a new SSH key about every two years. To remind my self to do this, I embed the year I created the key within its name. My last key was created in March 2023, which I have named [email protected]. I’m creating a new key now, at the beginning of 2025, which I’ll name with the current year. Each time I use it, I’m reminded when I created the key, and if it gets to be around 2 years, and I have some time free, I’ll create a new key. Of course I keep all of my older keys in case I need access to something I haven’t accessed for a while. My ssh-agent usually has my two most recent keys loaded. If I do need to use an older one, it is enough of a process to find and use the old one, that the first thing I’ll do is update my key as soon as I get into a system where an old key was needed.

Don’t use the default ssh-keygen comment

I also suggest that you make the SSH key comment something meaningful. If you don’t provide a comment, most ssh-keygen implementations default to your_username@you_machine name which just might be silly or meaningless. In a professional setting, it should clearly identify you. For example BrandonChecketts as a comment is better than me00101@billys2017_macbook_air. It should be meaningful both to you, and to whomever you are sharing it.

I mentioned including the creation month above, which I like to include in the comment because when sharing the public key, it subtly demonstrates that I am security conscious, have rotated it recently, and I know what I’m doing. The comment at the end of the key can be changed without affecting its functionality, so if I might change the comment depending on who I’m sharing it with. When I receive a public key from somebody else that contains a generic comment, I often change the comment to be include their name or email address so I can later remember to whom it belongs to.

Always use a passphrase

Your SSH key is just a tiny file on disk. If your machine is ever lost, stolen, or compromised in any way by an attacker, the file is pretty easy for them to copy. Without it being encrypted with a pass phrase, it is directly usable. And if someone has access to your SSH private key, they probably have access to your bash or terminal history and would know where to use it.

As such, it is important to protect your SSH private key with a decent pass phrase. To avoid typing your pass phrase over and over, use the SSH-Agent, which will remember it for your session.

Understand and use SSH-Agent Forwarding when applicable

SSH Agent Forwarding allows you to ssh into one machine, and then transparently “forward” your SSH keys to the that machine for use authenticating into a machine past it. I most often use this when authenticating to GitHub from a remote machine. Using Agent forwarding means that I don’t have to copy my SSH Private key onto the remote machine in order to authenticate to GitHub from there.

You shouldn’t, however, just blindly use SSH Agent Forwarding everywhere. If you access a compromised machine where an attacker may have access to your account or to the root account, you should NOT use agent forwarding since it is possible for them to intercept your private key. I’ve never seen this exploited, but since it is possible, you should only use SSH Agent Forwarding to systems which you trust.

The ssh-keygen Command

With all of the above context, this is the command you should use to create your ed25519 key:

ssh-keygen -t ed25519 -f ~/.ssh/your-key-filename -C "your-key-comment"

That will ask you for a pass phrase and then show you a randomart image that represents your public key when it is created. The randomart is just a visual representation of your key so that you can see it is different from others.

 $ ssh-keygen -t ed25519 -f ~/.ssh/[email protected] -C "[email protected]"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/[email protected]
Your public key has been saved in ~/.ssh/[email protected]
The key fingerprint is:
SHA256:HiCF8gbV6DpBTC2rq2IMudwBc5+QuB9NqeGtc3pmqEY brandon+2025@roundsphere
The key's randomart image is:
+--[ED25519 256]--+
| o.o.+.          |
|  * +..          |
| o O...          |
|+ A *. .         |
|.B % .  S        |
|=E* =  . .       |
|=+o=    .        |
|+==.=            |
|B..B             |
+----[SHA256]-----+

Obsessive/Compulsive Tip

This may be taking it too far, but I like to have a memorable few digits at the end of the key so that I can confirm the key got copied correctly. One of my keys ends in 7srus, so I think of it as my “7’s ‘R’ Us” key. You can do that over and over again until you find a key that you like with this one-liner:

rm newkey; rm newkey.pub; ssh-keygen -t ed25519 -f ./newkey -C "[email protected]" -N ''; cat newkey.pub;

That creates a key without a passphrase, so you can do it over and over quickly until you find a public key that you “like”. Then protect it with a passphrase with the command

ssh-keygen -p -f newkey

And obviously, then you rename it from newkey and to newkey.pub a more meaningful name.

Replacing your public key when you use it

As you access machines, make sure to add your new key and remove old keys from your ~/.ssh/authorized_keys file. At some point, you should remove your previous key from your ssh-agent and you’ll be forced to use the old key to get in, and replace it with the new key.

Is that complete? What other tips should others know about when creating an SSH Key in 2025 and beyond?

Adding ed25519 SSH Host Keys via cloud-init

SSH Host Keys are they Public / Private keys that identify a server when connecting to it via SSH.
Most people don’t understand very well how these work, and just quickly click, or type ‘yes’ to approve the Key Fingerprint
when you connect via SSH to a server.

The first time you connect to a server, you will see something like this:

The authenticity of host '[myremoteserver.com]:22 ([12.34.56.78]:22)' can't be established.
ED25519 key fingerprint is SHA256:Vqfv339yJU/zRADJ4SlgF8DcZ0d7Cy1zWX69C33d3e4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

This means that it is the first time your computer has connected to the remote SSH. It is asking if the Key Fingerprint is what you expected. Since we don’t tend to communicate key fingerprints in advance, we usually trust that this is correct and just type ‘yes’.

But this is an important part of the Authentication process. There are a number of possible ways that the remote server may NOT be the server you intend. You could have simply typed the hostname wrong. More nefarious examples might include DNS hijacking or rerouting of your traffic.

When you answer ‘yes’ to that question, the host key fingerprint is saved to a file on your machine in ~/.ssh/known_hosts. If you connect to the same host again, it won’t ask that question again, since you’ve already approved it.

Note that SSH Host Keys (sometimes called SSH Instance Keys) are in the same format, but have a different purpose than SSH User Keys with which most people are familiar. The Host Keys are intended to identify the MACHINE, while your user key is meant to identify YOU.

The SSH Host Key is usually created when an instance is turned on for the first time. When the SSH Server Starts, if it doesn’t find existing host keys, it creates them using a pseudo-random number generator. It kindof just magically happens without anyone having to think about it.

I happen to connect to a lot of servers that are turned on by AWS Auto Scaling Groups. Whenever a new server is launched, that instance creates new SSH Host Keys. If a server has been recreated since I last connected to it, I get this nasty error message:

user@my-machine ~ % ssh ubuntu@myremotemachine
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:Vqfv339yJU/zRADJ4SlgF8DcZ0d7Cy1zWX69C33d3e4.
Please contact your system administrator.
Add correct host key in /Users/myusername/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /Users/myusername/.ssh/known_hosts:16
Host key for [myremotemachine]:22 has changed and you have requested strict checking.
Host key verification failed.

This error message explains that the SSH Host Key of the machine to which I’ve attempted to connect doesn’t match what it used to be. This could be due to a man-in-the-middle attack, or it could be that the host key legitimately changed, as is what happens when my Auto-Scaling group creates a new instance.

You can “fix” this error by editing your ~/.ssh/known_hosts file and removing the offending line that is mentioned. In this example, it is line 16.

I’ve recently gotten tired of fixing my known_hosts file and have started changing my Auto-Scaling groups so that they use the same Host Key each time that the instance starts. That means I don’t get the error message, and it saves me ~10 seconds (and doesn’t break my train-of-thought) when connecting to an instance that has been replaced.

This is an example of what I enter into the UserData section of my CloudFormation template inside the LaunchTemplate section. It specifies two pre-generated SSH Keys so that each time the instances launches, it will have the same host key.

In order to generate these, I usually just launch an instance the first time without it, then grab the four files mentioned. The files are contained in:

  • /etc/ssh/ssh_host_ecdsa_key
  • /etc/ssh/ssh_host_ecdsa_key.pub
  • /etc/ssh/ssh_host_ed25519_key
  • /etc/ssh/ssh_host_ed25519_key.pub

You could also create these files in advance using ssh-keygen.
My example below uses the newer ecdsa and ed25519 keys, and avoids using the older rsa and dsa keys. This should work fine for most modern distributions and SSH Clients.

UserData: !Base64 |
  #cloud-config
  write_files:
    - path: /etc/motd
      owner: root:root
      permissions: '0644'
      content: |
        You are connected to my-hostname

  ssh_keys:
    ecdsa_private:
      -----BEGIN OPENSSH PRIVATE KEY-----
      put-your-private
      key-contents
      here
      -----END OPENSSH PRIVATE KEY-----
   ecdsa_public:
      ssh-ed25519 AAAAyour-public-key-contents-here ecdsa-my-hostname

    ed25519_private:
      -----BEGIN OPENSSH PRIVATE KEY-----
      put-your-private
      key-contents
      here
      -----END OPENSSH PRIVATE KEY-----
   ed25519_public:
      ssh-ed25519 AAAAyour-public-key-contents-here ed25519-my-hostname

There is one downside, that the host keys are now stored in my CloudFormation template, so I need to make sure and keep that secure. Anybody that has access to these keys could impersonate the server on which it is used.

It’s 2023. You Should Be Using an Ed25519 SSH Key (And Other Current Best Practices)

UPDATE: I’ve got an updated post containing SSH Key Best Practices for 2025


Original content from Sept 2023:

I often have to ask other IT professionals for the Public SSH key for access to a server or for other tasks. I really cringe when they ask me what that is or how to create one. I kindof cringe when they give me one from PuttyGen in its native format. I feel a little better when they provide a 4096-bit RSA key without needing an explanation. When somebody provides an Ed25519 key, I feel like I’m working with somebody who knows what they are doing.

A 4096-bit RSA Keys look like this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDowuIZFbN2EWbVwK9TG+O0S85yqr7EYc8Odv76H8+K6I7prrdS23c3rIYsKl2mU0PjOKGyyRET0g/BpnU8WZtDGH0lKuRaNUT5tpKvZ1iKgshdYlS5dy25RxpiVC3LrspjmKDY/NkkflKQba2WAF3a5M4AaHxmnOMydk+edBboZhklIUPqUginLglw7CRg/ck99M9kFWPn5PiITIrpSy2y2+dt9xh6eNKI6Ax8GQ4GPHTziGrxFrPWRkyLKtYlYZr6G259E0EsDPtccO5nXR431zLSR7se0svamjhskwWhfhCEAjqEjNUyIXpT76pBX/c7zsVTBc7aY4B1onrtFIfURdJ9jduYwn/qEJem9pETli+Vwu8xOiHv0ekXWiKO9FcON6U7aYPeiTUEkSDjNTQPUEHVxpa7ilwLZa+2hLiTIFYHkgALcrWv/clNszmgifdfJ06c7pOGeEN69S08RKZR+EkiLuV+dH4chU5LWbrAj/1eiRWzHc2HGv92hvS9s/c= someuser@brandonsLaptop

And for comparison, an Ed25519 Key looks like this:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBLEURucCueNvq4hPRklEMHdt5tj/bSbirlC0BkXrPDI someuser@ip-172-31-74-201

The Ed25519 key is much shorter, so initially you might think it is less secure. But these keys use a totally different algorithm, so although the key has fewer characters, it is, for all practical purposes, as secure as the RSA key above. You can ask your favorite search engine or AI for more details about the differences.

The Ed25519 algorithm has been around for ~10 years now. It is widely supported by any modern software, and as such is the current standard for most professional users. Creating a key is simple with the ssh-keygen command. But before jumping to the actual command, I wanted to also explain a couple other tips that I use, and think others should pick up as well.

Keys should be issued to individuals, not groups

You should never, ever share your private key with anybody. Ever. If a key is ever shared, you have to assume that the other party can impersonate you on any system in which it is used.

I’ve seen some organizations who create a new machine and use a new SSH Key on it. Then share the key with all of the individuals who need to access the machine. Perhaps this practice comes from AWS or other hosting providers who create an SSH key for you, along with a new machine, and the user not knowing any better.

Although it kindof works, that’s the backwards way of doing it. Individuals should own their own keys. They should be private. And you can add multiple public keys to resources where multiple people need access. You then revoke access by removing the public key, instead of having to re-issue a new key whenever the group changes. (Or worse, not changing the key at all!)

Rotating your keys

You should rotate your SSH keys on some kind of schedule. The main risk you are trying to avoid here is that if you have used the same key for 20 years, and then your laptop with your private key gets lost, or your key compromised, every machine that you’ve been granted access to over that time is potentially at risk, because administrators are notoriously bad about revoking access. By changing out your key regularly, you limit the potential access in the case of a compromised key. Generating a new SSH key also ensures that you are using more modern algorithms and key sizes.

I like to start a new key about every year. To remind my self to do this, I embed the year I created the key within its name. So I last created a key in March 2023, which I have named brandon+2022-03@roundsphere. When it gets to be 2024, I’ll be subtly reminded each time I use it that it’s time to create a new key. I keep all of my older keys if I need them. But they aren’t in memory or in my SSH-Agent. If I do need to use one, it is enough of a process to find the old one, that the first thing I’ll do is update my key as soon as I get in a system where an old key was needed.

Don’t use the default comment

Make the comment meaningful. If you don’t provide a comment, it defaults to your_username@you_machine name which just might be silly or meaningless. In a professional setting, it should clearly identify you. For example BrandonChecketts as a comment is better than me00101@billys2017_macbook_air. It should be meaningful both to you, and to whomever you are sharing it.

I mentioned including the creation month above, which I like because when sharing it, it subtly demonstrates that I am at least somewhat security conscious and I know what I’m doing. The comment at the end of the key isn’t necessary for the key to work correctly, so you can change it when sharing it. I often change the comment to be more meaningful if someone provides me with a key that doesn’t clearly indicate its owner.

Always use a passphrase

Your SSH key is just a tiny file on disk. If your machine is ever lost, stolen, or compromised in any way by an attacker, the file is pretty easy for them to copy. Without it being encrypted with a pass phrase, it is directly usable. And if someone has access to your SSH private key, they probably have access to your history and would know where to use it.

As such, it is important to protect your SSH private key with a decent pass phrase. Note that you can use SSH-Agent so you don’t need to type the passphrase every time you need to use the key.

The Command

This is the command you should use to create your ED25519 Key:

ssh-keygen -t ed25519 -f ~/.ssh/your-key-filename -C "your-key-comment"

That will ask you for a pass phrase and then show you a cool randomart image that represents your public key when it is created

 $ ssh-keygen -t ed25519 -f ./deleteme -C "brandon+2023-09@roundsphere"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./deleteme
Your public key has been saved in ./deleteme.pub
The key fingerprint is:
SHA256:HiCF8gbV6DpBTC2rq2IMudwAc5+QuB9NqeGtc3pmqEY brandon+2023-09@roundsphere
The key's randomart image is:
+--[ED25519 256]--+
| o.o.+.          |
|  * +..          |
| o O...          |
|+ B *. .         |
|.B % .  S        |
|=E* =  . .       |
|=+o=    .        |
|+==.=            |
|B..B             |
+----[SHA256]-----+

Obsessive/Compulsive Tip

I maybe have spent 10 minutes creating a key over an over until I found a key that ended in a few character that I like. One of my keys ends in 7srus, so I think of it as my “7’s ‘R’ Us” key. You can do that over and over again until you find a key that you like with this one-liner:

rm newkey; rm newkey.pub; ssh-keygen -t ed25519 -f ./newkey -C "[email protected]" -N ''; cat newkey.pub;

That creates a key without a passphrase, so you can do it over and over quickly until you find a public key that you “like”. Then protect it with a passphrase with the command

ssh-keygen -p -f newkey

And obviously, then you rename it from newkey to something more meaningful.

What else? Any other tips for creating an SSH key and looking like a professional in 20223?

How Do Clients Securely Connect to SSL & HTTPS Servers?

This question arose from Steven Chu on my previous post about MySQL SSL Connections without Client Certificates. How is the client able to securely connect to a server using SSL if it doesn’t already know or trust the Server Certificate?

It is important to understand that there are a few different, interrelated topics here. All of these involve SSL and certificates, but in differing ways, so they are often conflated. Secure communication over SSH shares the same concepts, but has different mechanisms.

  1. Encryption of the traffic between client and server.
  2. Verification that the server is who the client believes it to be.
  3. Authentication of the client to the server.

For SSL and HTTPS communication, the first two concepts are accomplished together because there is no point in communicating securely with a remote party if you can’t verify that the remote party is who they claim to be and that there isn’t a “Man in the Middle” able to intercept the secure traffic.

You actually communicate securely with unknown servers all of the time. When you loaded this web page, your browser didn’t know anything about www.brandonchecketts.com beforehand. Same thing when you load your bank’s website. You never configured your browser specifically to trust their website. So how is it able to verify that it is actually your bank, and not an attacker who is impersonating your bank?

Certificate Authorities

Anybody can create an SSL Certificate with any name on it. In my SSL Certificate Notes post, you can find instructions for creating an SSL certificate. Note that you simply type in the name for the certificate. So you could attempt to create a certificate for any host you care to try. However, an essential part of the process is the Signing of the Certificate. You can self-sign a certificate with any name on it. But if you want your certificate to be recognized and trusted by anybody else, you need to have a recognized Certificate Authority (CA) sign it. If you were to try to create a certificate for www.google.com, no Certificate Authority is going to sign that since you can’t validate that you are authorized to create certificatesnobody else in the world is going to trust it.

When a Certificate Authority signs a certificate, it is their job to verify that the certificate owner is who they claim to be in some way. On the public internet, that is largely done through DNS or Email validation. For example, on this site, I use a certificate issued by Amazon Web Services. In order obtain that certificate, I had to verify that I own the domain. Since the domain is also hosted at AWS, it is quite easy for me to create the DNS records for verification, and AWS can validate it within seconds. I couldn’t, for example, validate a certificate that was for ‘www.google.com’. I’d be unable to validate it with any certificate authority since I can’t make the required DNS entries or receive emails to the required email addresses for google.com.

Extended Validation certificates, offered by some Certificate Authorities, and recognized by some web browsers with a different color banner, often have additional verification steps other than just DNS or Email.

Intermediate Certificates and Multiple layers of Certificate Authoritiees

When a certificate is signed by a recognized Certificate Authority, your client can trust it, because it trusts the Certificate Authority. On the public Internet, most of the time there are multiple layers of Certificate Authorities.

On OSX, you can find the list of root certificates it trusts in the “Keychain Access” system app, in the “System Roots” section. On an Ubuntu or Debian Linux system, the trusted certificates are files that exist or are symlinked in `/etc/ssl/certs`. These systems have dozens to hundreds of certificates that they trust. Look closely and you’ll see that most of them expire 10+ years into the future. These “Root” certificates are highly protected and usually don’t directly sign certificates. The Certificate Authority will often delegate access to intermediate authorities with their own keys that can further sign certificates.

In Chrome, you can click the lock icon next to the URL, and find details about the certificate, including the intermediate certificates. As of the writing of this post, you can see that the SSL Certificate issued to brandonchecketts.com is issued by “Amazon”, which is trusted by the root certificate named “Amazon Root CA 1”. I can find that root certificate in the list of certificates trusted by my OSX system.

Client Authentication

I mentioned the third step above about the client authenticating to the server. In many cases, like this website, there is no need for the client to authenticate to the server since the content is public and intended to be viewed anonymously. If authentication is required, for instance to create a new post, then I simply log in with a username and password entered of the HTTPS connection. Same as you do every day.

Many well-meaning articles about generating SSL Certificates for services other than HTTPS often mention creating an SSL client certificate. The Client Certificate is then provided to the server so that the server can validate the client is who they claim to be. The Client Certificate is simply an alternate (often thought of as “more secure”) method of authenticating than a username and password, or sometimes even in addition to a username and password. In practice, I’ve seen that usernames and passwords transmitted over an encrypted connection are very common, well understood, and just as secure as using an SSL Client Certificate.

APISigning Now Works with Amazon Simple Email Service (SES)

APISigning.com has been signing Amazon Product Advertising requests for a couple of years now. Amazon recently announced their Simple Email Service that makes it easy to send emails via an API. The SES API requires that requests be authenticated using some cryptographic functions that are not easily available on all platforms or programming languages. In those cases, developers can use the APISigning SES Service to calculate the correct signature and perform the request on their behalf.

APISigning has free accounts that effectively allows 10k signing requests each month. Users who require additional requests can subscribe to a paid account with higher limits.

MySQLDump To a Remote Server

I was running out of disk space on a server today. The server had a large database table that was no longer used, so I wanted to archive it and then drop the table. But the server didn’t have enough disk space to dump it out to disk before copying it off to a remote server for archiving.

The first thought was to run mysqldump dump on the destination machine, and to access the database over the network. That however, doesn’t compress or encrypt the data. Plus I would have had to create a mysql user with permission to access the database remotely.

The solution I came up with worked out well: mysqldump directly to the remote host with this command:

mysqldump <DATABASE_NAME> [mysqldump options] | gzip -c | ssh user@remotehost "cat > /path/to/some-file.sql.gz"

That pipes the mysqldump command through gzip, then to through and SSH connection. SSH on the remote side runs the ‘cat’ command to read the stdin, then redirects that to the actual file where I want it saved.

Getting Dkimproxy Installed and Configured

Dkimproxy is a great program for getting Postfix to both sign and validate DomainKeys and DKIM messages. Prior to dkimproxy, one would have used dk-filter and dkim-filter which did DomainKeys and DKIM signing separately. dkimproxy is a newer version that combines the functionality into one program. Installing it can be a bit complicated because it isn’t available in most distro repositories, and requires several Perl modules that need to be installed. Configuring it can be difficult as well, because it involves making changes DNS and postfix, in addition to its own configuration.

I wrote these steps below as I went through a recent installation for a customer

You can install the required Perl modules through the RPM Forge Repository if you have it enabled with the command (Thanks JohnB for mentioning that):

yum install perl-Net-Server perl-Error perl-Mail-DKIM

Otherwise, you can install them manually with CPAN. First install the openssl-devel package (You’ll need it for CPAN to install Mail::DKIM)

yum install openssl-devel

Now install the required Perl modules

# perl -MCPAN -e shell
> install Net::Server
> install Error
> install Mail::DKIM

Download and install the actual dkimproxy code:

cd /usr/local/src
wget https://internap.dl.sourceforge.net/sourceforge/dkimproxy/dkimproxy-1.0.1.tar.gz
tar -xvzf dkimproxy-1.0.1.tar.gz
cd dkimproxy-1.0.1
./configure --prefix=/usr/local/dkimproxy
make
make install

You should now have the program installed in /usr/local/dkimproxy. A sample init file is included, so we can copy it into place also:

cp /usr/local/src/dkimproxy-1.0.1/sample-dkim-init-script.sh /etc/init.d/dkimproxy

Create a ‘dkim’ user and group, but lock the password:

useradd -d /usr/local/dkimproxy dkim
passwd -l dkim

That should be enough to get dkimproxy running, but it isn’t configured yet.

Create a key file for your domain

cd /usr/local/dkimproxy/etc/
openssl genrsa -out domain.tld.key 1024
openssl rsa -in domain.tld.key -pubout -out domain.tld.pub

Now create a DNS TXT record for mail._domainkey.domain.tld with the contents of domain.tld.pub. Your public key will span at least two lines, so combine all of the lines of the key together when putting it in your DNS record. The whole DNS record will look something like this:

k=rsa; t=s; p=MFwwDQYJ......0JMCAwEAAQ==

(Note that the key is pretty long and I’ve shortened it here)
You could now confirm the key is correct in your DNS:

[root@host etc]# host -ttxt mail._domainkey.domain.tls
mail._domainkey.domain.tld descriptive text "k=rsa\; t=s\; p=MFwwDQYJ......0JMCAwEAAQ=="

(Note that the key is pretty long and I’ve shortened it here)

Now tell dkimproxy about the key files, and configuration parameters. Create /usr/local/dkimproxy/etc/dkimproxy_out.conf with this content

# specify what address/port DKIMproxy should listen on
listen    127.0.0.1:10027

# specify what address/port DKIMproxy forwards mail to
relay     127.0.0.1:10028

# specify what domains DKIMproxy can sign for (comma-separated, no spaces)
domain    domain.tld

# specify what signatures to add
signature dkim(c=relaxed)
signature domainkeys(c=nofws)

# specify location of the private key
keyfile   /usr/local/dkimproxy/etc/domain.tld.key

# specify the selector (i.e. the name of the key record put in DNS)
selector  mail

And copy the sample inbound config to the real inbound config

cd /usr/local/dkimproxy/etc
cp dkimproxy_in.conf.example dkimproxy_in.conf

Now you should be able to start up dkimproxy, and configure it to start at boot:

/etc/init.d/dkimproxy start
chkconfig dkimproxy on

And the last step is just to modify the postfix configuration to tell it to forward messages sent to port 587 through dkimproxy for signing. I added these three sections to /etc/postfix/master.cf

### dkimproxy filter - see https://dkimproxy.sourceforge.net/postfix-outbound-howto.html
#
# modify the default submission service to specify a content filter
# and restrict it to local clients and SASL authenticated clients only
#
submission  inet  n     -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_sasl_auth_enable=yes
    -o content_filter=dksign:[127.0.0.1]:10027
    -o receive_override_options=no_address_mappings
    -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject

# specify the location of the DKIM signing proxy
# Note: the smtp_discard_ehlo_keywords option requires a recent version of
# Postfix. Leave it off if your version does not support it.
dksign    unix  -       -       n       -       10      smtp
    -o smtp_send_xforward_command=yes
    -o smtp_discard_ehlo_keywords=8bitmime,starttls

# service for accepting messages FROM the DKIM signing proxy
127.0.0.1:10028 inet  n  -      n       -       10      smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

If you want it to sign messages sent from the command line sendmail program, modify the pickup service to use the content_filter like this:

pickup    fifo  n       -       n       60      1       pickup
    -o content_filter=dksign:[127.0.0.1]:10027

Finally restart postfix with ‘postfix reload’, and you *should* have a working installation. You can now use my Domainkeys/Dkim validator to test and ensure that it is working.

Identifying Weak SSL or SSH Keys on CentOS

With the Debian OpenSSL problems, everybody is wanting to know if their server is vulnerable to any attacks. Fortunately, CentOS machines shouldn’t be directly affected and have fewer issues than if you are using Debian or Ubuntu derivatives. Unfortunately though, your system may still be vulnerable if you have any users that may have generated their keys on an affected machine. So it is definitely necessary to check, even if you are not running a distribution that is affected.

This is the steps I have been going through to look for any weak keys on a CentOS server

Download the weak key detector provided by Debian (there may be better tools to use by now). It is available on the announcement page. (I’m not linking to it intentionally).

[root@host ~]# cd /tmp
[root@host tmp]# wget https://security.debian.org/project/extra/dowkd/dowkd.pl.gz
--20:44:31--  https://security.debian.org/project/extra/dowkd/dowkd.pl.gz
Resolving security.debian.org... 128.31.0.36, 130.89.175.54, 212.211.132.32, ...
Connecting to security.debian.org|128.31.0.36|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14231783 (14M) [application/x-gzip]
Saving to: `dowkd.pl.gz'

100%[================================>] 14,231,783  6.42M/s   in 2.1s

20:44:33 (6.42 MB/s) - `dowkd.pl.gz' saved [14231783/14231783]

[root@host tmp]# gunzip dowkd.pl.gz

Then check a couple known files – Start out with your SSH host keys in /etc/ssh/

[root@host tmp]# perl dowkd.pl file /etc/ssh/*key*
/etc/ssh/ssh_host_dsa_key:1: warning: unparsable line
/etc/ssh/ssh_host_key:1: warning: unparsable line
summary: keys found: 4, weak keys: 0

Then check any certificates in /etc/pki/tls:

[root@host tmp]# for file in `find /etc/pki/tls/ -name "*key"`; do echo -n "$file - "; perl /tmp/dowkd.pl file $file; done
/etc/pki/tls/certs/mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/certs/secure.mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/private/localhost.key - summary: keys found: 1, weak keys: 0

Any for any SSL certificates that Apache might be using in /etc/httpd/conf/ssl.key/:

[root@host tmp]# perl dowkd.pl  file /etc/httpd/conf/ssl.key/*
summary: keys found: 4, weak keys: 0

And finally, any users who might have authorized a weak key via their authorized_users file:

[root@host tmp]# for file in `find / -name authorized_keys`; do echo -n "$file "; perl dowkd.pl file $file; done
/home/someuser/.ssh/authorized_keys summary: keys found: 6, weak keys: 0
summary: keys found: 7, weak keys: 0

Note that any that say ‘warning: no blacklist found’ means that the tool didn’t have a blacklist for the key type, so they might need to be checked with another tool unless you are sure that they are okay.

You should also check any other locations for keys. The locations could vary widely on different machines, depending on the configuration of your server. Those locations specified above should cover most of the default locations on a CentOS 4 or CentOS 5 server, but every server is different. If you don’t find it now, its quite likely that an attacker will later.

GnuPG Encryption with PHP

I found PHP’s documentation on the GnuPG functions to be pretty sparse, so thought I would share some specific steps that I went though in order to get everything working.

Prerequisites

First off, you have to install the GnuPG PHP libraries through pecl. It requires the GnuPG Made Easy (gpgme) packages to get working. The following shell commands will install the OS packages, install the GnuPG PHP libraries, then enable the PHP extension and restart Apache:

# apt-get install gnupg gpgme gpgme-devel

# pecl install gnupg

# echo extension=gnupg.so > /etc/php.d/gnupg.ini

# apachectl restart

Creating GnuPG Keys

Next, you need to create a set of keys to encrypt and decrypt your data. You’ll need to put the keys somewhere where the webserver can read and write to a directory. I’ll use /var/www/.gnupg since that is the default home directory for many Apache installations. After running the gpg command, answer the questions as prompted. User input is red in the output shown below.

# mkdir -p /var/www/.gnupg

# gpg --homedir /var/www/.gnupg --gen-keygpg
WARNING: unsafe permissions on homedir `/tmp/keys'

gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
gpg: keyring `/tmp/keys/secring.gpg' created
gpg: keyring `/tmp/keys/pubring.gpg' created
Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 2048
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 10y
Key expires at Fri Feb 23 16:35:14 2018 PST
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <[email protected]>"
Real name: Some User
Email address: [email protected]
Comment: This is a key for Some User
You selected this USER-ID:
    "Some User (This is a key for Some User) <[email protected]>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key. Enter your passphrase here
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /tmp/keys/trustdb.gpg: trustdb created
gpg: key 21CCC3D6 marked as ultimately trusted
public and secret key created and signed.
.... a bunch of random characters here....
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2018-02-24
pub   1024D/21CCC3D6 2008-02-27 [expires: 2018-02-24]
      Key fingerprint = FA45 1EE9 8772 70EF 1CFA  99CE 048A 6139 21CC C3D6
uid                  Some User (This is a key for Some User) <[email protected]>
sub   2048g/A83E754B 2008-02-27 [expires: 2018-02-24]
#chown -R apache:apache /var/www/.gnupg

Make note of the key fingerprint in the 4th from the bottom line. You’ll need this in your PHP code when referencing the key. Also, make sure that you write down your pass phrase somewhere. Your encrypted data will be useless if you don’t have the pass phrase.

Your Application

Now you can write your PHP code that will do the encryption. Here is a sample that encrypts, then decrypts something:

<?php
$CONFIG['gnupg_home'] = '/var/www/.gnupg';
$CONFIG['gnupg_fingerprint'] = 'FA451EE9877270EF1CFA99CE048A613921CCC3D6';

$data = 'this is some confidential information';

$gpg = new gnupg();
putenv("GNUPGHOME={$CONFIG['gnupg_home']}");
$gpg->seterrormode(GNUPG_ERROR_SILENT);
$gpg->addencryptkey($CONFIG['gnupg_fingerprint']);
$encrypted =  $this->gpg->encrypt($data);
echo "Encrypted text: \n<pre>$encrypted</pre>\n";

// Now you can store $encrypted somewhere.. perhaps in a MySQL text or blob field.

// Then use something like this to decrypt the data.
$passphrase = 'Your_secret_passphrase';
$gpg->adddecryptkey($CONFIG['gnugp_fingerprint'], $passphrase);
$decrypted = $gpg->decrypt($encrypted);

echo "Decrypted text: $decrypted";
?>

It would be best to store $passphrase somewhere completely separate from your application configuration. Perhaps an admin user would be required to enter the passphrase when looking up this information. That way your passphrase is not stored in your config file or anywhere that an attacker could potentially gain access to it.

Troubleshooting

Make sure that the web server can write to the GnuPG Home directory. This obviously is not ideal, but seems to be required in the testing that I have done. I’ve been able to set ‘secring.gpg’ to be owned by root, but that does little good since the directory it is in has to be writable.

You can raise the error mode to GNUPG_ERROR_WARNING to generate PHP warnings on GnuPG errors. That might help to track down where errors are occurring

Database encryption made easy

I’ve always wondered how one would securely store sensitive information in a MySQL database. A recent project has given me the opportunity to work on it, and I’ve been impressed on how easy it is to implement. MySQL provides an easy interface for encrypting data before storing it in the database. Simply use the AES_ENCRYPT and AES_DECRYPT functions when reading or writing to a table.

Simply make your column a blob field, then use something like this to write to the table

(using a PEAR::DB syntax)

$db->query("
UPDATE sometable
SET    some_col = AES_ENCRYPT( ?, ?)
WHERE something_else = ?
" array( $sensitive_value, $encryption_key, $index));

and something like this to read it back out

$value = $db->getOne("
SELECT AES_DECRYPT( some_col, ?)
FROM   sometable
WHERE something_else = ?
", array( $encryption_key, $index));

© 2025 Brandon Checketts

Theme by Anders NorenUp ↑