How Do Clients Securely Connect to SSL & HTTPS Servers?

This question arose from Steven Chu on my previous post about MySQL SSL Connections without Client Certificates. How is the client able to securely connect to a server using SSL if it doesn’t already know or trust the Server Certificate?

It is important to understand that there are a few different, interrelated topics here. All of these involve SSL and certificates, but in differing ways, so they are often conflated. Secure communication over SSH shares the same concepts, but has different mechanisms.

Encryption of the traffic between client and server.
Verification that the server is who the client believes it to be.
Authentication of the client to the server.

For SSL and HTTPS communication, the first two concepts are accomplished together because there is no point in communicating securely with a remote party if you can’t verify that the remote party is who they claim to be and that there isn’t a “Man in the Middle” able to intercept the secure traffic.

You actually communicate securely with unknown servers all of the time. When you loaded this web page, your browser didn’t know anything about www.brandonchecketts.com beforehand. Same thing when you load your bank’s website. You never configured your browser specifically to trust their website. So how is it able to verify that it is actually your bank, and not an attacker who is impersonating your bank?

Certificate Authorities

Anybody can create an SSL Certificate with any name on it. In my SSL Certificate Notes post, you can find instructions for creating an SSL certificate. Note that you simply type in the name for the certificate. So you could attempt to create a certificate for any host you care to try. However, an essential part of the process is the Signing of the Certificate. You can self-sign a certificate with any name on it. But if you want your certificate to be recognized and trusted by anybody else, you need to have a recognized Certificate Authority (CA) sign it. If you were to try to create a certificate for www.google.com, no Certificate Authority is going to sign that since you can’t validate that you are authorized to create certificatesnobody else in the world is going to trust it.

When a Certificate Authority signs a certificate, it is their job to verify that the certificate owner is who they claim to be in some way. On the public internet, that is largely done through DNS or Email validation. For example, on this site, I use a certificate issued by Amazon Web Services. In order obtain that certificate, I had to verify that I own the domain. Since the domain is also hosted at AWS, it is quite easy for me to create the DNS records for verification, and AWS can validate it within seconds. I couldn’t, for example, validate a certificate that was for ‘www.google.com’. I’d be unable to validate it with any certificate authority since I can’t make the required DNS entries or receive emails to the required email addresses for google.com.

Extended Validation certificates, offered by some Certificate Authorities, and recognized by some web browsers with a different color banner, often have additional verification steps other than just DNS or Email.

Intermediate Certificates and Multiple layers of Certificate Authoritiees

When a certificate is signed by a recognized Certificate Authority, your client can trust it, because it trusts the Certificate Authority. On the public Internet, most of the time there are multiple layers of Certificate Authorities.

On OSX, you can find the list of root certificates it trusts in the “Keychain Access” system app, in the “System Roots” section. On an Ubuntu or Debian Linux system, the trusted certificates are files that exist or are symlinked in `/etc/ssl/certs`. These systems have dozens to hundreds of certificates that they trust. Look closely and you’ll see that most of them expire 10+ years into the future. These “Root” certificates are highly protected and usually don’t directly sign certificates. The Certificate Authority will often delegate access to intermediate authorities with their own keys that can further sign certificates.

In Chrome, you can click the lock icon next to the URL, and find details about the certificate, including the intermediate certificates. As of the writing of this post, you can see that the SSL Certificate issued to brandonchecketts.com is issued by “Amazon”, which is trusted by the root certificate named “Amazon Root CA 1”. I can find that root certificate in the list of certificates trusted by my OSX system.

Client Authentication

I mentioned the third step above about the client authenticating to the server. In many cases, like this website, there is no need for the client to authenticate to the server since the content is public and intended to be viewed anonymously. If authentication is required, for instance to create a new post, then I simply log in with a username and password entered of the HTTPS connection. Same as you do every day.

Many well-meaning articles about generating SSL Certificates for services other than HTTPS often mention creating an SSL client certificate. The Client Certificate is then provided to the server so that the server can validate the client is who they claim to be. The Client Certificate is simply an alternate (often thought of as “more secure”) method of authenticating than a username and password, or sometimes even in addition to a username and password. In practice, I’ve seen that usernames and passwords transmitted over an encrypted connection are very common, well understood, and just as secure as using an SSL Client Certificate.

APISigning Now Works with Amazon Simple Email Service (SES)

APISigning.com has been signing Amazon Product Advertising requests for a couple of years now. Amazon recently announced their Simple Email Service that makes it easy to send emails via an API. The SES API requires that requests be authenticated using some cryptographic functions that are not easily available on all platforms or programming languages. In those cases, developers can use the APISigning SES Service to calculate the correct signature and perform the request on their behalf.

APISigning has free accounts that effectively allows 10k signing requests each month. Users who require additional requests can subscribe to a paid account with higher limits.

MySQLDump To a Remote Server

I was running out of disk space on a server today. The server had a large database table that was no longer used, so I wanted to archive it and then drop the table. But the server didn’t have enough disk space to dump it out to disk before copying it off to a remote server for archiving.

The first thought was to run mysqldump dump on the destination machine, and to access the database over the network. That however, doesn’t compress or encrypt the data. Plus I would have had to create a mysql user with permission to access the database remotely.

The solution I came up with worked out well: mysqldump directly to the remote host with this command:

mysqldump <DATABASE_NAME> [mysqldump options] | gzip -c | ssh user@remotehost "cat > /path/to/some-file.sql.gz"

That pipes the mysqldump command through gzip, then to through and SSH connection. SSH on the remote side runs the ‘cat’ command to read the stdin, then redirects that to the actual file where I want it saved.

Getting Dkimproxy Installed and Configured

Dkimproxy is a great program for getting Postfix to both sign and validate DomainKeys and DKIM messages. Prior to dkimproxy, one would have used dk-filter and dkim-filter which did DomainKeys and DKIM signing separately. dkimproxy is a newer version that combines the functionality into one program. Installing it can be a bit complicated because it isn’t available in most distro repositories, and requires several Perl modules that need to be installed. Configuring it can be difficult as well, because it involves making changes DNS and postfix, in addition to its own configuration.

I wrote these steps below as I went through a recent installation for a customer

You can install the required Perl modules through the RPM Forge Repository if you have it enabled with the command (Thanks JohnB for mentioning that):

yum install perl-Net-Server perl-Error perl-Mail-DKIM

Otherwise, you can install them manually with CPAN. First install the openssl-devel package (You’ll need it for CPAN to install Mail::DKIM)

yum install openssl-devel

Now install the required Perl modules

# perl -MCPAN -e shell
> install Net::Server
> install Error
> install Mail::DKIM

Download and install the actual dkimproxy code:

cd /usr/local/src
wget http://internap.dl.sourceforge.net/sourceforge/dkimproxy/dkimproxy-1.0.1.tar.gz
tar -xvzf dkimproxy-1.0.1.tar.gz
cd dkimproxy-1.0.1
./configure --prefix=/usr/local/dkimproxy
make
make install

You should now have the program installed in /usr/local/dkimproxy. A sample init file is included, so we can copy it into place also:

cp /usr/local/src/dkimproxy-1.0.1/sample-dkim-init-script.sh /etc/init.d/dkimproxy

Create a ‘dkim’ user and group, but lock the password:

useradd -d /usr/local/dkimproxy dkim
passwd -l dkim

That should be enough to get dkimproxy running, but it isn’t configured yet.

Create a key file for your domain

cd /usr/local/dkimproxy/etc/
openssl genrsa -out domain.tld.key 1024
openssl rsa -in domain.tld.key -pubout -out domain.tld.pub

Now create a DNS TXT record for mail._domainkey.domain.tld with the contents of domain.tld.pub. Your public key will span at least two lines, so combine all of the lines of the key together when putting it in your DNS record. The whole DNS record will look something like this:

k=rsa; t=s; p=MFwwDQYJ......0JMCAwEAAQ==

(Note that the key is pretty long and I’ve shortened it here)
You could now confirm the key is correct in your DNS:

[root@host etc]# host -ttxt mail._domainkey.domain.tls
mail._domainkey.domain.tld descriptive text "k=rsa\; t=s\; p=MFwwDQYJ......0JMCAwEAAQ=="

(Note that the key is pretty long and I’ve shortened it here)

Now tell dkimproxy about the key files, and configuration parameters. Create /usr/local/dkimproxy/etc/dkimproxy_out.conf with this content

# specify what address/port DKIMproxy should listen on
listen    127.0.0.1:10027

# specify what address/port DKIMproxy forwards mail to
relay     127.0.0.1:10028

# specify what domains DKIMproxy can sign for (comma-separated, no spaces)
domain    domain.tld

# specify what signatures to add
signature dkim(c=relaxed)
signature domainkeys(c=nofws)

# specify location of the private key
keyfile   /usr/local/dkimproxy/etc/domain.tld.key

# specify the selector (i.e. the name of the key record put in DNS)
selector  mail

And copy the sample inbound config to the real inbound config

cd /usr/local/dkimproxy/etc
cp dkimproxy_in.conf.example dkimproxy_in.conf

Now you should be able to start up dkimproxy, and configure it to start at boot:

/etc/init.d/dkimproxy start
chkconfig dkimproxy on

And the last step is just to modify the postfix configuration to tell it to forward messages sent to port 587 through dkimproxy for signing. I added these three sections to /etc/postfix/master.cf

### dkimproxy filter - see http://dkimproxy.sourceforge.net/postfix-outbound-howto.html
#
# modify the default submission service to specify a content filter
# and restrict it to local clients and SASL authenticated clients only
#
submission  inet  n     -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_sasl_auth_enable=yes
    -o content_filter=dksign:[127.0.0.1]:10027
    -o receive_override_options=no_address_mappings
    -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject

# specify the location of the DKIM signing proxy
# Note: the smtp_discard_ehlo_keywords option requires a recent version of
# Postfix. Leave it off if your version does not support it.
dksign    unix  -       -       n       -       10      smtp
    -o smtp_send_xforward_command=yes
    -o smtp_discard_ehlo_keywords=8bitmime,starttls

# service for accepting messages FROM the DKIM signing proxy
127.0.0.1:10028 inet  n  -      n       -       10      smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

If you want it to sign messages sent from the command line sendmail program, modify the pickup service to use the content_filter like this:

pickup    fifo  n       -       n       60      1       pickup
    -o content_filter=dksign:[127.0.0.1]:10027

Finally restart postfix with ‘postfix reload’, and you *should* have a working installation. You can now use my Domainkeys/Dkim validator to test and ensure that it is working.

Identifying Weak SSL or SSH Keys on CentOS

With the Debian OpenSSL problems, everybody is wanting to know if their server is vulnerable to any attacks. Fortunately, CentOS machines shouldn’t be directly affected and have fewer issues than if you are using Debian or Ubuntu derivatives. Unfortunately though, your system may still be vulnerable if you have any users that may have generated their keys on an affected machine. So it is definitely necessary to check, even if you are not running a distribution that is affected.

This is the steps I have been going through to look for any weak keys on a CentOS server

Download the weak key detector provided by Debian (there may be better tools to use by now). It is available on the announcement page. (I’m not linking to it intentionally).

[root@host ~]# cd /tmp
[root@host tmp]# wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
--20:44:31--  http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
Resolving security.debian.org... 128.31.0.36, 130.89.175.54, 212.211.132.32, ...
Connecting to security.debian.org|128.31.0.36|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14231783 (14M) [application/x-gzip]
Saving to: `dowkd.pl.gz'

100%[================================>] 14,231,783  6.42M/s   in 2.1s

20:44:33 (6.42 MB/s) - `dowkd.pl.gz' saved [14231783/14231783]

[root@host tmp]# gunzip dowkd.pl.gz

Then check a couple known files – Start out with your SSH host keys in /etc/ssh/

[root@host tmp]# perl dowkd.pl file /etc/ssh/*key*
/etc/ssh/ssh_host_dsa_key:1: warning: unparsable line
/etc/ssh/ssh_host_key:1: warning: unparsable line
summary: keys found: 4, weak keys: 0

Then check any certificates in /etc/pki/tls:

[root@host tmp]# for file in `find /etc/pki/tls/ -name "*key"`; do echo -n "$file - "; perl /tmp/dowkd.pl file $file; done
/etc/pki/tls/certs/mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/certs/secure.mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/private/localhost.key - summary: keys found: 1, weak keys: 0

Any for any SSL certificates that Apache might be using in /etc/httpd/conf/ssl.key/:

[root@host tmp]# perl dowkd.pl  file /etc/httpd/conf/ssl.key/*
summary: keys found: 4, weak keys: 0

And finally, any users who might have authorized a weak key via their authorized_users file:

[root@host tmp]# for file in `find / -name authorized_keys`; do echo -n "$file "; perl dowkd.pl file $file; done
/home/someuser/.ssh/authorized_keys summary: keys found: 6, weak keys: 0
summary: keys found: 7, weak keys: 0

Note that any that say ‘warning: no blacklist found’ means that the tool didn’t have a blacklist for the key type, so they might need to be checked with another tool unless you are sure that they are okay.

You should also check any other locations for keys. The locations could vary widely on different machines, depending on the configuration of your server. Those locations specified above should cover most of the default locations on a CentOS 4 or CentOS 5 server, but every server is different. If you don’t find it now, its quite likely that an attacker will later.

GnuPG Encryption with PHP

I found PHP’s documentation on the GnuPG functions to be pretty sparse, so thought I would share some specific steps that I went though in order to get everything working.

Prerequisites

First off, you have to install the GnuPG PHP libraries through pecl. It requires the GnuPG Made Easy (gpgme) packages to get working. The following shell commands will install the OS packages, install the GnuPG PHP libraries, then enable the PHP extension and restart Apache:

# apt-get install gnupg gpgme gpgme-devel

# pecl install gnupg

# echo extension=gnupg.so > /etc/php.d/gnupg.ini

# apachectl restart

Creating GnuPG Keys

Next, you need to create a set of keys to encrypt and decrypt your data. You’ll need to put the keys somewhere where the webserver can read and write to a directory. I’ll use /var/www/.gnupg since that is the default home directory for many Apache installations. After running the gpg command, answer the questions as prompted. User input is red in the output shown below.

# mkdir -p /var/www/.gnupg

# gpg --homedir /var/www/.gnupg --gen-keygpg

WARNING: unsafe permissions on homedir `/tmp/keys'

gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
gpg: keyring `/tmp/keys/secring.gpg' created
gpg: keyring `/tmp/keys/pubring.gpg' created
Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 2048
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 10y
Key expires at Fri Feb 23 16:35:14 2018 PST
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: Some User
Email address: some@user.com
Comment: This is a key for Some User
You selected this USER-ID:
    "Some User (This is a key for Some User) <some@user.com>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key. Enter your passphrase here
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /tmp/keys/trustdb.gpg: trustdb created
gpg: key 21CCC3D6 marked as ultimately trusted
public and secret key created and signed.
.... a bunch of random characters here....
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2018-02-24
pub   1024D/21CCC3D6 2008-02-27 [expires: 2018-02-24]
      Key fingerprint = FA45 1EE9 8772 70EF 1CFA  99CE 048A 6139 21CC C3D6
uid                  Some User (This is a key for Some User) <some@user.com>
sub   2048g/A83E754B 2008-02-27 [expires: 2018-02-24]

#chown -R apache:apache /var/www/.gnupg

Make note of the key fingerprint in the 4th from the bottom line. You’ll need this in your PHP code when referencing the key. Also, make sure that you write down your pass phrase somewhere. Your encrypted data will be useless if you don’t have the pass phrase.

Your Application

Now you can write your PHP code that will do the encryption. Here is a sample that encrypts, then decrypts something:

<?php
$CONFIG['gnupg_home'] = '/var/www/.gnupg';
$CONFIG['gnupg_fingerprint'] = 'FA451EE9877270EF1CFA99CE048A613921CCC3D6';

$data = 'this is some confidential information';

$gpg = new gnupg();
putenv("GNUPGHOME={$CONFIG['gnupg_home']}");
$gpg->seterrormode(GNUPG_ERROR_SILENT);
$gpg->addencryptkey($CONFIG['gnupg_fingerprint']);
$encrypted =  $this->gpg->encrypt($data);
echo "Encrypted text: \n<pre>$encrypted</pre>\n";

// Now you can store $encrypted somewhere.. perhaps in a MySQL text or blob field.

// Then use something like this to decrypt the data.
$passphrase = 'Your_secret_passphrase';
$gpg->adddecryptkey($CONFIG['gnugp_fingerprint'], $passphrase);
$decrypted = $gpg->decrypt($encrypted);

echo "Decrypted text: $decrypted";
?>

It would be best to store $passphrase somewhere completely separate from your application configuration. Perhaps an admin user would be required to enter the passphrase when looking up this information. That way your passphrase is not stored in your config file or anywhere that an attacker could potentially gain access to it.

Troubleshooting

Make sure that the web server can write to the GnuPG Home directory. This obviously is not ideal, but seems to be required in the testing that I have done. I’ve been able to set ‘secring.gpg’ to be owned by root, but that does little good since the directory it is in has to be writable.

You can raise the error mode to GNUPG_ERROR_WARNING to generate PHP warnings on GnuPG errors. That might help to track down where errors are occurring

Database encryption made easy

I’ve always wondered how one would securely store sensitive information in a MySQL database. A recent project has given me the opportunity to work on it, and I’ve been impressed on how easy it is to implement. MySQL provides an easy interface for encrypting data before storing it in the database. Simply use the AES_ENCRYPT and AES_DECRYPT functions when reading or writing to a table.

Simply make your column a blob field, then use something like this to write to the table

(using a PEAR::DB syntax)

$db->query("
UPDATE sometable
SET    some_col = AES_ENCRYPT( ?, ?)
WHERE something_else = ?
" array( $sensitive_value, $encryption_key, $index));

and something like this to read it back out

$value = $db->getOne("
SELECT AES_DECRYPT( some_col, ?)
FROM   sometable
WHERE something_else = ?
", array( $encryption_key, $index));