Month: May 2008

Creating a Permanent SSH Tunnel Between Linux Servers

I recently had a need to create a permanent SSH tunnel between Linux servers. My need was to allow regular non-encrypted MySQL connections over an encrypted tunnel, but there could be many other uses as well. Google can identify plenty of resources regarding the fundamental SSH commands for port forwarding but I didn’t ever find a good resource for setting up a connection and ensuring that it remains active, which is what I hope to provide here.

The SSH commands for port forwarding can be found in the ssh man page. The steps described here will create an unprivileged user named ‘tunnel’ on each server. That user will then be used to create the tunnel and run a script via cron to ensure that it remains up.

First, select one of the servers that will initiate the SSH connection. SSH allows you to map both local and remote ports, so it doesn’t really matter which end of the connection you choose to initiate the connection. I’ll refer to the box that initiates the connection as Host A, and the box that we connect to as Host B.

Create a ‘tunnel’ user on Host A:

[root@hosta ~]# useradd -d /home/tunnel tunnel
[root@hosta ~]# passwd tunnel       ## Set a strong password
[root@hosta ~]# su - tunnel           ## Become the user 'tunnel'

Now create a public/private key pair:

[tunnel@hosta ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tunnel/.ssh/id_rsa):    ## hit enter to accept the default
Enter passphrase (empty for no passphrase):                           ## don't use a  passphrase
Enter same passphrase again:
Your identification has been saved in /home/tunnel/.ssh/id_rsa.
Your public key has been saved in /home/tunnel/.ssh/id_rsa.pub.
The key fingerprint is:
6f:30:b8:e1:36:49:74:b9:32:68:6e:bf:3e:62:d3:c2 tunnel@hosta

Now cat out the id_rsa.pub file which contains the public key that we will need to put on host b:

[tunnel@hosta ~]# cat /.ssh/id_rsa.pub
ssh-rsa blahAAAAB3NzaC1yc2EAAAABIwAAAQEA......6BEKKCxTIxgBqjLP tunnel@hosta

Now create a ‘tunnel’ user on Host B and save the public key for tunnel@hosta in the authorized_keys file

[root@hostb ~]# useradd -d /home/tunnel tunnel
[root@hostb ~]# passwd tunnel       ## Set a strong password
[root@hostb ~]# su - tunnel
[tunnel@hostb ~]# mkdir .ssh
[tunnel@hostb ~]# vi .ssh/authorized_keys   ## Now paste in the public key for tunnel@hosta

At this point you should be able to ssh from tunnel@hosta to tunnel@hostb without using a password. Depending on your configuration, you might need to allow the user ‘tunnel’ in /etc/ssh/sshd_config. You might also set some SSH options like the destination port in ~/.ssh/config.

Now, create this script as hosta:/home/tunnel/check_ssh_tunnel.sh

createTunnel() {
    /usr/bin/ssh -f -N -L13306:hostb:3306 -L19922:hostb:22 tunnel@hostb
    if [[ $? -eq 0 ]]; then
        echo Tunnel to hostb created successfully
    else
        echo An error occurred creating a tunnel to hostb RC was $?
    fi
}
## Run the 'ls' command remotely.  If it returns non-zero, then create a new connection
/usr/bin/ssh -p 19922 tunnel@localhost ls
if [[ $? -ne 0 ]]; then
    echo Creating new tunnel connection
    createTunnel
fi

Save that file and make it executable:

chmod 700 ~/check_ssh_tunnel.sh

This script will attempt to SSH to localhost port 19922 and run the ‘ls’ command. If that fails, it will attempt to create the SSH tunnel. The command to create the SSH tunnel will tunnel local port 13306 to port 3306 on hostb. You should modify that as necessary for your configuration. It will also create a tunnel for local port 19922 to port 22 on hostb which the script uses for testing the connection.

Now just add that script to the user ‘tunnel’s crontab to check every few minutes, and it will automatically create a tunnel and reconnect it if something fails. When it does create a new connection it will send an email to the ‘tunnel’ user, so you can create a .forward file to forward those messages to you.

Identifying Weak SSL or SSH Keys on CentOS

With the Debian OpenSSL problems, everybody is wanting to know if their server is vulnerable to any attacks. Fortunately, CentOS machines shouldn’t be directly affected and have fewer issues than if you are using Debian or Ubuntu derivatives. Unfortunately though, your system may still be vulnerable if you have any users that may have generated their keys on an affected machine. So it is definitely necessary to check, even if you are not running a distribution that is affected.

This is the steps I have been going through to look for any weak keys on a CentOS server

Download the weak key detector provided by Debian (there may be better tools to use by now). It is available on the announcement page. (I’m not linking to it intentionally).

[root@host ~]# cd /tmp
[root@host tmp]# wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
--20:44:31--  http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
Resolving security.debian.org... 128.31.0.36, 130.89.175.54, 212.211.132.32, ...
Connecting to security.debian.org|128.31.0.36|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14231783 (14M) [application/x-gzip]
Saving to: `dowkd.pl.gz'

100%[================================>] 14,231,783  6.42M/s   in 2.1s

20:44:33 (6.42 MB/s) - `dowkd.pl.gz' saved [14231783/14231783]

[root@host tmp]# gunzip dowkd.pl.gz

Then check a couple known files – Start out with your SSH host keys in /etc/ssh/

[root@host tmp]# perl dowkd.pl file /etc/ssh/*key*
/etc/ssh/ssh_host_dsa_key:1: warning: unparsable line
/etc/ssh/ssh_host_key:1: warning: unparsable line
summary: keys found: 4, weak keys: 0

Then check any certificates in /etc/pki/tls:

[root@host tmp]# for file in `find /etc/pki/tls/ -name "*key"`; do echo -n "$file - "; perl /tmp/dowkd.pl file $file; done
/etc/pki/tls/certs/mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/certs/secure.mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/private/localhost.key - summary: keys found: 1, weak keys: 0

Any for any SSL certificates that Apache might be using in /etc/httpd/conf/ssl.key/:

[root@host tmp]# perl dowkd.pl  file /etc/httpd/conf/ssl.key/*
summary: keys found: 4, weak keys: 0

And finally, any users who might have authorized a weak key via their authorized_users file:

[root@host tmp]# for file in `find / -name authorized_keys`; do echo -n "$file "; perl dowkd.pl file $file; done
/home/someuser/.ssh/authorized_keys summary: keys found: 6, weak keys: 0
summary: keys found: 7, weak keys: 0

Note that any that say ‘warning: no blacklist found’ means that the tool didn’t have a blacklist for the key type, so they might need to be checked with another tool unless you are sure that they are okay.

You should also check any other locations for keys. The locations could vary widely on different machines, depending on the configuration of your server. Those locations specified above should cover most of the default locations on a CentOS 4 or CentOS 5 server, but every server is different. If you don’t find it now, its quite likely that an attacker will later.

Avoid Entering an SSL PassPhrase During Apache Startup with SSLPassPhraseDialog

When creating an SSL key for use with an Apache web server, you have an option of specifying a pass phrase on the certificate. This is a security feature that prevents somebody from being able to use the certificate, even if they have your key file. In theory it is a good idea, but because the key file requires a pass phrase any time it is read, that means that Apache has to prompt a user for the pass phrase.

Most people want Apache to start up automatically, so they remove the pass phrase completely. Apache provides an alternative though, which may or may not be useful. The trick to this is in the Apache SSLPassPhraseDialog setting. This setting defaults to ‘builtin’ which prompts the user for it when Apache starts. Alternatively, you can specify a script that reads the server name from STDIN and provides the pass phrase on STDOUT.

The apache config would look something like this:

SSLPassPhraseDialog  exec:/sbin/pp-exec

Then, you can create as simple or as complicated of a script as you would like in /sbin/pp-exec. Here is a very simple perl script with the passphrases hard-coded:

#!/usr/bin/perl
$server = $ARGV[0];
if ($server eq 'www.mydomain.com:443') {
    echo 'This is my pass phrase';
} elsif ($server eq 'www.otherdomain.com:443') {
    echo 'This is a different pass phrase';
}

Make sure that your script is readable and executable ONLY by root so that it is properly protected. Of course, if somebody can read your key file, that means that likely already have root access, which makes this simple script kindof pointless.

Fortunately, you can make that script as complicated as you want, perhaps saving the actual pass phrase in a database or LDAP. The really paranoid might come up with some way to text message an administrator and have them text the pass phrase back – or something equally complicated so that the pass phrase isn’t actually stored anywhere that an attacker could find it.

I guess the point is to have the pass phrase separate from the key file so that somebody would need both to make any use of it.

bcSpamblock Updated to Version 1.3

Thanks to jontiw for pointing out a potential problem in my bcSpamblock code.  He noted the the PHP crypt() function returns the salt along with the encrypted value.  My code was passing the salt to the visitor so that an attacker could potentially learn the salt value that a website was using and create valid responses.

I modified the code to strip out that salt before passing it to the user.  I also modified the data used to create the salt so that previous vulnerable version doesn’t use the same value for the site.  The wordpress plugin has also been updated as well.

I was happy to see other people looking through my code and pointing this type of issue out.

Sending Yourself a File as an Attachment From the Command Line

There are many occasions where I’ll have a file on a Linux machine that I want to email myself. If it is a text file, it’s pretty easy to just pipe the file to a mail command like this:

cat /home/brandon/some-file.txt | mail -s "Here is the file" some@body.com

That works well for the simple case of a text file. However, there are often cases where it either isn’t a text file, or maybe the text file is too large. Sending the file as an attachment would be ideal, but you can’t just pipe any type of data to the mail command.

Instead, I just learned how to use ‘mutt’ to send a file as an attachment. The command is something like this:

echo "see attached file" | mutt -s "Here is the file" -a /home/brandon/some-file.bin some@body.com

Where ‘See attached file’ is the body of the message. The -s argument of ‘Here is the file’ is the subject. /home/brandon/some-file.bin is the file I want to attach. And some@body.com is who I want to send the message to.

This would work well for quick files that you want to transfer to your PC – perhaps instead of using a file-transfer program like FTP. It can also be used in scripts. I had one customer, for example, who used this to email themselves a mysqldump of their database each night as a sortof backup.

Quick MRTG Install With Some Useful System Metrics

I often have a need to monitor some basic system metrics such as memory usage, disk space free, load average, and network usage. MRTG is an ideal solution because it is lightweight and can graph just about anything on a system. Many people only think of MRTG as a tool to graph network interfaces, but because it is just a way to use rrdtool it can be be used for practically anything.

Getting it installed is pretty easy as well. Most distributions provide a package for it, so it is as simple as running ‘yum install mrtg’ or ‘apt-get install mrtg’. Unfortunately the basic package is pretty raw and doesn’t by default know how to obtain much useful data. Over time I’ve compiled a /etc/mrtg/ directory that does a lot of the common things that I like to monitor. Simply uncompress this file to your /etc/mrtg directory and you should have some common useful metrics. You can further customize it per-server if you’d like to monitor some additional things.