Author: Brandon (Page 15 of 29)

Creating a Permanent SSH Tunnel Between Linux Servers

May 28, 2008 / Brandon / 17 Comments

I recently had a need to create a permanent SSH tunnel between Linux servers. My need was to allow regular non-encrypted MySQL connections over an encrypted tunnel, but there could be many other uses as well. Google can identify plenty of resources regarding the fundamental SSH commands for port forwarding but I didn’t ever find a good resource for setting up a connection and ensuring that it remains active, which is what I hope to provide here.

The SSH commands for port forwarding can be found in the ssh man page. The steps described here will create an unprivileged user named ‘tunnel’ on each server. That user will then be used to create the tunnel and run a script via cron to ensure that it remains up.

First, select one of the servers that will initiate the SSH connection. SSH allows you to map both local and remote ports, so it doesn’t really matter which end of the connection you choose to initiate the connection. I’ll refer to the box that initiates the connection as Host A, and the box that we connect to as Host B.

Create a ‘tunnel’ user on Host A:

[root@hosta ~]# useradd -d /home/tunnel tunnel
[root@hosta ~]# passwd tunnel       ## Set a strong password
[root@hosta ~]# su - tunnel           ## Become the user 'tunnel'

Now create a public/private key pair:

[tunnel@hosta ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tunnel/.ssh/id_rsa):    ## hit enter to accept the default
Enter passphrase (empty for no passphrase):                           ## don't use a  passphrase
Enter same passphrase again:
Your identification has been saved in /home/tunnel/.ssh/id_rsa.
Your public key has been saved in /home/tunnel/.ssh/id_rsa.pub.
The key fingerprint is:
6f:30:b8:e1:36:49:74:b9:32:68:6e:bf:3e:62:d3:c2 tunnel@hosta

Now cat out the id_rsa.pub file which contains the public key that we will need to put on host b:

[tunnel@hosta ~]# cat /.ssh/id_rsa.pub
ssh-rsa blahAAAAB3NzaC1yc2EAAAABIwAAAQEA......6BEKKCxTIxgBqjLP tunnel@hosta

Now create a ‘tunnel’ user on Host B and save the public key for tunnel@hosta in the authorized_keys file

[root@hostb ~]# useradd -d /home/tunnel tunnel
[root@hostb ~]# passwd tunnel       ## Set a strong password
[root@hostb ~]# su - tunnel
[tunnel@hostb ~]# mkdir .ssh
[tunnel@hostb ~]# vi .ssh/authorized_keys   ## Now paste in the public key for tunnel@hosta

At this point you should be able to ssh from tunnel@hosta to tunnel@hostb without using a password. Depending on your configuration, you might need to allow the user ‘tunnel’ in /etc/ssh/sshd_config. You might also set some SSH options like the destination port in ~/.ssh/config.

Now, create this script as hosta:/home/tunnel/check_ssh_tunnel.sh

createTunnel() {
    /usr/bin/ssh -f -N -L13306:hostb:3306 -L19922:hostb:22 tunnel@hostb
    if [[ $? -eq 0 ]]; then
        echo Tunnel to hostb created successfully
    else
        echo An error occurred creating a tunnel to hostb RC was $?
    fi
}
## Run the 'ls' command remotely.  If it returns non-zero, then create a new connection
/usr/bin/ssh -p 19922 tunnel@localhost ls
if [[ $? -ne 0 ]]; then
    echo Creating new tunnel connection
    createTunnel
fi

Save that file and make it executable:

chmod 700 ~/check_ssh_tunnel.sh

This script will attempt to SSH to localhost port 19922 and run the ‘ls’ command. If that fails, it will attempt to create the SSH tunnel. The command to create the SSH tunnel will tunnel local port 13306 to port 3306 on hostb. You should modify that as necessary for your configuration. It will also create a tunnel for local port 19922 to port 22 on hostb which the script uses for testing the connection.

Now just add that script to the user ‘tunnel’s crontab to check every few minutes, and it will automatically create a tunnel and reconnect it if something fails. When it does create a new connection it will send an email to the ‘tunnel’ user, so you can create a .forward file to forward those messages to you.

Identifying Weak SSL or SSH Keys on CentOS

May 22, 2008 / Brandon / 1 Comment

With the Debian OpenSSL problems, everybody is wanting to know if their server is vulnerable to any attacks. Fortunately, CentOS machines shouldn’t be directly affected and have fewer issues than if you are using Debian or Ubuntu derivatives. Unfortunately though, your system may still be vulnerable if you have any users that may have generated their keys on an affected machine. So it is definitely necessary to check, even if you are not running a distribution that is affected.

This is the steps I have been going through to look for any weak keys on a CentOS server

Download the weak key detector provided by Debian (there may be better tools to use by now). It is available on the announcement page. (I’m not linking to it intentionally).

[root@host ~]# cd /tmp
[root@host tmp]# wget https://security.debian.org/project/extra/dowkd/dowkd.pl.gz
--20:44:31--  https://security.debian.org/project/extra/dowkd/dowkd.pl.gz
Resolving security.debian.org... 128.31.0.36, 130.89.175.54, 212.211.132.32, ...
Connecting to security.debian.org|128.31.0.36|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14231783 (14M) [application/x-gzip]
Saving to: `dowkd.pl.gz'

100%[================================>] 14,231,783  6.42M/s   in 2.1s

20:44:33 (6.42 MB/s) - `dowkd.pl.gz' saved [14231783/14231783]

[root@host tmp]# gunzip dowkd.pl.gz

Then check a couple known files – Start out with your SSH host keys in /etc/ssh/

[root@host tmp]# perl dowkd.pl file /etc/ssh/*key*
/etc/ssh/ssh_host_dsa_key:1: warning: unparsable line
/etc/ssh/ssh_host_key:1: warning: unparsable line
summary: keys found: 4, weak keys: 0

Then check any certificates in /etc/pki/tls:

[root@host tmp]# for file in `find /etc/pki/tls/ -name "*key"`; do echo -n "$file - "; perl /tmp/dowkd.pl file $file; done
/etc/pki/tls/certs/mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/certs/secure.mydomain.ca.key - summary: keys found: 1, weak keys: 0
/etc/pki/tls/private/localhost.key - summary: keys found: 1, weak keys: 0

Any for any SSL certificates that Apache might be using in /etc/httpd/conf/ssl.key/:

[root@host tmp]# perl dowkd.pl  file /etc/httpd/conf/ssl.key/*
summary: keys found: 4, weak keys: 0

And finally, any users who might have authorized a weak key via their authorized_users file:

[root@host tmp]# for file in `find / -name authorized_keys`; do echo -n "$file "; perl dowkd.pl file $file; done
/home/someuser/.ssh/authorized_keys summary: keys found: 6, weak keys: 0
summary: keys found: 7, weak keys: 0

Note that any that say ‘warning: no blacklist found’ means that the tool didn’t have a blacklist for the key type, so they might need to be checked with another tool unless you are sure that they are okay.

You should also check any other locations for keys. The locations could vary widely on different machines, depending on the configuration of your server. Those locations specified above should cover most of the default locations on a CentOS 4 or CentOS 5 server, but every server is different. If you don’t find it now, its quite likely that an attacker will later.

Avoid Entering an SSL PassPhrase During Apache Startup with SSLPassPhraseDialog

May 17, 2008 / Brandon / 0 Comments

When creating an SSL key for use with an Apache web server, you have an option of specifying a pass phrase on the certificate. This is a security feature that prevents somebody from being able to use the certificate, even if they have your key file. In theory it is a good idea, but because the key file requires a pass phrase any time it is read, that means that Apache has to prompt a user for the pass phrase.

Most people want Apache to start up automatically, so they remove the pass phrase completely. Apache provides an alternative though, which may or may not be useful. The trick to this is in the Apache SSLPassPhraseDialog setting. This setting defaults to ‘builtin’ which prompts the user for it when Apache starts. Alternatively, you can specify a script that reads the server name from STDIN and provides the pass phrase on STDOUT.

The apache config would look something like this:

SSLPassPhraseDialog  exec:/sbin/pp-exec

Then, you can create as simple or as complicated of a script as you would like in /sbin/pp-exec. Here is a very simple perl script with the passphrases hard-coded:

#!/usr/bin/perl
$server = $ARGV[0];
if ($server eq 'www.mydomain.com:443') {
    echo 'This is my pass phrase';
} elsif ($server eq 'www.otherdomain.com:443') {
    echo 'This is a different pass phrase';
}

Make sure that your script is readable and executable ONLY by root so that it is properly protected. Of course, if somebody can read your key file, that means that likely already have root access, which makes this simple script kindof pointless.

Fortunately, you can make that script as complicated as you want, perhaps saving the actual pass phrase in a database or LDAP. The really paranoid might come up with some way to text message an administrator and have them text the pass phrase back – or something equally complicated so that the pass phrase isn’t actually stored anywhere that an attacker could find it.

I guess the point is to have the pass phrase separate from the key file so that somebody would need both to make any use of it.

bcSpamblock Updated to Version 1.3

May 14, 2008 / Brandon / 0 Comments

Thanks to jontiw for pointing out a potential problem in my bcSpamblock code.Â He noted the the PHP crypt() function returns the salt along with the encrypted value.Â My code was passing the salt to the visitor so that an attacker could potentially learn the salt value that a website was using and create valid responses.

I modified the code to strip out that salt before passing it to the user.Â I also modified the data used to create the salt so that previous vulnerable version doesn’t use the same value for the site.Â The wordpress plugin has also been updated as well.

I was happy to see other people looking through my code and pointing this type of issue out.

Sending Yourself a File as an Attachment From the Command Line

May 8, 2008 / Brandon / 1 Comment

There are many occasions where I’ll have a file on a Linux machine that I want to email myself. If it is a text file, it’s pretty easy to just pipe the file to a mail command like this:

cat /home/brandon/some-file.txt | mail -s "Here is the file" [email protected]

That works well for the simple case of a text file. However, there are often cases where it either isn’t a text file, or maybe the text file is too large. Sending the file as an attachment would be ideal, but you can’t just pipe any type of data to the mail command.

Instead, I just learned how to use ‘mutt’ to send a file as an attachment. The command is something like this:

echo "see attached file" | mutt -s "Here is the file" -a /home/brandon/some-file.bin [email protected]

Where ‘See attached file’ is the body of the message. The -s argument of ‘Here is the file’ is the subject. /home/brandon/some-file.bin is the file I want to attach. And [email protected] is who I want to send the message to.

This would work well for quick files that you want to transfer to your PC – perhaps instead of using a file-transfer program like FTP. It can also be used in scripts. I had one customer, for example, who used this to email themselves a mysqldump of their database each night as a sortof backup.

Quick MRTG Install With Some Useful System Metrics

May 1, 2008 / Brandon / 0 Comments

I often have a need to monitor some basic system metrics such as memory usage, disk space free, load average, and network usage. MRTG is an ideal solution because it is lightweight and can graph just about anything on a system. Many people only think of MRTG as a tool to graph network interfaces, but because it is just a way to use rrdtool it can be be used for practically anything.

Getting it installed is pretty easy as well. Most distributions provide a package for it, so it is as simple as running ‘yum install mrtg’ or ‘apt-get install mrtg’. Unfortunately the basic package is pretty raw and doesn’t by default know how to obtain much useful data. Over time I’ve compiled a /etc/mrtg/ directory that does a lot of the common things that I like to monitor. Simply uncompress this file to your /etc/mrtg directory and you should have some common useful metrics. You can further customize it per-server if you’d like to monitor some additional things.

Enabling PHP Syntax Highlighting for .html Files in vim

April 16, 2008 / Brandon / 0 Comments

vim has a lot of default file types and syntax dictionaries set up. It determines which syntax highlighting to do based on a files extension. I have a bunch of .html files that contain PHP code and finally got frustrated enough with the incorrect highlighting to figure out how to correct it. After bit of research and experimentation, but I was finally able to do it by creating ~/.vim/filetype.vim with this content:

    if exists("did_load_filetypes")
      finish
    endif
    augroup filetypedetect
      au! BufRead,BufNewFile *.html     setfiletype php
    augroup END

Determining What a Server is Used For

April 5, 2008 / Brandon / 0 Comments

When looking at an existing server, it is sometimes difficult to know everything that the server does. I have had many instances when hired for a new job some of them even include some dental services as discounts on teeth whitening, or when looking at a client’s server where I have to find out everything that the server is used for without knowing much about it. Often times there are many more uses than it was originally designed for.

Here is a list of things that I usually check to try and identify what a server is used for:

1- Look at any processes listening on a network socket. I use ‘netstat -lnp’ to identify any listening sockets and what processes are using them. Its common to see SSH, Apache, MySQL, and a mail server. Sometimes there are other things that you should know about, such as FTP, a control panel (ie: webmin on port 10000), and a variety of other processes

2- Cron jobs – A lot of systems have automated processes that run periodically from cron. Make sure to check all of the various cron locations:

The main system crontab:

/etc/crontab

Drop location for cron jobs typically installed with packages:

/etc/cron.d/*

Periodically run jobs:

/etc/cron.*/*

(ie /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly, etc)

User crontabs

/var/spool/cron/*

3- Look at running processes. I use ‘ps auxf’ to identify any other processes that might be running

4- Processes run at boot. On Redhat and derivitives, use ‘ chkconfig –list|grep “:on” ‘ to see all processes that start when the machine boots.

5- Websites configured in Apache: Run ‘ apachectl -t -D DUMP_VHOSTS ‘ to see what Virtual Host are configured in Apache

6- Consider the server’s name, and reverse DNS for any IP’s assigned to it. These may give some hints as to things the server is being used for. For example, if a server has the name ‘mail.mydomain.com’ associated with it somehow, you should probably take a closer look at the mail configuration than you might initially think to do.

That should be a pretty good start of identifying everything that a particular server is used for. Please leave a comment if there is something else that I should add to the list.

Problems to Anticipate When Upgrading From PHP4 to PHP5, and MySQL4 to MySQL5

March 19, 2008 / Brandon / 0 Comments

A client website just upgraded from PHP4 to PHP5 and MySQL4 to MySQL5 and completely broke. Doing such significant upgrades should have been tested first, but for some reason didn’t happen. I got invited to fix and ran across several problems:

MySQL queries containing some explicit JOINs broke. A simple query like this doesn’t work in MySQL5:

SELECT table1.*, table2.*
FROM table1, table2
LEFT JOIN table3 on table1.col1 = table3.col1

In MySQL 5, the JOIN operator now has a higher precedence than the comma operator, so it interprets the query differently. See this post or the MySQL documentation for more information. The quick fix is to put parenthesis around the tables in the FROM statement, like this:

SELECT table1.*, table2.*
FROM (table1, table2)
LEFT JOIN table3 on table1.col1 = table3.col1

The other significant problem was in the upgrade from PHP4 to PHP5, the XML parsing functions are completely different. PHP 4 used the domxml extentions, where PHP 5 uses a newere DOM extention.

From https://www.php.net/manual/en/ref.domxml.php:

It will, however, never be released with PHP 5, and will only be distributed with PHP 4. If you need DOM XML support with PHP 5 you can use the DOM extension. This domxml extension is not compatible with the DOM extension.

The solution for fixing this, however is quite a bit more complicated. I had to rewrite the XML producing scripts to use the new functionality. Fortunately, the new DOM functionality is pretty straightforward and easier to write, so porting it from one to the other is fairly straightforward, but does require some effort.

MyHosting.com Features are Seriously Lacking

March 10, 2008 / Brandon / 0 Comments

I got my first SourceForge Marketplace job a few days ago for an installation of Awstats for a customer.Â I’ve installed Awstats plenty of times and I sometimes have a qwirk or two, but it is generally pretty painless.

Not for an account at MyHosting.com.

Their features are seriously lacking and their security prevention measures too intrusive. Â I should have know that I was in for trouble when I realized that they don’t offer any way to create cron jobs.Â I ended up having to use a free account with Remote-Cron.com instead, which works, but is not ideal.

The layout of their directories was not very intuitive, and took too much tinkering to figure out.Â But the really annoying part is that they limit the file size created by the web server to just 100k. Â I was trying to parse a 40 MB of log files for a given month, and awstats would quickly die with an Internal Server Error. Â It took a while to finally figure out that it was choking because of the 100k file size limit.

I’ve submitted a ticket to their support asking to increase the file size limit, but will be interested to see if they will allow it.