Web Programming, Linux System Administation, and Entrepreneurship in Athens Georgia
If you are running a Perl script with the setuid bit, it actually runs a slightly modified version of Perl so that it is a bit more cautious. On a CentOS box, you need to install the ‘perl-suidperl’ package to get the necessary files installed. Otherwise you get an error like this:
[root@host bin]# ls -al myscript.pl -rws--S--- 1 mail mail 1218 Oct 1 13:09 myscript.pl [root@host bin]# ./myscript.pl Can't do setuid (cannot exec sperl)
I spent a good part of the day today working to make sure that a DomainKeys and DKIM server was properly signing messages. Here’s some of the confusing stuff I sorted out:
First off DomainKeys and DKIM are different slightly different technologies that do the same thing. My understanding is that Yahoo, Gmail, and others came up with DomainKeys as a solution to to solve email forgery. The EITF then tweaked the original DomainKeys spec a little and made their version called DKIM (DomainKeys Identified Mail). The two are very similar and I haven’t found anything that describes their technical differences.
First, I guess it helps to know basically how DomainKeys works. Essentially the administrator creates a Public/Private key pair on the mail server. The private key is used to sign the message while the public key is posted as a DNS TXT record for receiving mail servers to use for verification. Receiving mail servers then use the public key from DNS to decrypt the signature and make sure that the signed headers and message body were not modified in transit. I’ll describe DomainKeys here, and just understand that DKIM might implement things slightly differently.
The DomainKeys spec is available here
The DKIM spec is available here
Note that this doesn’t actually do any spam checking on its own. It simply verifies that the message came from an who it claims and wasn’t tampered with along the way. It is intended to be used for fighting phishing, and creating a reliable reputation-based services.
Originating mail server:
When a message is submitted to a mail server for signing, it uses the ‘From’, and ‘Sender’ headers, as well as the message body, in combination with the Private Key to generate a DomainKeys signature. This signature is added to the rest of the email headers and looks something like this:
DomainKey-Signature: a=rsa-sha1; s=mail; d=somedomain.com; c=simple; q=dns; b=PcrSGNIF8mZFBLPCz3UD5Na901YOAgWgHT5La7O799bnYkbDcqnpawVjB0fAcZQSb qxZk4iE5utyhaTij0NmS4tPx5qJHMzw3fwJKWXqV+7H5kyP8LkMRNLBDnzk9Om+
You’ll notice that the header contains several name=value pairs that describe things about the signature. Here are some important options that may be used in the Signature:
a= describes the hash algorithm used to create the key (usually rsa-sha1)
s= contains the selector (described later).
d= the domain to use for signing
q= the method for retrieving the public key (only dns is currently supported)
p= the base64 encoded signature
See section 3.3 in the DomainKeys specs for more details or Section 3.5 in the DKIM spec
After the MTA adds these headers, it forwards the message as normal to the recipient.
The receiving mail server can then perform some validation on messages that it receives which contain a DomainKey-Signature header. To do that the receiving MTA receives the message.
For every message (regardless of weather or not it has a DomainKeys-Signature header) it receives, it looks up the DomainKeys policy record for the senders domain. This is a text record with the name ‘_domainkey.DOMAINNAME.TLD’ where the ‘_domainkey’ part of that is always ‘_domainkey’. If the message does have a DomainKeys-Signature header, the -d parameter should match the domain portion of the ‘From’ header.
The policy record for a domain has these options:
Section 3.6.2 in the DomainKeys spec and 7.4 in DKIM specs
Of interest include:
o= can be either a hyphen (-) to indicate that the domain signs all email (and non-signed messages can be treated as invalid) or a tilde (~) to indicate that the domain signs some messages.
t= Set this to ‘y’ to indicate that you are testing. Receiving mail servers are not supposed to penalize you if the message does not verify and the ‘t’ flag is on.
A Policy record for ‘somedomain’ might look like this:
[root@host ~]# dig +short -ttxt _domainkey.somedomain.com "t=y; o=~; [email protected]"
(not that the ‘dig’ application is adding quotes around the record and adding backslashes before the semicolons.) the actual record in my DNS server looks like this:
t=y; o=~; [email protected]
You can use the DomainKeys Policy Record Checker to verify that your policy record is in an acceptable format.
Next, for messages that have a DomainKeys-Signature header, it uses the s= flag which describes the selector to create a DNS TXT query for
<selector>._domainkey.<domain>.<tld>. It will look something like this:
[root@host ~]# dig +short -ttxt mail._domainkey.domain.com "k=rsa; p=MHwwDQYJKoZI...a bunch of characters removed for formatting...UAfbN8pL/f6YwIDAQAB"
Of interest here are these options:
k= The key type (always ‘rsa’ for now)
g= (dkim only) used to limit the addresses that a public key can be used for. For instance to outsource some email without giving the third party permission to sign any address.
n= human-readable notes (not to be used for anything)
You can use the DomainKeys Selector Record Tester to verify that your selector is set up properly
After the receiving mail server has retrieved these records, it can decrypt the DomainKey-Signature header and verify that the message is authentic.
For end-to-end verification that everything works, there are a couple of testers available at Skylist (for DKIM and DomainKeys) and Deliverability.com
A few email auto-replies are also available the DomainKeys page
I’m about ready to wipe out my laptop’s hard drive and reinstall, but wanted to back it up first, just in case there is something I need on it in the future.  I was searching for open-source alternatives to Symantec Ghost and came across CloneZilla which looked like it would do the trick.  It is available as a Linux-based Live CD so that you can just boot off it and go.  With just a few minutes of playing with it, I was able to back up my entire laptop hard drive to a Samba share on a Windows PC.
I’ve been troubleshooting some possible problems on a mail server recently, and have been digging into TCP connections some. The ‘netstat’ command has a ‘-o’ option that displays some timers that are useful:
[mail]# netstat -on |grep 189.142.18.18 tcp 0 8 205.244.47.142:25 189.142.18.18:1256 ESTABLISHED on (17.00/4/0) tcp 0 452 205.244.47.142:25 189.142.18.18:2676 ESTABLISHED on (36.09/6/0)
This displays countdown timers for each TCP State. For example, if a connection is in FIN_WAIT and you run the command over and over with “watch”, you can watch the time count down to 0 and then go away. The man pages and documentation I could find didn’t explain the timers very well, so this is what I have learned by watching it. (read: this is not official).
When a connection is in the ESTABLISHED state, the timer can be either on or off. From what I can tell, the counter turns to ON when there is some kind of trouble with the connection. It looks like when a retransmission occurs, the timer is flipped ON, and then the countdown timer starts. The countdown timer has 3 numbers. The first is a countdown in seconds, the second is incremented for each retransmission, and the third one is always 0, so I’m not sure what it does
Now that I have a basic understanding of the output, I still have to figure out why these connections just hang. My guess at this point is that it is poorly written spamming software and maxed out bandwidth on all a whole much of compromised machines throughout the world that are hitting my mail servers.
I spent a good part of the day today troubleshooting memory problems on some postfix mail servers. Each smtpd process was using over 11 MB of Ram which seems really high. Each concurrent SMTP session has its own smtpd process, and with over 150 concurrent connections, that was using well over 1.5 GB of Ram.
[root@mail ~]# ps aux|grep -i smtpd |head -n1 postfix 3978 0.0 0.5 16096 11208 ? S 12:29 0:00 smtpd -n smtp -t inet -u
After some trial and error of temporarily disabling stuff in the main.cf file, I narrowed the memory usage to a regexp table in a transport map:
transport_maps = regexp:/etc/postfix/transport.regexp
The transport.regexp file had about 1400 lines in it to match various possible address variations for a stupid mailing list application. Each mailing list has 21 different possible commands (addresses). By combining those 21 different commands into a single regex, I was able to cut those 1400 lines down to about 70. Now the smtpd processes use just under 5mb each:
[root@mail ~]# ps aux|grep -i smtpd |head -n1 postfix 7634 0.0 0.2 9916 4996 ? S 13:31 0:00 smtpd -n smtp -t inet -u
So, by my math, a savings of about 6,000 kb of memory by removing 1300 lines from the regexp file means that each regexp used about 4.5 kb of memory. Overall, with 150+ simultaneous smtpd processes, that resulted in several hundred megs of memory saved on each mail server.
I ran into a problem today where a user would submit a new post in WordPress, and it would cause the web server to lock up. Restarting the web server would start Apache properly, and would serve static content fine until the user requested another page from WordPress where it would lock up again.
The configuration is a little odd, so it probably doesn’t happen to many users. In order for it to occur, you have to have the “WordPress Address” setting as a URL starting with ‘https’, and then write your post using a non-https URL. I tracked this down to a problem with the cron function built into wordpress. Specifically this bit of code in the spawn_cron() function in includes/cron.php
$cron_url = get_option( 'siteurl' ) . '/wp-cron.php';
$parts = parse_url( $cron_url );
if ($parts['scheme'] == 'https') {
// support for SSL was added in 4.3.0
if (version_compare(phpversion(), '4.3.0', '>=') && function_exists('openssl_open')) {
$argyle = @fsockopen('ssl://' . $parts['host'], $_SERVER['SERVER_PORT'], $errno, $errstr, 0.01);
} else {
return false;
}
} else {
$argyle = @ fsockopen( $parts['host'], $_SERVER['SERVER_PORT'], $errno, $errstr, 0.01 );
}
if ( $argyle )
fputs( $argyle,
"GET {$parts['path']}?check=" . wp_hash('187425') . " HTTP/1.0\\r\\n\\r\\n"
. "Host: {$_SERVER['HTTP_HOST']}rnrn"
);
The line that says:
$argyle = @fsockopen('ssl://' . $parts['host'], $_SERVER['SERVER_PORT'], $errno, $errstr, 0.01);
Assumes that you are hitting the current page on the same server/port as that returned by get_option( ‘siteurl’ ). Since the user was hitting the non-https version of the site, that would cause this code in the spawn_cron() function to connect to port 80 and try to establish an SSL connection. WordPress would get that request as “\x80|\x01\x03\x01”, and issue it the home page, which would, in-turn, re-run the cron function again. That sub-request would redo the same thing over, and that would continue until Apache ran out of connections. At that point it would try to request the page again, and would wait endlessly for a connection to open up, and never would.
So, to solve, I added one line, and modified another like this:
[root@server wp-includes]# diff cron.php cron.php.original
90,91c90
90,91c90
< $port = isset($parts['port']) ? $parts['port'] : 443;
< $argyle = @fsockopen('ssl://' . $parts['host'], $port, $errno, $errstr, 0.01);
---
> $argyle = @fsockopen('ssl://' . $parts['host'], $_SERVER['SERVER_PORT'], $errno, $errstr, 0.01);
96,97c95
< $port = isset($parts['port']) ? $parts['port'] : 80;
< $argyle = @ fsockopen( $parts['host'], $port, $errno, $errstr, 0.01 );
---
> $argyle = @ fsockopen( $parts['host'], $_SERVER['SERVER_PORT'], $errno, $errstr, 0.01 );
That makes it consider the port of the url returned by get_option( ‘siteurl’ ), instead of using the port you are currently connected on. It defaults to port 443 if the url begins with https, and port 80 if not.
I posted the fix to the wordpress forums at https://wordpress.org/support/topic/130492Â Hopefully this gets included in future releases of WordPress
When testing out Web or Mail servers, I often find myself telneting to the server and issuing raw commands directly. Doing this is incredibly useful for tracking down the source of many problems. Until now, I have never know how to do the same thing over encrypted channels like HTTPS or POP3S. However, I just discovered that the openSSL library has a simple tool that works great. Run the command:
openssl s_client -connect hostname:port
That will perform all of the SSL handshake and display the output for you, and then give you a regular prompt, just like telnet would. For SMTP over TLS it is a little more complicated because you generally would connect to the remote server and then issue the STARTTLS command to negotiate encryption. In that case, you could use the command:
openssl s_client -starttls smtp -crlf -connect host:port
That will tell the openssl client to connect, and send ‘STARTTLS’ before attempting to negotiate the encryption. After that, you’ll end up with a 220 response at which to proceed with your normal SMTP session
Modern versions of openSSL also allow STARTTLS with pop3:
openssl s_client -starttls pop3 -connect host:port
With my previous success with greylisting, I have decided that it definitely works well and is worth the little bit of effort it takes to get is installed.   Configuring postifx was very simple, and I (unfortunately) run several mail server that run Qmail.  After a few minutes of googling, I decided on qgreylist, which was the simplest implementation by far.
Several of the alternatives required patching and recompiling qmail, which I definitely didn’t want to do. qgreylist is just a simple Perl script that runs “in between” the tcpserver and the qmail-smtpd process.  You download it, change the path to it’s working directory, and tweak a couple other variables. Then copy it into a permanent location, and configure qmail’s smtpd process to send messages to it.  It took a little longer than postgrey, but not too bad.
I recently installed CentOS 5 on a server with a Promise PDC20621 SATA Raid card in it (according to lspci). This particular card, of course, is a FAKE raid device, meaning that the physical card is nothing more than a regular SATA controller, and they provide drivers that emulate RAID functionality. This is supposed to be useful for Windows users that don’t have a native software raid service available, but it is kindof useless for Linux since most distros provide md for creating a software raid device.
When trying to create a new software raid array, I would get a bunch of errors about the devices being busy, like this:
[root@www ~]# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 64K mdadm: Cannot open /dev/sda1: Device or resource busy mdadm: Cannot open /dev/sdb1: Device or resource busy mdadm: Cannot open /dev/sdc1: Device or resource busy mdadm: Cannot open /dev/sdd1: Device or resource busy
lsof didn’t show any processes that were using these files, and it took a little while to finally find out that ‘dmraid’ was the culprit. dmraid is the linux driver for fake raid controllers like the Promise FastTrak and nVidia on-board SATA controllers. From what I could tell, it is loaded from initrd and automatically attaches itself to any partitions that are of type ‘fd’ (Linux raid autodetect).
After a few hours of googling for answers, I had become pretty familiar with the topic. Many of the search results were from people trying to get mdraid working for these devices before it was stable and widely included in distros.
Unfortunately, it looks like the default CentOS 5 install has the dmraid drivers built into the initrd, and there was no way to disable it from taking control of the drives. I tried looking for an argument to pass to the kernel to disable dmraid support, but couldn’t find anything. A few of the posts and emails that I came across on the subject suggested removing the ‘dmraid’ package, and a few people appeared to have some success with that. But when I tried a ‘yum erase dmraid’ on my box, it wanted to remove the kernel, which would probably be bad.
After a little more searching, I found that mkinitrd had an option to rebuild the initrd without dmraid support. The was an upgrade available for my kernel, so I did a ‘yum update’ to install a new one, which also gave me one to fall-back to if this didn’t work. Once the new kernel was running, I installed the ‘kernel-devel’ and ‘kernel-headers’ packages to pull down some necessary headers, then ran this command to create a new initrd without the troublesome dmraid drivers:
mkinitrd --omit-dmraid /boot/NO_DMRAID_initrd-2.6.18-8.1.6.el5.img 2.6.18-8.1.6.el5
Then, simply change /etc/grub.conf to create an option that pointed to my new initrd. My /etc/grub.conf looks like this:
default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu ## My new non-dmraid boot option title CentOS (2.6.18-8.1.6.el5) WITHOUT DMRAID GARBAGE root (hd0,0) kernel /vmlinuz-2.6.18-8.1.6.el5 ro root=/dev/hda1 initrd /NO_DMRAID_initrd-2.6.18-8.1.6.el5.img ## The regular option title CentOS (2.6.18-8.1.6.el5) root (hd0,0) kernel /vmlinuz-2.6.18-8.1.6.el5 ro root=/dev/hda1 initrd /initrd-2.6.18-8.1.6.el5.img ## My working backup option: title CentOS (2.6.18-8.el5) root (hd0,0) kernel /vmlinuz-2.6.18-8.el5 ro root=/dev/hda1 initrd /initrd-2.6.18-8.el5.img
Now, I just rebooted off the first option, and it didn’t load all of the dmraid junk. I can now access the partitions without the ‘resource busy’ problem, and create a software raid array like I’m used to.
I administer about 20 different web applications, each of which uses Apache authentication to control access. In the past, I’ve just used simple htpasswd authentication because it works and is readily available. However when adding or removing employee’s access, it required pretty manual editing of all of the htpasswd files every time that we added or removed and employee
I just starting using mod_auth_mysql which provides a way to centralize the authentication. It is available as a package on any distro that I’ve used, and is pretty simple to configure. Just create a database with the following tables:
CREATE TABLE users ( user_name CHAR(30) NOT NULL, user_passwd CHAR(20) NOT NULL, PRIMARY KEY (user_name) ); CREATE TABLE groups ( user_name CHAR(30) NOT NULL, user_group CHAR(20) NOT NULL, PRIMARY KEY (user_name, user_group) );
Populate the users table with username/passwords taken straight from the .htpasswd file. Optionally, you can make users a member of a group via the groups table. Create a database user with permission to SELECT from those two tables.
Then configure the following in the Apache config or .htaccess file for each your web applications:
AuthName "Some Webapp" AuthType Basic AuthMySQLEnable on AuthMySQLHost myauthserver.someplace.com AuthMySQLUser YourDatabaseName AuthMySQLPassword YourDatabaseUserPassword AuthMySQLDB YourDatabaseName AuthMySQLUserTable users AuthMySQLNameField user_name AuthMySQLPasswordField user_passwd AuthMySQLGroupTable groups AuthMySQLGroupField user_group require valid-user #require group ThisApp
Now you can centrally manage your Apache authentication. Uncomment the ‘require group’ line and add an appropriate entry in the groups table for any users you want to allow specifically to this app.
© 2025 Brandon Checketts
Theme by Anders Noren — Up ↑