Increasing the number of simultaneous SASL authentication servers with Postfix

Posted on July 5th, 2012 in General,Linux System Administration,Mail by Brandon

I had a customer complaining lately that messages sent via Gmail to one of my mail servers was occasionally receiving SMTP Authentication failures and bounce backs from Gmail. Fortunately he noticed it happening mainly when he sent a messages to multiple recipients and was able to send me some of the bounces for me to track it down pretty specifically in the postfix logs.

The Error message via Gmail was:

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 535 535 5.7.0 Error: authentication failed: authentication failure (SMTP AUTH failed with the remote server) (state 7).

This was a little odd, because the SMTP AUTH failure is what I would typically expect with a mistyped username and password. However, I could see that plenty of messages were being sent from the same client. By looking at the specific timestamp of the bounced message, I tracked down the relevant log segment shown below. It indicates 5 concurrent SMTPD sessions where the SASL authentication was successful on 4 of them and failed on the 5th.

Jul  5 12:43:39 mail postfix/smtpd[13602]: connect from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:39 mail postfix/smtpd[13602]: setting up TLS connection from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:39 mail postfix/smtpd[14113]: connect from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:39 mail postfix/smtpd[14113]: setting up TLS connection from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:39 mail postfix/smtpd[14115]: connect from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:39 mail postfix/smtpd[14115]: setting up TLS connection from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:39 mail postfix/smtpd[14116]: connect from mail-bk0-f49.google.com[209.85.214.49]
Jul  5 12:43:39 mail postfix/smtpd[14117]: connect from mail-bk0-f49.google.com[209.85.214.49]
Jul  5 12:43:39 mail postfix/smtpd[14116]: setting up TLS connection from mail-bk0-f49.google.com[209.85.214.49]
Jul  5 12:43:39 mail postfix/smtpd[14117]: setting up TLS connection from mail-bk0-f49.google.com[209.85.214.49]
Jul  5 12:43:39 mail postfix/smtpd[13602]: TLS connection established from mail-bk0-f50.google.com[209.85.214.50]: TLSv1 with cipher RC4-SHA (128/128 bits)
Jul  5 12:43:39 mail postfix/smtpd[14113]: TLS connection established from mail-bk0-f50.google.com[209.85.214.50]: TLSv1 with cipher RC4-SHA (128/128 bits)
Jul  5 12:43:39 mail postfix/smtpd[14115]: TLS connection established from mail-bk0-f50.google.com[209.85.214.50]: TLSv1 with cipher RC4-SHA (128/128 bits)
Jul  5 12:43:39 mail postfix/smtpd[14116]: TLS connection established from mail-bk0-f49.google.com[209.85.214.49]: TLSv1 with cipher RC4-SHA (128/128 bits)
Jul  5 12:43:39 mail postfix/smtpd[14117]: TLS connection established from mail-bk0-f49.google.com[209.85.214.49]: TLSv1 with cipher RC4-SHA (128/128 bits)
Jul  5 12:43:40 mail postfix/smtpd[13602]: 2846B11AC5E2: client=mail-bk0-f50.google.com[209.85.214.50], sasl_method=PLAIN, sasl_username=someuser@somedomain.com
Jul  5 12:43:40 mail postfix/smtpd[14113]: 3290811AC5E3: client=mail-bk0-f50.google.com[209.85.214.50], sasl_method=PLAIN, sasl_username=someuser@somedomain.com
Jul  5 12:43:40 mail postfix/smtpd[14115]: 3C4AD11AC5E4: client=mail-bk0-f50.google.com[209.85.214.50], sasl_method=PLAIN, sasl_username=someuser@somedomain.com
Jul  5 12:43:40 mail postfix/cleanup[13420]: 2846B11AC5E2: message-id=
Jul  5 12:43:40 mail postfix/cleanup[14092]: 3290811AC5E3: message-id=
Jul  5 12:43:40 mail postfix/smtpd[14116]: warning: SASL authentication failure: Password verification failed
Jul  5 12:43:40 mail postfix/smtpd[14116]: warning: mail-bk0-f49.google.com[209.85.214.49]: SASL PLAIN authentication failed: authentication failure
Jul  5 12:43:40 mail postfix/cleanup[14121]: 3C4AD11AC5E4: message-id=
Jul  5 12:43:40 mail postfix/qmgr[32242]: 2846B11AC5E2: from=, size=10564, nrcpt=1 (queue active)
Jul  5 12:43:40 mail postfix/qmgr[32242]: 3290811AC5E3: from=, size=10566, nrcpt=1 (queue active)
Jul  5 12:43:40 mail postfix/smtpd[14116]: disconnect from mail-bk0-f49.google.com[209.85.214.49]
Jul  5 12:43:40 mail postfix/qmgr[32242]: 3C4AD11AC5E4: from=, size=10568, nrcpt=1 (queue active)
Jul  5 12:43:40 mail postfix/smtpd[13602]: disconnect from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:40 mail postfix/smtpd[14113]: disconnect from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:40 mail postfix/smtpd[14115]: disconnect from mail-bk0-f50.google.com[209.85.214.50]
Jul  5 12:43:40 mail postfix/smtpd[14117]: D4F2411AC5E5: client=mail-bk0-f49.google.com[209.85.214.49], sasl_method=PLAIN, sasl_username=someuser@somedomain.com
Jul  5 12:43:41 mail postfix/cleanup[13420]: D4F2411AC5E5: message-id=
Jul  5 12:43:41 mail postfix/qmgr[32242]: D4F2411AC5E5: from=, size=10565, nrcpt=1 (queue active)
Jul  5 12:43:41 mail postfix/smtpd[14117]: disconnect from mail-bk0-f49.google.com[209.85.214.49]

In looking into the SASL component a bit, I noticed that there were 5 simultaneous SASL servers running. The first one looks like a parent with 4 child processes.

[root@mail postfix]# ps -ef |grep sasl
root      9253     1  0 Mar15 ?        00:00:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a rimap -r -O 127.0.0.1
root      9262  9253  0 Mar15 ?        00:00:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a rimap -r -O 127.0.0.1
root      9263  9253  0 Mar15 ?        00:00:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a rimap -r -O 127.0.0.1
root      9264  9253  0 Mar15 ?        00:00:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a rimap -r -O 127.0.0.1
root      9265  9253  0 Mar15 ?        00:00:04 /usr/sbin/saslauthd -m /var/run/saslauthd -a rimap -r -O 127.0.0.1

So it seemed likely that the 4 child processes were in use and that Postfix couldn’t open a connection to a 5th simultaneous SASL authentication server, so it responded with a generic SMTP AUTH failure.

To fix, I simply added a couple of extra arguments to the saslauthd command that is run. I added a ‘-c’ parameter to enable caching, and ‘-n 10′ to increase the number of servers to 10. On my CentOS server, I accomplished that by modifying /etc/sysconfig/saslauthd to look like this:

# Directory in which to place saslauthd's listening socket, pid file, and so
# on.  This directory must already exist.
SOCKETDIR=/var/run/saslauthd

# Mechanism to use when checking passwords.  Run "saslauthd -v" to get a list
# of which mechanism your installation was compiled with the ablity to use.
MECH=rimap

# Additional flags to pass to saslauthd on the command line.  See saslauthd(8)
# for the list of accepted flags.
FLAGS="-r -O 127.0.0.1 -c -n 10"

After restarting saslauthd, and a quick test, it looks good so far.

Quick Fix for Dovecot’s “save failed to INBOX: Timeout while waiting for lock”

Posted on July 13th, 2011 in Mail by Brandon

I had a mail server lock up and die on me tonight. The data center replaced some hardware in order to get it to power back up. Everything started up fine, but I was unable to retrieve my own email. My mail client was complaining about timeouts on a single account. In looking at the log, I found some lines similar to these in the dovecot mail logs:

Jul 13 04:19:09 secure deliver(user@domain.com): msgid=<20110713105301.E4C2C3293DC@myserver.com>: save failed to INBOX: Timeout while waiting for lock
Jul 13 04:19:09 secure deliver(user@domain.com): msgid=<20110713105702.817483293E5@myserver.com>: save failed to INBOX: Timeout while waiting for lock
Jul 13 04:19:09 secure deliver(user@domain.com): msgid=<20110713105802.3F68E3293E7@myserver.com>: save failed to INBOX: Timeout while waiting for lock
Jul 13 04:19:09 secure deliver(user@domain.com): msgid=<20110713105402.B5DBA3293DE@myserver.com>: save failed to INBOX: Timeout while waiting for lock
Jul 13 04:19:09 secure deliver(user@domain.com): msgid=<20110713102202.6045E3293A6@myserver.com>: save failed to INBOX: Timeout while waiting for lock

Googling the error message didn’t find any quick results, so I attempted a more brute-force method. Inside the user’s maildir, I just moved some dovecot files out of the way:

[root@host /vmail/domain/user/]# mv dovecot-uidlist.lock dovecot.index dovecot.index.cache dovecot.index.log /tmp/badfiles

After moving those files out of the way, I restarted dovecot with

/etc/init.d/dovecot restart

and my client was able to connect without problems.

SpamAssassin + Postfix to Deliver Spam to an Alternate Mailbox

Posted on April 10th, 2009 in General,Linux System Administration,Mail,Spam by Brandon

A fairly simple method of having postfix send messages through SpamAssassin, and then delivering the messages identified as spam to a separate mailbox:

Configure SpamAssassin as normal in master.cf:

smtp      inet  n       -       -       -       -       smtpd -o content_filter=spamassassin
2525      inet  n       -       -       -       -       smtpd -o content_filter=spamassassin
spamassassin unix -     n       n       -       -       pipe user=spamd argv=/usr/bin/spamc -f -e    /usr/sbin/sendmail -oi -f ${sender} ${recipient}

Configure Spamassassin to log the status and score in a new header by adding this to /etc/spamassassin/local.cf:

## Add a spamassassin header to all messages so that we can filter them out
add_header all Status _YESNO_, score=_SCORE_ required=_REQD_ tests=_TESTS_ autolearn=_AUTOLEARN_ version=_VERSION_

And then configure postfix to look for that header and deliver it to an alternate mailbox:

/etc/postfix/main.cf:

header_checks = regexp:/etc/postfix/header_checks

/etc/postfix/header_checks:

/^X-Spam-Status: Yes/ REDIRECT spam@yourdomain.com

Now your messages identified as spam will be sent to spam@yourdomain.com so that you can occasionally check for false positives.

Note that there is more work that goes into setting up SpamAssassin, but there are plenty of howtos around for getting that configured.

Vacation (AutoReply) Message with Virtual Users and Postfix

Posted on August 8th, 2008 in General,Linux System Administration,Mail by Brandon

I’ve previously written about both Virtual Mail users, and about enabling vacation messages for postfix. The next step was to get vacation working with virtual users.

My first thought was to try and make the sendmail ‘vacation’ program work with virtual users, but after digging into that a bit, it looked like more trouble than it was worth. I remembered that PostfixAdmin had some kind of support for this, so decided to check it out, which proved a much more promising solution.

PostfixAdmin ships with a perl-based script that can be piped an email message and then will send an auto-reply to the sender. The script is able to grab a customized subject and message body from a MySQL database and then reply to senders as appropriate. It also keeps track of who it has auto-replied to so that each sender only gets one auto-reply in a given length of time.

The instructions for implementing it can be found at http://postfixadmin.svn.sourceforge.net/viewvc/postfixadmin/trunk/VIRTUAL_VACATION/INSTALL.TXT?view=markup. I found the documentation to be fairly straightforward.

Essentially, when a user enables the auto-reply, it adds an email address to the aliases table that points to user#domain@autoreply.yourdomain.com. You then configure postfix to send everything to the ‘autoreply.yourdomain.com’ domain to the vacation script, which then can read the original recipient’s address and respond as desired.

I now have this working on our hosted mail solution, so that RoundSphere mail customers now have auto-reply functionality. In addition, I made an addition to the webmail application (RoundCube) so that users can modify their vacation message themselves instead of having to have a mail administrator do it through postfixadmin.

A Case for Choosing Good Server Names

Posted on July 3rd, 2008 in General,Linux System Administration,Mail,MySQL by Brandon

This morning, I had a client call me bright and early, frantic about some mail problems they were having.  All of their mail servers had stopped accepting incoming SMTP connections for some reason, and they couldn’t figure out why.

After a little bit of investigation, I found that they were using postfix with MySQL-based virtual domains.   The MySQL authentication was failing, which meant that postfix was unable to look up any valid recipient names.   That, in turn was causing tons of retried connections, until they hit the maximum number of connections where Postfix would refuse additional connections.

The problem is that these mail servers were initially set up with some dumb names for some reason.    A new administrator noticed the silly names in their Reverse DNS entries and changed them to some more sensible names.  The MySQL permissions were based off of the hostnames, so when the names in Reverse DNS changed, it broke the permissions, and the clients were unable to connect.

Solving the problem was simple enough – I just corrected the MySQL permissions, and then had to deal with some huge mail queues for a little while as all of the messages waiting to come in were finally allowed all at once.

The moral of the story is to use sensible names to start out with.   These names were chosen to be sortof funny I guess, but it didn’t end up being so amusing in the midst of all of the problems it caused.  As a side note, I usually do MySQL permissions based on IP Address as well, so that you further reduce this kind of problem.

Postfix Vacation Message

Posted on June 26th, 2008 in General,Linux System Administration,Mail by Brandon

The idea of a vacation message is kindof odd to me, but I had one client request it today, so took a look at configuring it. On RHEL/CentOS distros, the ‘vacation’ binary is distributed with sendmail, and is not available with postfix, so you have to build it yourself. Fortunately, it is about the easiest thing I have ever compiled.

[root@host ~]# yum install gdbm-devel
[root@host ~]# cd /usr/local/src/
[root@host ~]# wget http://internap.dl.sourceforge.net/sourceforge/vacation/vacation-1.2.7.0-rc2.tar.gz
[root@host ~]# tar -xvzf vacation*
[root@host ~]# cd vacation-1.2.7.0-rc2
[root@host ~]# make
[root@host ~]# make install

That’s it. Not even a configure script. That should install the vacation binary in /usr/bin/vacation.

Now just create a vacation message by putting a ‘.vacation.msg’ in the user’s home directory with the auto-reply content:

Subject: On vacation message.

I'm on vacation and will not be reading my mail for a while.
Your mail will be dealt with when I return.

And finally, create a .forward file that tells your mail program to deliver to the vacation program:

\myuser, "|/usr/bin/vacation  myuser"

That should be it. I tested and verified that it works. Note that you have to provide a to: header with the recipient’s address.

Next, I might try some experiments to see if I can get it to work with virtual users.

Getting Dkimproxy Installed and Configured

Posted on June 21st, 2008 in Encryption,General,Linux System Administration,Mail,Spam by Brandon

Dkimproxy is a great program for getting Postfix to both sign and validate DomainKeys and DKIM messages. Prior to dkimproxy, one would have used dk-filter and dkim-filter which did DomainKeys and DKIM signing separately. dkimproxy is a newer version that combines the functionality into one program. Installing it can be a bit complicated because it isn’t available in most distro repositories, and requires several Perl modules that need to be installed. Configuring it can be difficult as well, because it involves making changes DNS and postfix, in addition to its own configuration.

I wrote these steps below as I went through a recent installation for a customer

You can install the required Perl modules through the RPM Forge Repository if you have it enabled with the command (Thanks JohnB for mentioning that):

yum install perl-Net-Server perl-Error perl-Mail-DKIM

Otherwise, you can install them manually with CPAN. First install the openssl-devel package (You’ll need it for CPAN to install Mail::DKIM)

yum install openssl-devel

Now install the required Perl modules

# perl -MCPAN -e shell
> install Net::Server
> install Error
> install Mail::DKIM

Download and install the actual dkimproxy code:

cd /usr/local/src
wget http://internap.dl.sourceforge.net/sourceforge/dkimproxy/dkimproxy-1.0.1.tar.gz
tar -xvzf dkimproxy-1.0.1.tar.gz
cd dkimproxy-1.0.1
./configure --prefix=/usr/local/dkimproxy
make
make install

You should now have the program installed in /usr/local/dkimproxy. A sample init file is included, so we can copy it into place also:

cp /usr/local/src/dkimproxy-1.0.1/sample-dkim-init-script.sh /etc/init.d/dkimproxy

Create a ‘dkim’ user and group, but lock the password:

useradd -d /usr/local/dkimproxy dkim
passwd -l dkim

That should be enough to get dkimproxy running, but it isn’t configured yet.

Create a key file for your domain

cd /usr/local/dkimproxy/etc/
openssl genrsa -out domain.tld.key 1024
openssl rsa -in domain.tld.key -pubout -out domain.tld.pub

Now create a DNS TXT record for mail._domainkey.domain.tld with the contents of domain.tld.pub. Your public key will span at least two lines, so combine all of the lines of the key together when putting it in your DNS record. The whole DNS record will look something like this:

k=rsa; t=s; p=MFwwDQYJ......0JMCAwEAAQ==

(Note that the key is pretty long and I’ve shortened it here)
You could now confirm the key is correct in your DNS:

[root@host etc]# host -ttxt mail._domainkey.domain.tls
mail._domainkey.domain.tld descriptive text "k=rsa\; t=s\; p=MFwwDQYJ......0JMCAwEAAQ=="

(Note that the key is pretty long and I’ve shortened it here)

Now tell dkimproxy about the key files, and configuration parameters. Create /usr/local/dkimproxy/etc/dkimproxy_out.conf with this content

# specify what address/port DKIMproxy should listen on
listen    127.0.0.1:10027

# specify what address/port DKIMproxy forwards mail to
relay     127.0.0.1:10028

# specify what domains DKIMproxy can sign for (comma-separated, no spaces)
domain    domain.tld

# specify what signatures to add
signature dkim(c=relaxed)
signature domainkeys(c=nofws)

# specify location of the private key
keyfile   /usr/local/dkimproxy/etc/domain.tld.key

# specify the selector (i.e. the name of the key record put in DNS)
selector  mail

And copy the sample inbound config to the real inbound config

cd /usr/local/dkimproxy/etc
cp dkimproxy_in.conf.example dkimproxy_in.conf

Now you should be able to start up dkimproxy, and configure it to start at boot:

/etc/init.d/dkimproxy start
chkconfig dkimproxy on

And the last step is just to modify the postfix configuration to tell it to forward messages sent to port 587 through dkimproxy for signing. I added these three sections to /etc/postfix/master.cf

### dkimproxy filter - see http://dkimproxy.sourceforge.net/postfix-outbound-howto.html
#
# modify the default submission service to specify a content filter
# and restrict it to local clients and SASL authenticated clients only
#
submission  inet  n     -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_sasl_auth_enable=yes
    -o content_filter=dksign:[127.0.0.1]:10027
    -o receive_override_options=no_address_mappings
    -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject

# specify the location of the DKIM signing proxy
# Note: the smtp_discard_ehlo_keywords option requires a recent version of
# Postfix. Leave it off if your version does not support it.
dksign    unix  -       -       n       -       10      smtp
    -o smtp_send_xforward_command=yes
    -o smtp_discard_ehlo_keywords=8bitmime,starttls

# service for accepting messages FROM the DKIM signing proxy
127.0.0.1:10028 inet  n  -      n       -       10      smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

If you want it to sign messages sent from the command line sendmail program, modify the pickup service to use the content_filter like this:

pickup    fifo  n       -       n       60      1       pickup
    -o content_filter=dksign:[127.0.0.1]:10027

Finally restart postfix with ‘postfix reload’, and you *should* have a working installation. You can now use my Domainkeys/Dkim validator to test and ensure that it is working.

Google Spam Filtering Sounds Great but I Can’t Sign Up

Posted on February 6th, 2008 in General,Linux System Administration,Mail,Spam by Brandon

Google announced yesterday new services and pricing based on their Posting message filtering service.  The service sounds great, and I’ve been looking at moving away from my current mail filtering service for a couple months now.  Pricing starts out at only $3.00 per user per year.  I did a little checking around, and verified that I can add domain aliases and user aliases and that it looks like they can be tied to a single $3.00 account.

That sounds exactly like what I need.  I have a bunch of domains, and use several email addresses at each one that all forward to a single Inbox.  For $3.00 a year, it sounds like a great savings over my alternate plan which was creating my own MailScanner box.  Plus with Google, I won’t have to worry about redundancy, or keeping my own filtering up to date.

Perfect, so I went to sign up.  I put in my domain name, agreed to the TOS, then put in my credit card information and hit submit:

Google Won’t let me sign up for Posting

Oops, looks like something went wrong there.   That’s not the best way to instill confidence into your new customers.

Converting mbox’s to maildir format

Posted on February 1st, 2008 in General,Linux System Administration,Mail by Brandon

There is a handy utility for converting mbox style mailboxes into maildir format at http://batleth.sapienti-sat.org/projects/mb2md/

To convert all of the mailboxes on your server:

Edit /etc/sudoers and comment out the env_keep section. These variables make it so that the sudo command keeps some environment variables and tries to put things in the wrong directory.

Download mb2db, unzip it, and copy the binary to /bin (where all users can access it)

# wget http://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz
# gunzip mb2db-3.20.pl.gz
# cp mb2db-3.20.pl.gz / bin

Then run this command to convert all of the mailboxes into maildir format.

cd /var/spool/mail

for username in `ls`; do echo $username; sudo -u $username /bin/mb2md -m -d Maildir; done

That will create a directory called Maildir in each user’s home directory. Then just configure your MTA to deliver mail there, and your IMAP server to pick it up there
In postfix, add this to /etc/postfix/main.cf

home_mailbox = mail/

And in Dovecot, change this in /etc/dovecot.conf

mail_location=maildir:~/mail/

Now you can edit /etc/sudoers and uncomment the env_keep section.

CentOS 5 Virtual Mail Toaster Howto

Posted on November 29th, 2007 in General,Linux System Administration,Mail by Brandon

I have recently configured several CentOS virtual mail servers.  It took me quite a while to figure it out the first time or two, but has gotten significantly easier since then. Initially, I pieced information together from a half-dozen or so various other howto’s that were either designed for a different distro, or were outdated (or both).

So when I put together another server last night, I made careful notes when installing it and generated a howto document.   It walks a user all the way from a clean CentOS 5 install, through to a functioning virtual mail server.  It uses postfixadmin as a web interface for managing the domains and accounts.  All domain and user information is stored in a MySQL database.   Postfix is installed for the MTA, and Dovecot for the POP3/IMAP server.    It doesn’t require system accounts for any of the users.  All mail services are accessible over encrypted SSL/TLS protocols.

Next Page »