Month: July 2007

The IPTrack application that I wrote saves a bunch of Netflow data generated by routers into a MySQL database for analysis and summarization. During peak usage times, it inserts about 50k rows per minute into a table. To keep the table at a manageable size, it then summarizes the useful data into a lower-volume table and deletes any Netflow data older than 10 minutes out of the table. This particular table usually has between 300k and 500k rows in it at any given time.

The MEMORY storage engine operates completely from Ram. Of course this has the advantage of being very quick compared to File based access. The downside is that any data in it is lost if the MySQL instance has to be restarted for any reason. Since my application just uses it for temporary data anyway, and it doesn’t matter all that much if I lose up to 10 minutes of data, this seemed perfectly acceptable.

So, I tried converting the table from MyISAM to MEMORY. For the first couple minutes it looked pretty promising. Disk IO and the machine’s load was pretty small and it was cruising along. But, after ten minutes, when it got to the point where it was purging old data from the table, it came to a grinding halt. The delete statement, which took only a few seconds using the MyISAM engine just kept going and going. It ended up taking about 9 MINUTES to complete. That, of course, is pretty unacceptable. The MEMORY storage engine was supposed to be faster than MyISAM, so this made no sense. The table was already pretty well indexed. I was deleting based on the timestamp column:

DELETE FROM flows WHERE timestamp < ?

and the timestamp column had an index. I had no idea what could make it take so long. I ran across a bug report of somebody else having the problem. The reply was that this was just the nature of hash indexes. The bug report was from 2004 though, so I posted my problem to the mysql forums in search of an answer.

I just noticed that somebody replied to my post and said to use btree indexes instead of hash indexes (the default for the MEMORY engine). Sure enough, I recreated my indexes using btree and it works perfectly now.

CREATE INDEX timestamp_btree USING BTREE ON flows (timestamp);

Thanks to KimSeong Loh on the MySQL forums for solving this for me

awstats is my favorite web statistics program. It provides quite a bit of interesting data about my websites, and I usually set it up for any site that I work on. One issue that I’ve had with it though, is that in a shared setup, by default, it uses the HTTP HOST header to check which domain to display statistics for. The config files and data files are all saved with the full hostname of the domain for the stats it contains. So it treats “www.brandonchecketts.com” different from “brandonchecketts.com”.

This has always been just a slight annoyance to me, and I have just remembered to make sure to put the ‘www’ on the URL when I’m looking at my statistics. Today, though, I thought I would dig into it a little bit to fix it permanently.

Turns out it was pretty easy to change this behavior so that it always just removes the ‘www’ from the domain name for any of the files it looks for. Simply add this on line 1160 of awstats.pl:

$SiteConfig =~ s/^www\\.//g;

It should go right after the $FileConfig= line, and right before the foreach loop. Since all of my database files were created with the ‘www’ in them, I had to go through and rename all of those database files to remove the www from them. Now I can hit either URL and get the same data.

I started a forum thread on the SourceForge forums to announce it to others and see if anybody else finds it useful.

I’ve recently setting up a bunch of hosts with a new Amanda backup server. I like to see, though, how much space each server that I’m backing up uses. Amanda stores a bunch of info in the ‘curinfo’ directory for each host and disk that is being backed up, but I haven’t found any good tools for querying or displaying that. So, I wrote my own. This script looks through all of the files in the ‘curinfo’ directory, and prints out a summary of how much space each disk/host is taking up:

#!/usr/bin/perl

## View Amanda disk space usage per host/disk
## Author:  Brandon Checketts
## Website: http://avazio.com/

$curinfo_dir = "/etc/amanda/avazio/curinfo";

opendir(DH, $curinfo_dir);
while($host = readdir(DH)) {
  next if($host =~ m/^./);
  if( -d "$curinfo_dir/$host") {
    opendir(DH2, "$curinfo_dir/$host");

    while($disk = readdir(DH2)) {
      next if($disk =~ m/^./);
      if( -f "$curinfo_dir/$host/$disk/info") {
        open(FH, "< $curinfo_dir/$host/$disk/info");
        while(my $line = ) {
          if($line =~ m/^history: ([0-9]+) ([0-9]+) ([0-9]+) ([0-9]+) ([0-9]+)/) {
            ## Example line: history: 1 2319760 2319760 1184766354 1121
            ## Line format:  history [lvl] [rawsize] [compsize] [timestamp] [unk?]
            $space->{$host}->{$disk}->{'rawsize'}  += ($2 / 1024);
            $space->{$host}->{$disk}->{'compsize'} += ($3 / 1024);

          }
        }
      }
    }
    closedir(DH2)
  }
}
closedir(DH);

$grandtotal_rawsize  = 0;
$grandtotal_compsize = 0;
foreach my $host (keys(%{$space})) {
  print "n$hostn";
  $thishost = $space->{$host};
  $thishost_rawsize  = 0;
  $thishost_compsize = 0;
  foreach my $disk (keys(%{$thishost})) {
    $thisdisk = $space->{$host}->{$disk};
    $thishost_rawsize    += $thisdisk->{'rawsize'};
    $thishost_compsize   += $thisdisk->{'compsize'};
    $grandtotal_rawsize  += $thisdisk->{'rawsize'};
    $grandtotal_compsize += $thisdisk->{'compsize'};
    $disk =~ s/_///g;
    printf("  %-40s %-6i Mb   %-6i Mb\\n", $disk, $thisdisk->{'rawsize'}, $thisdisk->{'compsize'});
  }
  printf("  TOTAL:                                   %-6i Mb  %-6i Mb\\n",  $thishost_rawsize, $thishost_compsize);
}

printf("GRAND TOTAL:                                 %-6i MB  %-6i MB\\n", $grandtotal_rawsize, $grandtotal_compsize);

The company I work at is having to change who their Dialup modem pool goes through. The new company blocks outgoing SMTP by default to prevent spamming. To enable port 25 for our mail servers, they we have to send them some attributes in the Radius ACCEPT packet during authentication. I haven’t really gotten into the Radius server that we use, because it has always worked and we’ve never had to change anything on it until now. So I began digging into FreeRadius, and found out that it is pretty useful.

Our server was configured to authenticate against a MySQL database that contains all of the configuration. I believe it is a pretty standard configuration. In the radius configuration, it defines certain queries to run for authentication, and also for attributes to send in the accept packet. Specifically, it’s was configured to do this query for the attributes:

authorize_reply_query = "
  SELECT id,UserName,Attribute,Value,op 
  FROM ${authreply_table} 
  WHERE Username = '%{SQL-User-Name}'
  ORDER BY id"

So, I can just put additional rows into the ${authreply_table} to add the attributes that this new provider requires. With the default setup, though, I would have to add the attributes for each user. That would get to be a mess, because I would have to modify the billing system which populates the radius database. Instead I modified the SQL query to include rows where the Username = ALLUSERS, like this:

authorize_reply_query = "
    SELECT id,UserName,Attribute,Value,op
    FROM ${authreply_table}
    WHERE (
        Username = '%{SQL-User-Name}' OR
        Username = 'ALLUSERS'
    ) 
   ORDER BY id"

Now, I can just insert rows for those attributes once, and no further modification is necessary. Pretty handy.

We’re installing the Amanda backup client on a bunch of servers that I administer. We’ve decided to tunnel Amanda connections through SSH though, and that feature isn’t available in the current RPM builds that are available.

We started out compiling it from scratch on each box, and then following a bunch of steps to add the appropriate users, create directories, setup SSH keys, etc. Obviously, that gets to be a tedious process pretty quick and is prone to errors and missing steps. The obvious solution is to create an RPM for this, since that is what they are for.

Creating an RPM is pretty easy. I found some useful instructions here and here. Basically, the buildrpm package goes through the standard configure; make; make install steps for you. You can create simple shell scripts to do whatever you want at each of those processes when building the RPM. Additionaly, you can also run commands before and after RPM installation, and before and after RPM removal. For this particular script, I added a bunch of stuff to the postinstall script. As an example, here is my .spec file:

Summary: Amanda Client with SSH
Name: amanda_client_ssh
Version: 2.5.2p1
Release: 2
License: GPL
Group: Amanda Backup
Source: amanda_client_ssh-2.5.2p1.tgz

BuildRoot: %{_builddir}/%{name}-root

%description
Amanda Client compiled with SSH authentication

%prep

%setup
./configure '--a-bunch' '--of options' --'can go here'

%build
make

%install
rm -rf $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install
if [[ ! -d $RPM_BUILD_ROOT/etc/amanda ]]; then
    mkdir $RPM_BUILD_ROOT/etc/amanda
fi
cat >> $RPM_BUILD_ROOT/etc/amanda/amanda-client.conf < EOF
conf "MyConfig"
index_server "backup.mydomain.com"
tape_server "backup.mydomain.com"
auth "ssh"
ssh_keys "/root/.ssh/id_rsa_amrecover"
EOF

%pre

%post
useradd -M -n -g disk -o -r -d /var/lib/amanda -s /bin/bash
        -c "Amanda user" -u 33 amanda >/dev/null 2>&1 || :

if [[ ! -d /var/lib/amanda/.ssh ]]; then
    mkdir -p /var/lib/amanda/.ssh
fi
if [[ ! -d /var/log/amanda ]]; then
    mkdir -p /var/log/amanda
fi

touch /etc/amandates

cat >> /etc/amanda/exclude <
/var/spool/mail
/tmp
EOF
mkdir ~amanda/gnutar-lists
chown amanda:disk /etc/amandates /etc/amanda/exclude ~amanda/gnutar-lists /var/log/amanda/

if [[ ! -f /root/.ssh/id_rsa_amrecover ]]; then
    ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa_amrecover
fi

cat >> /var/lib/amanda/.ssh/authorized_keys <EOF
ssh-rsa abcdefgABigLongPublicKeyGoesHere== amanda@backup.webpipe.net
EOF

%preun
userdel amanda

%clean
rm -rf $RPM_BUILD_ROOT

%files
... a bunch of files listed here ...

%changelog

It took a couple revisions of trial and error to get everything correct. That part took the longest because I didn’t see an option to skip the configure and make steps, so anytime I changed the postinstall commands it had to rebuild the whole app. But now that I’ve got the RPM, installing it should be pretty straightforward on the rest of the boxes it needs to be installed on.

I had some problems with a mail server today and had to reboot it. After booting it back up all the services started, but any attempts to access a mailbox via IMAP generated an error that said “Unable to open this mailbox”. Testing it through telnet looked like this:

[root@ny ~]# telnet localhost 143
>> Trying 127.0.0.1
>> Connected to mail.somedomain.com (127.0.0.1).
>> Escape character is '^]'.
>> * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
>> THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright
>> 1998-2004 Double Precision, Inc.  See COPYING for distribution information.
<< 01 LOGIN user@domain.com mypassword
>> 01 OK LOGIN Ok.
<< 02 LIST "" *
>> * LIST (HasNoChildren) "." "INBOX.Junk"
>> * LIST (HasNoChildren) "." "INBOX.Drafts"
>> * LIST (HasNoChildren) "." "INBOX.Trash"
>> * LIST (HasNoChildren) "." "INBOX.Sent"
>> * LIST (Marked HasChildren) "." "INBOX"
>> 02 OK LIST completed
<< 03 SELECT INBOX
>> 03 NO Unable to open this mailbox.

After digging into file permissions, restarting courier-imap, and anything else I could think of, I was getting puzzled. Then a fellow sysadmin noticed that time time on the box was way off:

mail:~# date
Wed Jan  7 04:09:17 MST 1925

I corrected that and it imap suddenly started working again. I wish that courier-imap would have given a little more descriptive error message, but glad that it was nothing difficult to fix.