Multi-threaded perl

I’ve been experimenting on multi-threading in perl for a new project, and am impressed with how straightforward it is. Before digging into it, I never really considered doing anything with it because it was always kindof ‘mysterious’ to me. Now, I’m seeing how useful it is to have multiple threads that are able to share variables.

In the application I’m rewriting, I used to have one script that listened for network data, then saved that out to a file. I had another script that read through the output files, and then inserted the data into a database. Now, with a multi-threaded program, I just have one thread that listens, and another thread (or multiple threads) that parse the data and manipulate it however I want. In this case, that saves a lot of disk activity, and makes the program a lot more efficient, and straight-forward.
I’m also able to use the Thread::Queue module to create a queue that the listener process can add to, and then have ‘worker’ threads that can go through the data and format/summarize/whatever I’m going to do with it.

I’m looking forward to seeing how this all works out.  I’m impressed so far.

Firefox Load Time Analyzer is cool, but …

A while ago, a friend of mine showed me the Load Time Analyzer for Firefox.  It’s a pretty cool tool that can analyze all of the different HTTP requests that make up a page load.  Then it can display a simple graph showing how long all of the requests took.  It could be very useful for trying to identify requests that are slow to load.

I recently had a reason to use the plugin, so I turned it on and found what I needed to.  I removed the toolbar for it when I was done with it.  But the past couple days, my browser has gotten painfully slow after being open for a couple hours.   It was using a ton of memory, and just loading a simple page, or opening a new tab would sometimes take ten seconds.

Turns out that the Load Time Analyzer was still running, even though the toolbar wasn’t being displayed.  It was keeping track of everything page I visited, which is why it was using so much memory and running so slowly.   To actually disable it, you have to go into your ‘Addons’ menu, and actually disable it.   That, of course, requires a browser restart, which is kindof annoying.

It would be nice if future versions had an option to disable it without a browser restart.  For now, make sure to enable it only while you need to use it.

DHCP ‘always-broadcast’ confusion

I run a DHCP server using Linux’s dhcpd program to serve addresses to a bunch of clients.  These clients are connected over several wireless links, and the radios are sometimes quirky.  Specifically, some clients never get the DHCPOFFER unless the ‘always-broadcast’ parameter is on.  This usually works out fine.

Today, however, we had a problem where we just saw a bunch of incoming DHCPDISCOVER messages, to which the server would reply with a DHCPOFFER. The devices would just continually send discover messages, and none would ever DHCPREQUEST an address.

From what I can conclude, I think that the clients were confused when they received multiple broadcast responses back for their DHCPDISCOVER message.  The client would then send another discover message, which just caused a never-ending loop of requests and offers.

To resolve the problem, I turned off always-broadcast for a few minutes.  This made the clients wait for a random period of time before discovering again.  Some clients accepted the IP fine even though it wasn’t broadcast.  For the ones that didn’t, I then re-enabled always-broadcast, and they picked up an address the next time that they tried.

For a long term solution, we’re working on subnetting the two /24 networks that are currently together into smaller /26 or /27 blocks.  That should reduce the possibility of having this happen again.

How to choose which NIC gets ‘eth0’

When a Linux server has multiple network interfaces, it may be be necessary to choose which NIC gets assigned which name. In particular, we recently had to swap out a NIC that couldn’t handle 100 Meg/Full duplex. After swapping the NIC, if the OS was allowed to choose the interface names by itself, it had them backwards from what they previously were. We have a bunch of networking scripts that had the interface names hard coded, so we didn’t want to change all of those.

After some googling, I came across this page that described how to do it. This allows you to choose which NIC gets which name, based on the MAC address. Simply create a file in /etc/udev/rules.d that contains something like this:

KERNEL=="eth?", SYSFS{address}=="aa:bb:cc:dd:ee:ff", NAME="eth0"
KERNEL=="eth?", SYSFS{address}=="00:11:22:33:44:55", NAME="eth1"

Syslog server

With all of the network configuration I’ve been doing lately, I’ve decided to set up a central syslog server that the routers and switches can log to. That will create a central place that I can look for warnings and errors that are occurring on these devices. It also makes it so I can save the logs for extended periods of time, and use normal Linux tools to search and parse through them.

For some reason, I found it difficult to find instructions on how to create a centralized syslog server. However, its incredibly easy. You just have to configure the Linux syslogd process to listen on a remote interface, then configure the logs like anything else in syslog. In CentOS, you just edit /etc/sysconfig/syslog and add a “-r” to the “SYSLOGD_OPTIONS” line. Then allow port 514 through your firewall from your router’s IP addresses.

On each router, set these global configuration commands:

logging facility local1
logging source-interface FastEthernet0/1

On the syslog server, configure your /etc/syslog.conf with something like this:

## Router Logs
local1.* /var/log/routers/router-core.log
local2.* /var/log/routers/router-border.log
local3.* /var/log/routers/switch-1.log
local4.* /var/log/routers/switch-2.log

I also like to set up logrotate with this in /etc/logrotate.d/routers

/var/logs/routers/*.log {
rotate 52

My next step will be creating a LogWatch script to email me anything that I should be concerned about

rdate synchronization problems

A bunch of the servers I manage have started having a problem with their daily time sync. Whenever rdate fails, it prints an error message to STDOUT. Since all of the servers are running the sync via cron.daily, this generates a bunch of emails to me, and doesn’t attempt to sync the time for another day.

As a result, I’ve written a simple wrapper for rdate that retries it several times and only prints an error if it was unsuccessful after all of those.

Here is the script. This was written for a CentOS server, so the program paths may need to be changed if you are running it on another distro


## Output nothing if sync eventually works
## Output an error if unsuccessful after $MAX_RETRIES


while [[ $RC != 0 && $RETRIES -lt $MAX_RETRIES ]]
/usr/bin/rdate -s $TIMESERVER
RETRIES=`/usr/bin/expr $RETRIES + 1`
done > /dev/null 2>&1

if [[ $RC != 0 ]]; then
echo "Error syncing time to $TIMESERVER after $MAX_RETRIES tries"
echo "Last RC was $RC"

Nice change: A novel with real technology

On a recent plane trip I was able to read ‘The Cockoo’s Egg‘ by Clifford Stoll. It was a nice to have real technology described for a change. The book is a real-life account of how a Unix System Administrator tracked down a hacker in the early 80’s. Obviously, its quite dated, but the story was pretty interesting, and it was interesting to learn about the vulnerabilities in some of the applications.

The book makes a good balance between the technical stuff and telling the story. It technical enough that I enjoyed it, but still simple enough that my wife can follow it.

The coolest, most efficient way to copy a directory between servers

I was recently quizzed about the quickest, most efficient way to copy an entire directory between servers. I typically do this by tar’ing it up on one server, copying it to the other, then extracting it. However, this has a couple obvious problems. One, is that it requires large chunks of disk space to hold the archive file on both the source and destination. If you are low on disk space, this can be a real pain. The other bad thing, is that it a waist of time since it reads through all of the files three times (read, copy, extract).

The original thought I had was to use “scp -r” which will recursively copy a directory over to the destination. This, however, doesn’t copy directories that start with a dot, and it doesn’t preserve file ownership information.

The best way, is to use a combination of tar and ssh. The idea is to tar the files up to STDOUT, then create an SSH session to the remote host, and extract from STDIN. After you’ve got the idea, the command is pretty simple:

tar -cvzf – /path/to/local/dir | ssh root@remotebox “cd /path/to/extract; tar -xvzf -”

That’s it. One simple command and it can save you tons of time creating, copying, and extracting by doing it all at once.

RoundCube Webmail interface ready for Prime Time?

I’ve been working with RoundCube, which is an Ajaxy webmail interface to an IMAP server. The software is still in beta, but I’ve been impressed with it so far.  I’ve integrated into‘s site, so that it matches the rest of their site and had to hack it up a bit to add some dynamic content.  I’ve also just installed it for testing on another mail server that I use to see if I can find any bugs in it.

There are a couple features I’d like to see implemented.   It would be nice to be able to change your email account password from the preferences section.   Also, I’d like to set up a more transparent way to have it look up the backend server based on your email domain.  Guess I’ll subscribe to their dev list and see if I can contribute any code.

Replacing routers

I’ve just spent the last week or so replacing the core networking infrastructure where I work. It involved plenty of late-night work to avoid disrupting service during peak hours. We ended up replacing two of our major routers and implemented a simple tiered architecture which should allow the company to grow much bigger than they are currently.

Since starting work here, I have been the Cisco expert, because I had my CCNP certification (which has since expired). Although I did well on the exams, I had very little practical experience working on live routers. In the couple years that I have been working here, I have learned a lot about configuring routers and switches, and setting up BGP and OSPF routing.

I’ll probably won’t get into a job where these are my primary functions, but I think that its good to understand how networking and routing works. It makes debugging and troubleshooting much easier