Reconfiguring CPAN from scratch

Sometimes I’ll be on an old box and need to install a CPAN perl module for something.  But when starting up CPAN, it tries to hit a bunch of mirrors that no longer exist.  I just found out that you can easily clear the CPAN configuration and  have it re-ask you all of the inital configuration questions.   Simply use the command ‘o conf init’ at the cpan> prompt.

I usually install Bundle::CPAN first so that you get a bunch of the important stuff, then install whatever else you need.

32 bit counters in ifconfig

A customer asked today why our control panel showed their server with over 10 GB of traffic, but the ifconfig on their server showed that they had done just over 1 GB of traffic.  Their server had not been rebooted for several months, so it should have accounted for all traffic during the current month.

What they didn’t realize was that the counters used by ifconfig are 32 bit counters, so they can only go as high as about 4.3 GB (technically 2^32 = 4,294,967,296 bytes) before rolling over.  With 10 GB of traffic during a month, that would mean that they roll over a little more than 2 times per month, or about every 2 weeks.

When color-blindness sucks

I spent most of the afternoon and evening rewiring the cable and data in our house. I’m also terminating a couple of cables into wall-jacks and putting a small punch down block in a central area so that things are wired a little better.

I got most of the way through everything, and was doing a cable where a room isn’t finished, so I was going to leave the RJ-45 end on.   I wired it up at the punch down block just like I did all of the others, but I couldn’t get it to work.  I tore the ends apart and put them back together a couple times to make sure everything was connected good and tight.

It turns out, that I had the greens and browns mixed up on the connector I was using.  The other jacks that I had done, were at least consistently wrong on both ends, so they worked.  But since this one was ending in an RJ-45, it actually mattered, and didn’t work.  Stupid color-blindness.
Anyways, I’m a perfectionist, so went back and re-did most of the termination with the correct colors.

Multi-threaded perl

I’ve been experimenting on multi-threading in perl for a new project, and am impressed with how straightforward it is. Before digging into it, I never really considered doing anything with it because it was always kindof ‘mysterious’ to me. Now, I’m seeing how useful it is to have multiple threads that are able to share variables.

In the application I’m rewriting, I used to have one script that listened for network data, then saved that out to a file. I had another script that read through the output files, and then inserted the data into a database. Now, with a multi-threaded program, I just have one thread that listens, and another thread (or multiple threads) that parse the data and manipulate it however I want. In this case, that saves a lot of disk activity, and makes the program a lot more efficient, and straight-forward.
I’m also able to use the Thread::Queue module to create a queue that the listener process can add to, and then have ‘worker’ threads that can go through the data and format/summarize/whatever I’m going to do with it.

I’m looking forward to seeing how this all works out.  I’m impressed so far.

Firefox Load Time Analyzer is cool, but …

A while ago, a friend of mine showed me the Load Time Analyzer for Firefox.  It’s a pretty cool tool that can analyze all of the different HTTP requests that make up a page load.  Then it can display a simple graph showing how long all of the requests took.  It could be very useful for trying to identify requests that are slow to load.

I recently had a reason to use the plugin, so I turned it on and found what I needed to.  I removed the toolbar for it when I was done with it.  But the past couple days, my browser has gotten painfully slow after being open for a couple hours.   It was using a ton of memory, and just loading a simple page, or opening a new tab would sometimes take ten seconds.

Turns out that the Load Time Analyzer was still running, even though the toolbar wasn’t being displayed.  It was keeping track of everything page I visited, which is why it was using so much memory and running so slowly.   To actually disable it, you have to go into your ‘Addons’ menu, and actually disable it.   That, of course, requires a browser restart, which is kindof annoying.

It would be nice if future versions had an option to disable it without a browser restart.  For now, make sure to enable it only while you need to use it.

DHCP ‘always-broadcast’ confusion

I run a DHCP server using Linux’s dhcpd program to serve addresses to a bunch of clients.  These clients are connected over several wireless links, and the radios are sometimes quirky.  Specifically, some clients never get the DHCPOFFER unless the ‘always-broadcast’ parameter is on.  This usually works out fine.

Today, however, we had a problem where we just saw a bunch of incoming DHCPDISCOVER messages, to which the server would reply with a DHCPOFFER. The devices would just continually send discover messages, and none would ever DHCPREQUEST an address.

From what I can conclude, I think that the clients were confused when they received multiple broadcast responses back for their DHCPDISCOVER message.  The client would then send another discover message, which just caused a never-ending loop of requests and offers.

To resolve the problem, I turned off always-broadcast for a few minutes.  This made the clients wait for a random period of time before discovering again.  Some clients accepted the IP fine even though it wasn’t broadcast.  For the ones that didn’t, I then re-enabled always-broadcast, and they picked up an address the next time that they tried.

For a long term solution, we’re working on subnetting the two /24 networks that are currently together into smaller /26 or /27 blocks.  That should reduce the possibility of having this happen again.

How to choose which NIC gets ‘eth0’

When a Linux server has multiple network interfaces, it may be be necessary to choose which NIC gets assigned which name. In particular, we recently had to swap out a NIC that couldn’t handle 100 Meg/Full duplex. After swapping the NIC, if the OS was allowed to choose the interface names by itself, it had them backwards from what they previously were. We have a bunch of networking scripts that had the interface names hard coded, so we didn’t want to change all of those.

After some googling, I came across this page that described how to do it. This allows you to choose which NIC gets which name, based on the MAC address. Simply create a file in /etc/udev/rules.d that contains something like this:

KERNEL=="eth?", SYSFS{address}=="aa:bb:cc:dd:ee:ff", NAME="eth0"
KERNEL=="eth?", SYSFS{address}=="00:11:22:33:44:55", NAME="eth1"

Syslog server

With all of the network configuration I’ve been doing lately, I’ve decided to set up a central syslog server that the routers and switches can log to. That will create a central place that I can look for warnings and errors that are occurring on these devices. It also makes it so I can save the logs for extended periods of time, and use normal Linux tools to search and parse through them.

For some reason, I found it difficult to find instructions on how to create a centralized syslog server. However, its incredibly easy. You just have to configure the Linux syslogd process to listen on a remote interface, then configure the logs like anything else in syslog. In CentOS, you just edit /etc/sysconfig/syslog and add a “-r” to the “SYSLOGD_OPTIONS” line. Then allow port 514 through your firewall from your router’s IP addresses.

On each router, set these global configuration commands:

logging facility local1
logging source-interface FastEthernet0/1
logging 10.0.0.123

On the syslog server, configure your /etc/syslog.conf with something like this:

## Router Logs
local1.* /var/log/routers/router-core.log
local2.* /var/log/routers/router-border.log
local3.* /var/log/routers/switch-1.log
local4.* /var/log/routers/switch-2.log

I also like to set up logrotate with this in /etc/logrotate.d/routers

/var/logs/routers/*.log {
weekly
rotate 52
compress
missingok
notifempty
}

My next step will be creating a LogWatch script to email me anything that I should be concerned about

rdate synchronization problems

A bunch of the servers I manage have started having a problem with their daily time sync. Whenever rdate fails, it prints an error message to STDOUT. Since all of the servers are running the sync via cron.daily, this generates a bunch of emails to me, and doesn’t attempt to sync the time for another day.

As a result, I’ve written a simple wrapper for rdate that retries it several times and only prints an error if it was unsuccessful after all of those.

Here is the script. This was written for a CentOS server, so the program paths may need to be changed if you are running it on another distro

#!/bin/bash

## Output nothing if sync eventually works
## Output an error if unsuccessful after $MAX_RETRIES

TIMESERVER="time.nist.gov"
MAX_RETRIES="3"

RETRIES=0
while [[ $RC != 0 && $RETRIES -lt $MAX_RETRIES ]]
do
/usr/bin/rdate -s $TIMESERVER
RC=$?
RETRIES=`/usr/bin/expr $RETRIES + 1`
done > /dev/null 2>&1

if [[ $RC != 0 ]]; then
echo "Error syncing time to $TIMESERVER after $MAX_RETRIES tries"
echo "Last RC was $RC"
fi

Nice change: A novel with real technology

On a recent plane trip I was able to read ‘The Cockoo’s Egg‘ by Clifford Stoll. It was a nice to have real technology described for a change. The book is a real-life account of how a Unix System Administrator tracked down a hacker in the early 80’s. Obviously, its quite dated, but the story was pretty interesting, and it was interesting to learn about the vulnerabilities in some of the applications.

The book makes a good balance between the technical stuff and telling the story. It technical enough that I enjoyed it, but still simple enough that my wife can follow it.