Month: March 2007

Sometimes I’ll be on an old box and need to install a CPAN perl module for something.  But when starting up CPAN, it tries to hit a bunch of mirrors that no longer exist.  I just found out that you can easily clear the CPAN configuration and  have it re-ask you all of the inital configuration questions.   Simply use the command ‘o conf init’ at the cpan> prompt.

I usually install Bundle::CPAN first so that you get a bunch of the important stuff, then install whatever else you need.

A customer asked today why our control panel showed their server with over 10 GB of traffic, but the ifconfig on their server showed that they had done just over 1 GB of traffic.  Their server had not been rebooted for several months, so it should have accounted for all traffic during the current month.

What they didn’t realize was that the counters used by ifconfig are 32 bit counters, so they can only go as high as about 4.3 GB (technically 2^32 = 4,294,967,296 bytes) before rolling over.  With 10 GB of traffic during a month, that would mean that they roll over a little more than 2 times per month, or about every 2 weeks.

I spent most of the afternoon and evening rewiring the cable and data in our house. I’m also terminating a couple of cables into wall-jacks and putting a small punch down block in a central area so that things are wired a little better.

I got most of the way through everything, and was doing a cable where a room isn’t finished, so I was going to leave the RJ-45 end on.   I wired it up at the punch down block just like I did all of the others, but I couldn’t get it to work.  I tore the ends apart and put them back together a couple times to make sure everything was connected good and tight.

It turns out, that I had the greens and browns mixed up on the connector I was using.  The other jacks that I had done, were at least consistently wrong on both ends, so they worked.  But since this one was ending in an RJ-45, it actually mattered, and didn’t work.  Stupid color-blindness.
Anyways, I’m a perfectionist, so went back and re-did most of the termination with the correct colors.

A while ago, a friend of mine showed me the Load Time Analyzer for Firefox.  It’s a pretty cool tool that can analyze all of the different HTTP requests that make up a page load.  Then it can display a simple graph showing how long all of the requests took.  It could be very useful for trying to identify requests that are slow to load.

I recently had a reason to use the plugin, so I turned it on and found what I needed to.  I removed the toolbar for it when I was done with it.  But the past couple days, my browser has gotten painfully slow after being open for a couple hours.   It was using a ton of memory, and just loading a simple page, or opening a new tab would sometimes take ten seconds.

Turns out that the Load Time Analyzer was still running, even though the toolbar wasn’t being displayed.  It was keeping track of everything page I visited, which is why it was using so much memory and running so slowly.   To actually disable it, you have to go into your ‘Addons’ menu, and actually disable it.   That, of course, requires a browser restart, which is kindof annoying.

It would be nice if future versions had an option to disable it without a browser restart.  For now, make sure to enable it only while you need to use it.

When a Linux server has multiple network interfaces, it may be be necessary to choose which NIC gets assigned which name. In particular, we recently had to swap out a NIC that couldn’t handle 100 Meg/Full duplex. After swapping the NIC, if the OS was allowed to choose the interface names by itself, it had them backwards from what they previously were. We have a bunch of networking scripts that had the interface names hard coded, so we didn’t want to change all of those.

After some googling, I came across this page that described how to do it. This allows you to choose which NIC gets which name, based on the MAC address. Simply create a file in /etc/udev/rules.d that contains something like this:

KERNEL=="eth?", SYSFS{address}=="aa:bb:cc:dd:ee:ff", NAME="eth0"
KERNEL=="eth?", SYSFS{address}=="00:11:22:33:44:55", NAME="eth1"

With all of the network configuration I’ve been doing lately, I’ve decided to set up a central syslog server that the routers and switches can log to. That will create a central place that I can look for warnings and errors that are occurring on these devices. It also makes it so I can save the logs for extended periods of time, and use normal Linux tools to search and parse through them.

For some reason, I found it difficult to find instructions on how to create a centralized syslog server. However, its incredibly easy. You just have to configure the Linux syslogd process to listen on a remote interface, then configure the logs like anything else in syslog. In CentOS, you just edit /etc/sysconfig/syslog and add a “-r” to the “SYSLOGD_OPTIONS” line. Then allow port 514 through your firewall from your router’s IP addresses.

On each router, set these global configuration commands:

logging facility local1
logging source-interface FastEthernet0/1
logging 10.0.0.123

On the syslog server, configure your /etc/syslog.conf with something like this:

## Router Logs
local1.* /var/log/routers/router-core.log
local2.* /var/log/routers/router-border.log
local3.* /var/log/routers/switch-1.log
local4.* /var/log/routers/switch-2.log

I also like to set up logrotate with this in /etc/logrotate.d/routers

/var/logs/routers/*.log {
weekly
rotate 52
compress
missingok
notifempty
}

My next step will be creating a LogWatch script to email me anything that I should be concerned about

A bunch of the servers I manage have started having a problem with their daily time sync. Whenever rdate fails, it prints an error message to STDOUT. Since all of the servers are running the sync via cron.daily, this generates a bunch of emails to me, and doesn’t attempt to sync the time for another day.

As a result, I’ve written a simple wrapper for rdate that retries it several times and only prints an error if it was unsuccessful after all of those.

Here is the script. This was written for a CentOS server, so the program paths may need to be changed if you are running it on another distro

#!/bin/bash

## Output nothing if sync eventually works
## Output an error if unsuccessful after $MAX_RETRIES

TIMESERVER="time.nist.gov"
MAX_RETRIES="3"

RETRIES=0
while [[ $RC != 0 && $RETRIES -lt $MAX_RETRIES ]]
do
/usr/bin/rdate -s $TIMESERVER
RC=$?
RETRIES=`/usr/bin/expr $RETRIES + 1`
done > /dev/null 2>&1

if [[ $RC != 0 ]]; then
echo "Error syncing time to $TIMESERVER after $MAX_RETRIES tries"
echo "Last RC was $RC"
fi

On a recent plane trip I was able to read ‘The Cockoo’s Egg‘ by Clifford Stoll. It was a nice to have real technology described for a change. The book is a real-life account of how a Unix System Administrator tracked down a hacker in the early 80’s. Obviously, its quite dated, but the story was pretty interesting, and it was interesting to learn about the vulnerabilities in some of the applications.

The book makes a good balance between the technical stuff and telling the story. It technical enough that I enjoyed it, but still simple enough that my wife can follow it.