When a Linux server has multiple network interfaces, it may be be necessary to choose which NIC gets assigned which name. In particular, we recently had to swap out a NIC that couldn’t handle 100 Meg/Full duplex. After swapping the NIC, if the OS was allowed to choose the interface names by itself, it had them backwards from what they previously were. We have a bunch of networking scripts that had the interface names hard coded, so we didn’t want to change all of those.

After some googling, I came across this page that described how to do it. This allows you to choose which NIC gets which name, based on the MAC address. Simply create a file in /etc/udev/rules.d that contains something like this:

KERNEL=="eth?", SYSFS{address}=="aa:bb:cc:dd:ee:ff", NAME="eth0"
KERNEL=="eth?", SYSFS{address}=="00:11:22:33:44:55", NAME="eth1"

With all of the network configuration I’ve been doing lately, I’ve decided to set up a central syslog server that the routers and switches can log to. That will create a central place that I can look for warnings and errors that are occurring on these devices. It also makes it so I can save the logs for extended periods of time, and use normal Linux tools to search and parse through them.

For some reason, I found it difficult to find instructions on how to create a centralized syslog server. However, its incredibly easy. You just have to configure the Linux syslogd process to listen on a remote interface, then configure the logs like anything else in syslog. In CentOS, you just edit /etc/sysconfig/syslog and add a “-r” to the “SYSLOGD_OPTIONS” line. Then allow port 514 through your firewall from your router’s IP addresses.

On each router, set these global configuration commands:

logging facility local1
logging source-interface FastEthernet0/1
logging 10.0.0.123

On the syslog server, configure your /etc/syslog.conf with something like this:

## Router Logs
local1.* /var/log/routers/router-core.log
local2.* /var/log/routers/router-border.log
local3.* /var/log/routers/switch-1.log
local4.* /var/log/routers/switch-2.log

I also like to set up logrotate with this in /etc/logrotate.d/routers

/var/logs/routers/*.log {
weekly
rotate 52
compress
missingok
notifempty
}

My next step will be creating a LogWatch script to email me anything that I should be concerned about

A bunch of the servers I manage have started having a problem with their daily time sync. Whenever rdate fails, it prints an error message to STDOUT. Since all of the servers are running the sync via cron.daily, this generates a bunch of emails to me, and doesn’t attempt to sync the time for another day.

As a result, I’ve written a simple wrapper for rdate that retries it several times and only prints an error if it was unsuccessful after all of those.

Here is the script. This was written for a CentOS server, so the program paths may need to be changed if you are running it on another distro

#!/bin/bash

## Output nothing if sync eventually works
## Output an error if unsuccessful after $MAX_RETRIES

TIMESERVER="time.nist.gov"
MAX_RETRIES="3"

RETRIES=0
while [[ $RC != 0 && $RETRIES -lt $MAX_RETRIES ]]
do
/usr/bin/rdate -s $TIMESERVER
RC=$?
RETRIES=`/usr/bin/expr $RETRIES + 1`
done > /dev/null 2>&1

if [[ $RC != 0 ]]; then
echo "Error syncing time to $TIMESERVER after $MAX_RETRIES tries"
echo "Last RC was $RC"
fi

On a recent plane trip I was able to read ‘The Cockoo’s Egg‘ by Clifford Stoll. It was a nice to have real technology described for a change. The book is a real-life account of how a Unix System Administrator tracked down a hacker in the early 80’s. Obviously, its quite dated, but the story was pretty interesting, and it was interesting to learn about the vulnerabilities in some of the applications.

The book makes a good balance between the technical stuff and telling the story. It technical enough that I enjoyed it, but still simple enough that my wife can follow it.

I’ve been working with RoundCube, which is an Ajaxy webmail interface to an IMAP server. The software is still in beta, but I’ve been impressed with it so far.  I’ve integrated into hooah.com‘s site, so that it matches the rest of their site and had to hack it up a bit to add some dynamic content.  I’ve also just installed it for testing on another mail server that I use to see if I can find any bugs in it.

There are a couple features I’d like to see implemented.   It would be nice to be able to change your email account password from the preferences section.   Also, I’d like to set up a more transparent way to have it look up the backend server based on your email domain.  Guess I’ll subscribe to their dev list and see if I can contribute any code.

I’ve just spent the last week or so replacing the core networking infrastructure where I work. It involved plenty of late-night work to avoid disrupting service during peak hours. We ended up replacing two of our major routers and implemented a simple tiered architecture which should allow the company to grow much bigger than they are currently.

Since starting work here, I have been the Cisco expert, because I had my CCNP certification (which has since expired). Although I did well on the exams, I had very little practical experience working on live routers. In the couple years that I have been working here, I have learned a lot about configuring routers and switches, and setting up BGP and OSPF routing.

I’ll probably won’t get into a job where these are my primary functions, but I think that its good to understand how networking and routing works. It makes debugging and troubleshooting much easier