Category: Linux System Administration (Page 7 of 11)

Quick MRTG Install With Some Useful System Metrics

May 1, 2008 / Brandon / 0 Comments

I often have a need to monitor some basic system metrics such as memory usage, disk space free, load average, and network usage. MRTG is an ideal solution because it is lightweight and can graph just about anything on a system. Many people only think of MRTG as a tool to graph network interfaces, but because it is just a way to use rrdtool it can be be used for practically anything.

Getting it installed is pretty easy as well. Most distributions provide a package for it, so it is as simple as running ‘yum install mrtg’ or ‘apt-get install mrtg’. Unfortunately the basic package is pretty raw and doesn’t by default know how to obtain much useful data. Over time I’ve compiled a /etc/mrtg/ directory that does a lot of the common things that I like to monitor. Simply uncompress this file to your /etc/mrtg directory and you should have some common useful metrics. You can further customize it per-server if you’d like to monitor some additional things.

Determining What a Server is Used For

April 5, 2008 / Brandon / 0 Comments

When looking at an existing server, it is sometimes difficult to know everything that the server does. I have had many instances when hired for a new job some of them even include some dental services as discounts on teeth whitening, or when looking at a client’s server where I have to find out everything that the server is used for without knowing much about it. Often times there are many more uses than it was originally designed for.

Here is a list of things that I usually check to try and identify what a server is used for:

1- Look at any processes listening on a network socket. I use ‘netstat -lnp’ to identify any listening sockets and what processes are using them. Its common to see SSH, Apache, MySQL, and a mail server. Sometimes there are other things that you should know about, such as FTP, a control panel (ie: webmin on port 10000), and a variety of other processes

2- Cron jobs – A lot of systems have automated processes that run periodically from cron. Make sure to check all of the various cron locations:

The main system crontab:

/etc/crontab

Drop location for cron jobs typically installed with packages:

/etc/cron.d/*

Periodically run jobs:

/etc/cron.*/*

(ie /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly, etc)

User crontabs

/var/spool/cron/*

3- Look at running processes. I use ‘ps auxf’ to identify any other processes that might be running

4- Processes run at boot. On Redhat and derivitives, use ‘ chkconfig –list|grep “:on” ‘ to see all processes that start when the machine boots.

5- Websites configured in Apache: Run ‘ apachectl -t -D DUMP_VHOSTS ‘ to see what Virtual Host are configured in Apache

6- Consider the server’s name, and reverse DNS for any IP’s assigned to it. These may give some hints as to things the server is being used for. For example, if a server has the name ‘mail.mydomain.com’ associated with it somehow, you should probably take a closer look at the mail configuration than you might initially think to do.

That should be a pretty good start of identifying everything that a particular server is used for. Please leave a comment if there is something else that I should add to the list.

Problems to Anticipate When Upgrading From PHP4 to PHP5, and MySQL4 to MySQL5

March 19, 2008 / Brandon / 0 Comments

A client website just upgraded from PHP4 to PHP5 and MySQL4 to MySQL5 and completely broke. Doing such significant upgrades should have been tested first, but for some reason didn’t happen. I got invited to fix and ran across several problems:

MySQL queries containing some explicit JOINs broke. A simple query like this doesn’t work in MySQL5:

SELECT table1.*, table2.*
FROM table1, table2
LEFT JOIN table3 on table1.col1 = table3.col1

In MySQL 5, the JOIN operator now has a higher precedence than the comma operator, so it interprets the query differently. See this post or the MySQL documentation for more information. The quick fix is to put parenthesis around the tables in the FROM statement, like this:

SELECT table1.*, table2.*
FROM (table1, table2)
LEFT JOIN table3 on table1.col1 = table3.col1

The other significant problem was in the upgrade from PHP4 to PHP5, the XML parsing functions are completely different. PHP 4 used the domxml extentions, where PHP 5 uses a newere DOM extention.

From https://www.php.net/manual/en/ref.domxml.php:

It will, however, never be released with PHP 5, and will only be distributed with PHP 4. If you need DOM XML support with PHP 5 you can use the DOM extension. This domxml extension is not compatible with the DOM extension.

The solution for fixing this, however is quite a bit more complicated. I had to rewrite the XML producing scripts to use the new functionality. Fortunately, the new DOM functionality is pretty straightforward and easier to write, so porting it from one to the other is fairly straightforward, but does require some effort.

MyHosting.com Features are Seriously Lacking

March 10, 2008 / Brandon / 0 Comments

I got my first SourceForge Marketplace job a few days ago for an installation of Awstats for a customer.Â I’ve installed Awstats plenty of times and I sometimes have a qwirk or two, but it is generally pretty painless.

Not for an account at MyHosting.com.

Their features are seriously lacking and their security prevention measures too intrusive. Â I should have know that I was in for trouble when I realized that they don’t offer any way to create cron jobs.Â I ended up having to use a free account with Remote-Cron.com instead, which works, but is not ideal.

The layout of their directories was not very intuitive, and took too much tinkering to figure out.Â But the really annoying part is that they limit the file size created by the web server to just 100k. Â I was trying to parse a 40 MB of log files for a given month, and awstats would quickly die with an Internal Server Error. Â It took a while to finally figure out that it was choking because of the 100k file size limit.

I’ve submitted a ticket to their support asking to increase the file size limit, but will be interested to see if they will allow it.

Google Spam Filtering Sounds Great but I Can’t Sign Up

February 6, 2008 / Brandon / 3 Comments

Google announced yesterday new services and pricing based on their Postini message filtering service. The service sounds great, and I’ve been looking at moving away from my current mail filtering service for a couple months now. Pricing starts out at only $3.00 per user per year. I did a little checking around, and verified that I can add domain aliases and user aliases and that it looks like they can be tied to a single $3.00 account.

That is exactly like what I need. I have a bunch of domains, and use several email addresses at each one that all forward to a single Inbox. For $3.00 a year, it sounds like a great savings over my alternate plan which was creating my own MailScanner box. Plus with Google, I won’t have to worry about redundancy, or keeping my own filtering up to date.

Perfect, so I went to sign up. I put in my domain name, agreed to the TOS, then put in my credit card information and hit submit:

Google won't let me sign up for Posting

Oops, looks like something went wrong there. That’s not the best way to instill confidence into your new customers.

GoDaddy’s DNS Doesn’t Update SOA Serial

February 4, 2008 / Brandon / 2 Comments

I recently moved one of my blog’s to it’s own IP address, but strangely Google’s feed readers are still picking up that site on the original IP Address. It has been several days now, and Google is still requesting the site at the old IP Address. I did some digging and found that even though I changed the IP Address, the SOA Serial didn’t get incremented. As a result, Google’s DNS servers are using cached records and not requesting new ones because the serial hasn’t changed.

This seems like a pretty serious problem for GoDaddy. I double checked everything again tonight by creating some new records. The new records resolve to the IP’s that I specified, but the serial remains unchanged.

I tried various things that definitely should have caused the serial number to be incremented:
– Adding a new A record,
– Modifying an A record
– Deleting an A record

None of which updated the serial as it should have.

Finally, I noticed on the main page for my domain (the one that lists the name servers, registrant info, etc) that next to Name Servers: it said Last Update: 11/23/2007, which coincided with the date of the serial. I was finally able to update the Serial by acting like I was changing my name servers, then just submitting the page without making any changes.

It seems this is a fundamentally broken DNS system though. Frankly, I’m pretty surprised to have something like that from GoDaddy, where I’m sure you do DNS for hundreds of thousands of domains. While troubleshooting, I emailed GoDaddy’s support a couple times and were less than helpful. Their basic response was:

We are unable to update the SOA serial on demand. This information is updated periodically, and is the way our systems currently process. We apologize for any inconvenience this may cause.

Converting mbox’s to maildir format

February 1, 2008 / Brandon / 3 Comments

There is a handy utility for converting mbox style mailboxes into maildir format at https://batleth.sapienti-sat.org/projects/mb2md/

To convert all of the mailboxes on your server:

Edit /etc/sudoers and comment out the env_keep section. These variables make it so that the sudo command keeps some environment variables and tries to put things in the wrong directory.

Download mb2db, unzip it, and copy the binary to /bin (where all users can access it)

# wget https://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz
# gunzip mb2db-3.20.pl.gz
# cp mb2db-3.20.pl.gz / bin

Then run this command to convert all of the mailboxes into maildir format.

cd /var/spool/mail

for username in `ls`; do echo $username; sudo -u $username /bin/mb2md -m -d Maildir; done

That will create a directory called Maildir in each user’s home directory. Then just configure your MTA to deliver mail there, and your IMAP server to pick it up there
In postfix, add this to /etc/postfix/main.cf

home_mailbox = mail/

And in Dovecot, change this in /etc/dovecot.conf

mail_location=maildir:~/mail/

Now you can edit /etc/sudoers and uncomment the env_keep section.

Using Jailkit for chrooting shell accounts

January 17, 2008 / Brandon / 1 Comment

I’ve toyed around with chrooting a shell account to a directory before, but never really done it before. Today a customer wanted it done, so I had a chance to figure it all out. I’ve considered the using chrooted ssh before, but that requires a patch to SSH. Today I came across jailkit which leaves SSH alone, but implements the chroot as the users shell. It seemed pretty straightforward, plus provides some utilities for creating the jail.

cd /usr/local/src
wget https://olivier.sessink.nl/jailkit/jailkit-2.4.tar.gz
tar -xvzf jailkit-2.4.tar.gz
cd jailkit-2.4
./configure && make && make install

The tools were then available. Their examples said to put the jail environment, but I figured I might want to create per-user jails, so I created it in /home/jail-someuser like this:

jk_init -v -j /home/jail-someuser basicshell editors extendedshell netutils ssh sftp scp

That creates the directory and copies all of the specified programs into place inside the jail. In addition, it also copies all of necessary libraries as well – which is much easier than finding them with ldd.

Now, just create the actual user account and some directories for inside the jail:

mkdir /home/jail-someuser/home/someuser
useradd -d /home/jail-someuser/./home/someuser -s /usr/sbin/jk_chrootsh
chown someuser:someuser home/jail-someuser/./home/someuser
mkdir /home/jail-someuser/tmp
chmod a+rwx /home/jail-someuser/tmp

I was then able to log in by SSHing to the box as someuser. Upon logging in, I noticed that the default debian bash login script had some problems because the ‘id’ command wasn’t available. Also, vi wasn’t available, so I copied both of those programs those into the jail (fortunately their required libraries seem to already be there)

Overall it was pretty painless to install and get working. I’m quite impressed.

The new wave of HTTP referrer spam

January 15, 2008 / Brandon / 3 Comments

I’ve noticed an increase in HTTP Referrer spam on my own web site and in some websites that I manage. See Wikipedia’s articles on the HTTP Referrer and Referrer spam for a definition of what exactly referrer spam is.

Wikipedia, and some other pages on the Internet that I found describing referrer spam say that the spammer’s intent is to end up on published web stats pages in order to create links to their site. I don’t think that is (or no longer is) the case.

I would argue that the real intent of these spammers is to get the website owner who is looking at the stats, to click on their links. Most users who have a blog or small website check their statistics often, and are really interested when they find a new site that appears to be linking to theirs. It is very likely that they will intentionally look at any new incoming links.

As evidence along this route, I just noticed that I got 4 hits on one of my sites with the following referrer:

https://www.amazon.com/s/ref=sr_pg_4&tag=somespamer_20

I’m familiar with Amazon’s link structure and immediately noticed that it was an affiliate URL. If you hit that URL, then Amazon will attribute your click as coming from the spammer. Amazon will set a cookie that contains the spammers affiliate ID, and any purchase that you make at Amazon in the next 30 days will be credited to the spammer. They will then get a 4% commission on your purchases.

Obviously, not everybody buys something from Amazon once a month, but I’d bet that enough people do to make it worth the risk. Fortunately, it looks like Amazon has already caught on to this one, and that particular link just goes to an error page now.

That is a pretty deceitful and probably successful tactic for the spammer. Creating referrer spam is incredibly easy. I don’t think there is any great way to detect it either. I’ve seen some WordPress plugins and such that attempt to deal with it, but I don’t think there is much going on in this area yet.

My first thought would be to request the referred page and look for links to your site. That has some potential problems working reliably on a large scale though. Also, it might enable a sortof distributed denial of service by proxy attack.

Another possible way to fight referrer spam would involve a blacklist. t could contain both IP Addresses of known spammers, and the links that they are spamming. I found one called referrercop that looks like it is owned by Google now, so that may show some promise – although it doesn’t look like it has been updated recently.

PHP Performance – isset() versus empty() versus PHP Notices

December 25, 2007 / Brandon / 7 Comments

I’m cleaning up a lot of PHP code and always program with PHP error_reporting set to E_ALL and display_errors turned on so that I make sure to catch any PHP messages that come up. Since starting on this site, I have fixed literally hundreds (maybe thousands) of PHP Notices about using uninitialized variables and non-existent array indexes.

I have been fixing problems like this where $somevar is sometimes undefined:

if ($somevar)

by changing it to:

if (isset($somevar) && $somevar)

This successfully gets rid of the NOTICEs, but adds some overhead because PHP has to perform two checks. After fixing a lot of this in this manner, I’ve noticed that the pages seem to be generated a little slower.

So, to provide some conclusive results to myself, I wrote up a quick benchmarking script – available at php_empty_benchmark.php. It goes through 1,000,000 tests using each of these methods:

if ($a) – This generates a notice if $a is not set
if (isset($a)) – A simple clean way to check if the variable is set (note that it is not equivalent to the one above)
if (isset($a) && ($a) – The one that I have been using which is equivalent to if($a), but doesn’t generate a notice.
if (!empty($a)) – This is functionally equivalent to if($a), but doesn’t generate a notice.

It measures the time to perform 1 million tests using a defined percentage of values that are set.Â It then computes the difference as a percentage of the time taken for the original test (the one that generates the notices).Â Â A ‘diff’ of 100 means that the execution time is the same, greater than 100 means that it is faster, and less than 100 means that it is slower. A typical test produced these results:

    With NOTICE: 0.19779300689697
    With isset:  0.19768500328064 / Diff: 100.05463419811
    With both:   0.21704912185669 / Diff: 91.128222590815
    with !empty: 0.19779801368713 / Diff: 99.997468735875

In summary, using the if (isset($a) && $a) syntax is about 8-10% slower than generating the PHP Notice. Using !empty() should be a drop-in replacement that doesn’t generate the notice and has virtually no performance impact. Using ifset() also has no performance impact, but is not exactly the same as ‘if($a)’ since isset() will return true if the variable is set to a false value. I included it here, because it often make the code a little more readable than the !empty($a) syntax. For example:

$myvalue = !empty($_REQUEST['myvalue']) ? $_REQUEST['myvalue'] : '';

Versus

$myvalue = isset($_REQUEST['myvalue']) ? $_REQUEST['myvalue'] : '';