Category: General (Page 7 of 26)

Speed up a Linux Software Raid Rebuild

January 25, 2010 / Brandon / 2 Comments

I’m setting up software raid on a running server and it is taking forever for the initial sync of the raid drives on the 1TB hard disks. It has been running for about 6 hours and says that it will take about 5 days (7400 minutes) as this pace:

[root@host ~]# cat /proc/mdstat
md1 : active raid1 sdb3[1] sda3[2]
      974559040 blocks [2/1] [_U]
      [>....................]  recovery =  3.9% (38109184/974559040) finish=7399.1min speed=2108K/sec

I did some read and write tests directly to the drives using dd to make sure that they were working okay, and they can operate at about 100 MB/s

[root@host ~]# dd if=/dev/zero of=/dev/sda2 bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 10.8882 seconds, 96.3 MB/s
[root@host ~]# dd if=/dev/zero of=/dev/sdb2 bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 11.1162 seconds, 94.3 MB/s
[root@host ~]# dd if=/dev/sda2 of=/dev/null bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 10.2829 seconds, 102 MB/s
[root@host ~]# dd if=/dev/sdb2 of=/dev/null bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 10.5109 seconds, 99.8 MB/s

What I failed to realize is that there is a configurable limit for the min and max speed of the rebuild. Those parameters are configured in /proc/sys/dev/raid/speed_limit_min and /proc/sys/dev/raid/speed_limit_max. They default to a pretty slow 1MB/s minimum which was causing it to take forever.

Increasing the maximum limit didn’t automatically make it faster either. I had to increase the minimum limit to get it to jump up to a respectable speed.

[root@host ~]# echo 100000 > /proc/sys/dev/raid/speed_limit_min

[root@host ~]# watch cat /proc/mdstat
Every 2.0s: cat /proc/mdstat 
md1 : active raid1 sdb3[1] sda3[2]
      974559040 blocks [2/1] [_U]
      [=>...................]  recovery =  7.7% (75695808/974559040) finish=170.5min speed=87854K/sec

Now it is up around 87 MB/s and will take just a few hours to complete the rest of the drive.

PHP Wrapper Class for a Read-only database

January 4, 2010 / Brandon / 0 Comments

This is a pretty special case of a database wrapper class where I wanted to discard any updates to the database, but want SELECT queries to run against an alternative read-only database. In this instance, I have a planned outage of a primary database server, but would like the public-facing websites and web services to remain as accessible as possible.

I wrote this quick database wrapper class that will pass all SELECT queries on to a local replica of the database, and silently discard any updates. On this site almost all of the functionality still works, but it obviously isn’t saving and new information while the primary database is unavailable.

Here is my class. This is intended as a wrapper to an ADOdb class, but it is generic enough that I think it would work for many other database abstraction functions as well as seamless data pump.

class db_unavailable {
    var $readonly_db;

    function __construct($readonly_db)
    {
        $this->query_db = $readonly_db;
    }

    function query($sql)
    {
        $args = func_get_args();
        if (preg_match("#(INSERT INTO|REPLACE INTO|UPDATE|DELETE)#i", $args[0])) {
            // echo "Unable to do insert/replace/update/delete query: $sql\n";
            return true;
        } else {
            return call_user_func_array(array($this->readonly_db, 'query'), $args);
        }
    }

    function __call($function, $args)
    {
        return call_user_func_array(array($this->readonly_db, $function), $args);
    }
}

I simply create my $query_db object that points to the read-only database. Then create my main $db object as a new db_unavailable() object. Any select queries against $db will behave as they normally do, and data-modifying queries will be silently discarded.

LUG Presentation on SQL Basics

December 17, 2009 / Brandon / 0 Comments

I gave a presentation tonight at my local Linux Users Group meeting on SQL Basics. I had a fun time preparing the presentation and made up a bunch of examples having to do with Santa’s database.

It started out with a simple table for kids who were either naughty or nice. We then added some reports to that. Then imported kids’ wish lists from CSV files. From there we were able to generate some manufacturing reports for the workshop.

When we joined the wish list table with the kids table, we were then able to generate a sleigh-loading report which included only gifts for kids who had been good. Then we got even more complicated and introduced several joins with some complicated mathematics to select gifts for kids within a certain radius from a given zip code.

The presentation is available for download here. And Brian recorded part of the presentation which is available to view on uStream.tv

Booting from a Software Raid device on Ubunto Karmic (9.10)

October 26, 2009 / Brandon / 4 Comments

Its a few days away from Karmic’s official release, and I’m putting together a new computer and thought I would give the new version a try. I set everything up with Software raid, and it is still annoying to me that to do so on Ubuntu still requires the Alternate CD and a text-based install.

I configured the /boot partition to use a RAID1 array, as I normally do. After getting most of the way through the install, I got a big red screen with the following error:

====================================
Unable to install GRUB in /dev/md0
Executing 'grub-install /dev/md0 failed'
This is a fatal error
====================================

I tried several variations on continuing, but nothing seemed to work. Turns out that a known issue with Karmic is that this doesn’t work. Actually the known issue says that grub is only installed to the first drive, but that was not my experience.

Bug 420748 gives some detail on the issue, and it seems that the change to Grub2 might have something to do with it. From the red screen, I chose to proceed without installing a boot loader. I Rebooted to Alternate CD, and chose to repair a broken installation. Once to a shell, I ran these commands to install grub to the master boot records on both drives

grub-install /dev/sda
grub-install /dev/sdb

I removed the CD and rebooted. It ended up at a grub shell. At which point I ran the following to boot into the system (note the difference in grub2 commands)

set root=(hd0,1)
linux /vmlinuz-2.6.31-14-generic ro root=/dev/md1 (use tab-completion on the kernel image filename)
initrd /initrd.img-2.6.31-14-generic (again, use tab-completion)
boot

At that point, it started up normally. Once logged in, I started a terminal and then ran this command as root to create a minimal grub.cfg file (note that it is no longer called menu.lst)

grub-mkconfig > /boot/grub/grub.cfg

I’m now able to reboot into a working system. So much for a flawless install experience.

Setting Up Virtualmin on an OpenVZ Guest

September 7, 2009 / Brandon / 0 Comments

I’m experimenting with a hosting control panel and am interested in Virtualmin. I generally avoid web-based control panels, because they generally make direct configuration via the command line and manually editing config files very difficult. However one of Virtualmin’s goals is to not interfere with such manual configurations. I’ve had plenty of clients who use Webmin, and they seem to do a good job, so Virtualmin seems like a good choice.

These are the steps that I went through to get a new OpenVZ guest set up with the GPL version of Virtualmin.

Download a CentOS 5 OS template and create the guest

# wget https://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz
# vzctl create <VEID> --ostemplate centos-5-x86_64

I replaced all of these limits in /etc/vz/<VEID>.conf. This is based off of a different running machine with some fairly generous limits. Most importantly, it includes 1GB of RAM.

# UBC parameters (in form of barrier:limit)
KMEMSIZE="43118100:44370492"
LOCKEDPAGES="256:256"
PRIVVMPAGES="262144:262144"
SHMPAGES="21504:21504"
NUMPROC="2000:2000"
PHYSPAGES="0:9223372036854775807"
VMGUARPAGES="65536:65536"
OOMGUARPAGES="26112:9223372036854775807"
NUMTCPSOCK="360:360"
NUMFLOCK="380:420"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="10321920:16220160"
TCPRCVBUF="1720320:2703360"
OTHERSOCKBUF="4504320:16777216"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="5000:5000"
DCACHESIZE="3409920:3624960"
NUMFILE="18624:18624"
AVNUMPROC="180:180"
NUMIPTENT="128:128"

Then set up some host-specific parameters and start it up.

# vzctl set <VEID> --ipadd 10.0.0.1 --hostname yourhostname.com --nameserver 4.2.2.1 --diskspace 4G --save
# vzctl start <VEID>
# vzctl enter <VEID>

You are now logged in to the guest, where you can download and install virtualmin

# yum update
# cd /root
# wget https://software.virtualmin.com/gpl/scripts/install.sh
# sh install.sh
 Continue? (y/n) y

That should install without significant errors. Finally, set a password for root, and then log in to Virtualmin to go through the post-installation configuration

passwd root

No-Referer Plugin for Firefox 3.7

August 28, 2009 / Brandon / 4 Comments

Not allowing Firefox to install plugins from previous versions is kindof annoying. Especially when the developer doesn’t maintain something. The No-Referer plugin is very simple and handy, but hasn’t been updated to allow it to be installed in Firefox 3.5 or above.

In many cases, all that needs to be done is a version number needs to be incremented in the install.rdf file. I’ve posed a new version of this that works for FireFox 3.7 here

Converting Tables to InnoDB Takes Forever

August 25, 2009 / Brandon / 0 Comments

I have been familiar with some of the benefits of the InnoDB storage engine for a long time. Of particular interest to me has been row-level locking which should significantly reduce some locking problems that I have on a big table during peak times.

I have made various attempts over the past six months or so to convert this table to InnoDB, but each attempt has always taken a tremendously long time and I have always ended up canceling the query before it completes, because it doesn’t seem to be making any progress.

I finally have had a reason and opportunity to dig into this more and have spent the last day or two learning and experimenting with InnoDB. Of particular use was this set of slides from a 2007 presentation on InnoDB performance. It had some very good information about understanding how InnoDB works.

Slide 9 of the presentation includes this extremely helpful bit:

PRIMARY KEY in random order are costly and lead to table fragmentation (primary key inserts should normally be in an ascending order)
Load data in primary_key order if you can

In the data that I have been attempting to convert from MyISAM, my primary keys are inserted in whatever order that that the data has occurred. When converting that to InnoDB, it would basically run a bunch of insert statements using my unordered data. Each insert would require the storage engine to move a bunch of data in the InnoDB table around to keep the primary keys in order.

In my case, I was converting a MyISAM table containing about 1.9 million rows and occupying 600 MB of disk space. That took over 8 hours using the unordered data. After ordering the data and retrying, it then took about 10 minutes.

Also, I learned the usefulness of the ‘SHOW TABLE STATUS’ command. It is semi-useful in seeing the progress of the import. It is a bit strange to me it shows the number of rows in the table changing by 10% every second, but it is better than nothing:

mysql> SHOW TABLE STATUS WHERE name = 'sometable'

*************************** 1. row ***************************
           Name: books_innodb
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 2158501
 Avg_row_length: 286
    Data_length: 617611264
Max_data_length: 0
   Index_length: 70008832
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2009-08-25 10:03:43
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options:
        Comment: InnoDB free: 1799168 kB

Also worth noting is the data inserted in random order used about 30% more space according to the Data_length value.

AmazonAPISigning.com

July 26, 2009 / Brandon / 0 Comments

Amazon Associate Web Services will start requiring API Requests to contain a cryptographic signature on August 15th. Any website that uses their Product API’s will need to be modified to correctly sign the requests using their Secret Key.

I’ve just created AmazonAPISigning.com which is a website that offers services to help in making the transition to the signed requests. Specifically, it offers a programming service to modify websites code to implement the required changes. It also offers a free API Signing service for those websites that aren’t able to implement the signatures within their own code for whatever reason. The signing service is specifically intended for widgets and tools that are implemented completely via JavaScript, and thus aren’t able to keep their private key hidden from the Javascript Source code.

The signing service may also serve as a really quick solution for webmasters to be able to start signing requests. Their website code simply needs to change the hostname used in the Amazon API requests, and the service will start calculating the signatures on their behalf.

ProFTPd allows multipled DefaultRoot lines for flexible chrooting

July 2, 2009 / Brandon / 0 Comments

The ProFTPd documentation gives good examples of how to use the DefaultRoot directive to chroot users to a specific directory.

A customer today wanted to have different chroot directories for different groups of users. The documentation didn’t mention if it was okay to include multiple DefaultRoot lines. After some experimenting, I can verify that it is allowed and works well.

I used something like this in /etc/proftpd/proftpd.conf

DefaultRoot                     ~ jailed
DefaultRoot                     ~/../.. othergroup

Users in the group ‘jailed’ are chrooted to their own home directory immediately upon logging in. Users in the ‘othergroup’ are chrooted two levels up from their home directory. If you want to get really specific, each user generally has a group of their own, so you can effectively do this a the user-level as well.

PHP Code to Sign any Amazon API Requests

June 30, 2009 / Brandon / 22 Comments

Starting next month, any requests to the Amazon Product Advertising API need to be cryptographically signed. Amazon has given about three months notice and the deadline is quickly approaching. I use the Amazon web services on several sites and came up a fairly generic way to convert an existing URL to a signed URL. I’ve tested with several sites and a variety of functions, and this is working well for me so far:

function signAmazonUrl($url, $secret_key)
{
    $original_url = $url;

    // Decode anything already encoded
    $url = urldecode($url);

    // Parse the URL into $urlparts
    $urlparts       = parse_url($url);

    // Build $params with each name/value pair
    foreach (split('&', $urlparts['query']) as $part) {
        if (strpos($part, '=')) {
            list($name, $value) = split('=', $part, 2);
        } else {
            $name = $part;
            $value = '';
        }
        $params[$name] = $value;
    }

    // Include a timestamp if none was provided
    if (empty($params['Timestamp'])) {
        $params['Timestamp'] = gmdate('Y-m-d\TH:i:s\Z');
    }

    // Sort the array by key
    ksort($params);

    // Build the canonical query string
    $canonical       = '';
    foreach ($params as $key => $val) {
        $canonical  .= "$key=".rawurlencode(utf8_encode($val))."&";
    }
    // Remove the trailing ampersand
    $canonical       = preg_replace("/&$/", '', $canonical);

    // Some common replacements and ones that Amazon specifically mentions
    $canonical       = str_replace(array(' ', '+', ',', ';'), array('%20', '%20', urlencode(','), urlencode(':')), $canonical);

    // Build the sign
    $string_to_sign             = "GET\n{$urlparts['host']}\n{$urlparts['path']}\n$canonical";
    // Calculate our actual signature and base64 encode it
    $signature            = base64_encode(hash_hmac('sha256', $string_to_sign, $secret_key, true));

    // Finally re-build the URL with the proper string and include the Signature
    $url = "{$urlparts['scheme']}://{$urlparts['host']}{$urlparts['path']}?$canonical&Signature=".rawurlencode($signature);
    return $url;
}

To use it, just wrap your Amazon URL with the signAmazonUrl() function and pass it your original string and secret key as arguments. As an example:

$xml = file_get_contents('https://webservices.amazon.com/onca/xml?some-parameters');

becomes

$xml = file_get_contents(signAmazonUrl('https://webservices.amazon.com/onca/xml?some-parameters', $secret_key));

Like most all of the variations of this, it does require the hash functions be installed to use the hash_hmac() function. That function is generally available in PHP 5.1+. Older versions will need to install it with Pecl. I tried using a couple of versions that try to create the Hash in pure PHP code, but none worked and installing it via Pecl was pretty simple.

(Note that I’ve slightly revised this code a couple of times to fix small issues that have been noticed)