The 2-hour presentation is about half slides and half demo. Throughout the course of the presentation I set up a simple master-slave. Then I add a second slave. Taking it a step farther I set up the three servers to replicate in a chain, and finally I configure them to replicate in a full circle so that changes made on one are propagated to all of the others. I intentionally do things that break replication at certain points to show some of the limitations and configurable features that can help it to work.

Slides for the presentation are available OpenOffice format.

The presentation was recorded, so hopefully the SELF team will have those videos available shortly.

When I develop a site though, I typically have a header and footer, and then all of the content pages exist as PHP files that simply include that header and footer. Converting a static site like that into the CMS takes a bunch of copy/pasting. I always try to avoid such tedious jobs, and so developed a script that will import those static pages into a GetSimple installation.

To run this script, I wanted to import a bunch of files in a ’static’ directory where I had moved all of the static files to. I then ran this from the command line to import all of the content into GetSimple

# for file in `find static -type f`
> do
> ./getsimple_import_file.php $file
> done

The script is available as getsimple_import_file.php

It takes a little configuration before running it. It works by simulating the data that you would submit when creating the page through the web interface, so we have to fake the necessary session cookie. Uncomment the bit in the middle that will display your cookie and run the script once. You’ll need to copy your cookie name and value into the script before doing any actual imports.

Once you’ve done that, you will probably want to change the regular expression that attempts to grab the page title from your file. You may also want to manipulate how it figures the URL to use.

Feel free to post comments here if you found this useful, or made any changes you’d like to share with other users

DROP TABLE IF EXISTS `mytable`;
CREATE TABLE `mytable` (
  `somecol` varchar(10) NOT NULL default '',
   ... other columns ...
  PRIMARY KEY  (`somecol`),
  KEY `isbn10` (`somecol`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

The problem is that the developer has altered the table and re-importing the test data would undo those changes. Editing the text file is impractical because of its size (500 MB gzipped). So I came up with this workaround which just slightly alters the SQL using sed so that it doesn’t try to drop or recreate the table. It comments out the DROP TABLE line, and creates the new table in the test database instead of the real database.

zcat bigfile.sql.gz |sed "s/DROP/-- DROP/"|sed "s/CREATE TABLE /CREATE TABLE test./"|mysql databasename

 /bin/sleep/sleep   `/usr/bin/expr $RANDOM % 600`

Purists will note that it isn’t truly random. The maximum value for $RANDOM is 32767 which is not evenly divisible by most likely values, but it is close enough.

# yum install subversion mod_dav_svn mod_python trac

Create a directory for your repositories, and an initial repository for testing, and create your htpasswd file. Then create a trac environment and set it up.

# mkdir /home/svn/
# svnadmin create testrepo
# chown -R apache:apache /home/svn/*
# htpasswd -c  /home/svn/.htpasswd brandon

#mkdir /home/trac/
# trac-admin /home/trac/ initenv
    ... answer questions as appropriate ...
# chown apache:apache /home/trac/*
# htpasswd -c  /home/svn/.htpasswd brandon

Add this to your Apache configuration in the relevant place (I like to put it under an SSL VirtualHost)

    <Location /svn>
        DAV svn
        SVNParentPath /home/svn/
        #SVNListParentPath on
        # Authentication
        AuthType Basic
        AuthName "RoundSphere SVN Repository"
        AuthUserFile /home/svn/.htpasswd
        Order deny,allow
        Require valid-user
    </Location>
    <Location /trac>
        SetHandler mod_python
        PythonHandler trac.web.modpython_frontend
        PythonOption TracEnv /home/trac
        PythonOption TracUriRoot /trac
        # Authentication
        AuthType Basic
        AuthName “MyCompany Trac Environment"
        AuthUserFile /home/svn/.htpasswd
        Require valid-user
    </Location>

Now test to make sure that you can view your test repository in a browser and that it prompts for a username and password as desired:

https://your-hostname/svn/testrepo/

You should retrieve a plain looking page that mentions the name of your repository and that it is at Revision 0

You should also be able to access your trac installation at

https://your-hostname/trac/

Customize your logo, change the home page, start making some tickets, using the wiki and get to work.

[root@ci /]# pear install phpunit/PHPUnit
Did not download optional dependencies: pear/Image_GraphViz, pear/Log, channel://pear.symfony-project.com/YAML, use --alldeps to download automatically
phpunit/PHPUnit requires PEAR Installer (version >= 1.8.1), installed version is 1.4.9
phpunit/PHPUnit can optionally use package "pear/Image_GraphViz" (version >= 1.2.1)
phpunit/PHPUnit can optionally use package "pear/Log"
phpunit/PHPUnit can optionally use package "channel://pear.symfony-project.com/YAML" (version >= 1.0.2)
phpunit/PHPUnit can optionally use PHP extension "json"
phpunit/PHPUnit can optionally use PHP extension "xdebug" (version >= 2.0.5)
No valid packages found
install failed

The trick is to install the PEAR package with –force to make it go through

[root@ci /]# pear upgrade --force PEAR
warning: pear/PEAR dependency package "pear/Archive_Tar" installed version 1.3.6 is not the recommended version 1.3.3
warning: pear/Archive_Tar requires PEAR Installer (version >= 1.5.4), installed version is 1.4.9
downloading PEAR-1.9.0.tgz ...
Starting to download PEAR-1.9.0.tgz (291,634 bytes)
.............................................................done: 291,634 bytes
downloading Archive_Tar-1.3.6.tgz ...
Starting to download Archive_Tar-1.3.6.tgz (17,600 bytes)
...done: 17,600 bytes
upgrade ok: channel://pear.php.net/Archive_Tar-1.3.6
upgrade ok: channel://pear.php.net/PEAR-1.9.0
PEAR: Optional feature webinstaller available (PEAR's web-based installer)
PEAR: Optional feature gtkinstaller available (PEAR's PHP-GTK-based installer)
PEAR: Optional feature gtk2installer available (PEAR's PHP-GTK2-based installer)
To install use "pear install pear/PEAR#featurename"

From there, you can continue on the the PHPUnit Install

However, installing it on a modern 64-Bit Ubuntu 9.10 install took just a bit of manipulation to get it to work. Pandora provides some basic instructions for Linux users here, even though Linux is officially unsupported. Those instructions, along with the Adobe AIR notes here provided enough information for me to get it installed and working.

Here’s what I did:

• Start out at the Pandora One site
• Click on the "Download Pandora Desktop" link and save that file to /tmp
• Follow the link to Install Adobe Air and save that file to /tmp also
• Open a shell, and chmod the Adobe Air installer to 755 and then run it.
• Go through the Adobe AIR install until it completes
• Once Adobe AIR is installed, you will need to put some 32-bit libraries in place to make it run correctly. Some of the steps on Adobe’s site work, and some don’t, so this is what I did
• Download the two .deb files for libnss3 and Libnspr4 to /tmp
• From your shell, run:

   sudo file-roller ./libnss3-1d_3.12.0~beta3-0ubuntu1_i386.deb
  

• Navigate to data.tar.gz => /usr => lib. Click on all of the files in that directory and click Extract. Type in /usr/lib32/ so that they extract there, then close all of the file-roller windows.
• Do the same thing with the libnspr4 .deb file that you downloaded
• Copy the adobe cert store into place with this command:

   sudo cp /usr/lib/libadobecertstore.so /usr/lib32
  

• Now you can finally install the Pandora application by running:

  sudo Adobe\ AIR\ Application\ Installer /tmp/pandora_2_0_2.air
  

  That should install the application correctly. It will add an icon to Applications / Accessories.

• Upon starting up the Pandora One client, it currently complains about connecting to an untrusted server for me. I have to click to accept for this session each time

Now you should be able to play your Pandora music from your 64-bit Ubuntu 9.10 box.

[root@host ~]# cat /proc/mdstat
md1 : active raid1 sdb3[1] sda3[2]
      974559040 blocks [2/1] [_U]
      [>....................]  recovery =  3.9% (38109184/974559040) finish=7399.1min speed=2108K/sec

I did some read and write tests directly to the drives using dd to make sure that they were working okay, and they can operate at about 100 MB/s

[root@host ~]# dd if=/dev/zero of=/dev/sda2 bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 10.8882 seconds, 96.3 MB/s
[root@host ~]# dd if=/dev/zero of=/dev/sdb2 bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 11.1162 seconds, 94.3 MB/s
[root@host ~]# dd if=/dev/sda2 of=/dev/null bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 10.2829 seconds, 102 MB/s
[root@host ~]# dd if=/dev/sdb2 of=/dev/null bs=1024 count=1024000
    1048576000 bytes (1.0 GB) copied, 10.5109 seconds, 99.8 MB/s

What I failed to realize is that there is a configurable limit for the min and max speed of the rebuild. Those parameters are configured in /proc/sys/dev/raid/speed_limit_min and /proc/sys/dev/raid/speed_limit_max. They default to a pretty slow 1MB/s minimum which was causing it to take forever.

Increasing the maximum limit didn’t automatically make it faster either. I had to increase the minimum limit to get it to jump up to a respectable speed.

[root@host ~]# echo 100000 > /proc/sys/dev/raid/speed_limit_min

[root@host ~]# watch cat /proc/mdstat
Every 2.0s: cat /proc/mdstat
md1 : active raid1 sdb3[1] sda3[2]
      974559040 blocks [2/1] [_U]
      [=>...................]  recovery =  7.7% (75695808/974559040) finish=170.5min speed=87854K/sec

Now it is up around 87 MB/s and will take just a few hours to complete the rest of the drive.

I wrote this quick database wrapper class that will pass all SELECT queries on to a local replica of the database, and silently discard any updates. On this site almost all of the functionality still works, but it obviously isn’t saving and new information while the primary database is unavailable.

Here is my class. This is intended as a wrapper to an ADOdb class, but it is generic enough that I think it would work for many other database abstraction functions as well.

class db_unavailable {
    var $readonly_db;

    function __construct($readonly_db)
    {
        $this->query_db = $readonly_db;
    }

    function query($sql)
    {
        $args = func_get_args();
        if (preg_match("#(INSERT INTO|REPLACE INTO|UPDATE|DELETE)#i", $args[0])) {
            // echo "Unable to do insert/replace/update/delete query: $sql\n";
            return true;
        } else {
            return call_user_func_array(array($this->readonly_db, 'query'), $args);
        }
    }

    function __call($function, $args)
    {
        return call_user_func_array(array($this->readonly_db, $function), $args);
    }
}

I simply create my $query_db object that points to the read-only database. Then create my main $db object as a new db_unavailable() object. Any select queries against $db will behave as they normally do, and data-modifying queries will be silently discarded.

It started out with a simple table for kids who were either naughty or nice. We then added some reports to that. Then imported kids’ wish lists from CSV files. From there we were able to generate some manufacturing reports for the workshop.

When we joined the wish list table with the kids table, we were then able to generate a sleigh-loading report which included only gifts for kids who had been good. Then we got even more complicated and introduced several joins with some complicated mathematics to select gifts for kids within a certain radius from a given zip code.

The presentation is available for download here. And Brian recorded part of the presentation which is available to view on uStream.tv or here. (We’re still experimenting with getting the video recording set up correctly)