APISigning Now Works with Amazon Simple Email Service (SES)

Posted on April 26th, 2011 in Encryption,General,Programming,Websites by Brandon

APISigning.com has been signing Amazon Product Advertising requests for a couple of years now. Amazon recently announced their Simple Email Service that makes it easy to send emails via an API. The SES API requires that requests be authenticated using some cryptographic functions that are not easily available on all platforms or programming languages. In those cases, developers can use the APISigning SES Service to calculate the correct signature and perform the request on their behalf.

APISigning has free accounts that effectively allows 10k signing requests each month. Users who require additional requests can subscribe to a paid account with higher limits.

KnitMeter.com Has Been Upgraded

Posted on March 24th, 2011 in PHP,Programming,Websites by Brandon

KnitMeter Logo

KnitMeter Has Been Upgraded

KnitMeter.com was originally started over four years ago in December of 2007 as a small project that my wife thought would be useful. Since then, the site hasn’t changed much, but it has managed to grow to thousands of users who have knit nearly 20 thousand miles of yarn. I’ve received numerous requests and have finally gotten a chance to impliment what many of you have been requesting for a while now. New features on the site include:

  • Users can now add entries for knitting, crocheting, and spinning
  • Completely new and modernized design and logo
  • You can customize your widgets directly on KnitMeter.com rather than editing the code for the widget on your website
  • The website and the KnitMeter Facebook Application are now completely integrated. Entries added in one will be displayed and counted in the other
  • The Facebook application can (again) publish your entries to your news feed, but only when you tell it to
  • You can chose to make your profile public, which will display some of the most recent entries on the KnitMeter home page with a link to your website
  • Added several new timeframes, including specific calendar years (ie: I knit 4.3 miles in 2010)
  • Numerous technical changes that should make the site faster to use and make it easier to make future changes

These new features have been rolled out over the past couple of weeks. I appreciate the patience of those who have dealt with a few bugs over that time, and I believe that everything should be pretty bug-free now. I encourage you to check out the new site and to start adding up the mileage for your own projects. The next major milestone will be when we have gone through enough yarn to go around the earth (about 24,901 miles). At the present rate, we should hit that figure in about 3-5 months.

Happy Knitting, Crocheting, and Spinning,
Brandon Checketts
KnitMeter.com

Website Performance: Tables Versus CSS

Posted on October 16th, 2010 in General,PHP,Programming,Website Performance,Websites by Brandon

Most website designers have been using CSS for page layout for several years now, but I occasionally see some websites that continue to use HTML tables for layout. As I’ve been focusing on website performance lately, I’ve found some references that modern browsers render sites using tables for layout slower than they do sites that use CSS. I decided to investigate and confirmed that there are many possible situations where sites using large tables will appear to load much slower than those using CSS. I put together two pages to confirm:

This page uses <div> elements for layout
and
This pages uses a large table for layout

On both pages I’ve added a 5-second sleep near the end of the page to show what might happen if the server was slow, if there were network problems, or any other number of things may have happened.

Notice that the page created using a table changes a lot after the delay. I’ve tried it in Firefox 3 which extends the main (yellow) content section all of the way to the right until it receives the rest of the document, at which point it has to shrink that part to make room for the section on the right. Internet Explorer behaves even worse. It leaves a blank white page until after the delay, at which point it draws the whole table.

By contrast, the page created with CSS positioning shows all of the content above the delay and has it in the correct position. When the rest of the document is sent it just fills in the appropriate content, but doesn’t have to re-arrange anything on the page.

Enabling HTTP Page Caching with PHP

Posted on April 29th, 2010 in PHP,Programming,Websites by Brandon

I’ve been doing a lot of work on BookScouter.com lately to reduce page load time and generally increase the performance of the website for both users and bots. One of the tips that the load time analyzer points out is to enable an expiration time for static content. That is easy enough for images and such by using an Apache directive such as:

    ExpiresActive On
    ExpiresByType image/gif A2592000
    ExpiresByType image/jpg A2592000
    ExpiresByType image/png A2592000

But pages generated with PHP by default have the Pragma: no-cache header set, so that the users’ browsers do not cache the content at all. In most cases, even hitting the back button will generate another request to the server which must be completely processed by the script. You may be able to cache some of the most intensive operations inside your script, but this solution will eliminate that request completely. Simply add this code to the top of any page that contains semi-static content. It effectively sets the page expiration time to one hour in the future. So if a visitor hits the same URL within that hour, the page is served locally from their browser cache instead of making a trip to the server. It also sends an HTTP 304 (Not Modified) response code if the user requests to reload the page within the specified time. That may or may-not be desired based on your site.

$expire_time = 60*60; // One Hour
header('Expires: '.gmdate('D, d M Y H:i:s \G\M\T', time() + $expire_time));
header("Cache-Control: max-age={$expire_time}");
header('Last-Modified: '.gmdate('D, d M Y H:i:s \G\M\T', time()));
header('Pragma: public');

if ((!empty($_SERVER['HTTP_IF_MODIFIED_SINCE'])) && (time() - strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']) < = $expire_time)) {
    header('HTTP/1.1 304 Not Modified');
    exit;
}

Skipping the DROP TABLE, CREATE TABLE statements in a large mysqldump file.

Posted on April 28th, 2010 in General,Linux System Administration,MySQL,Programming by Brandon

I have a large table of test data that I’m copying into some development environments. I exported the table with a mysqldump which has a DROP TABLE and CREATE TABLE statements at the top

DROP TABLE IF EXISTS `mytable`;
CREATE TABLE `mytable` (
  `somecol` varchar(10) NOT NULL default '',
   ... other columns ...
  PRIMARY KEY  (`somecol`),
  KEY `isbn10` (`somecol`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

The problem is that the developer has altered the table and re-importing the test data would undo those changes. Editing the text file is impractical because of its size (500 MB gzipped). So I came up with this workaround which just slightly alters the SQL using sed so that it doesn’t try to drop or recreate the table. It comments out the DROP TABLE line, and creates the new table in the test database instead of the real database.

zcat bigfile.sql.gz |sed "s/DROP/-- DROP/"|sed "s/CREATE TABLE /CREATE TABLE test./"|mysql databasename

Installing SVN and Trac on a CentOS 5 server

Posted on March 19th, 2010 in General,Linux System Administration,Programming by Brandon

Make sure that you have the RPMForge repository enabled. Install Subversion, mod_dav_svn, and trac. This will install a few required dependencies (ie: neon and some python utils)

# yum install subversion mod_dav_svn mod_python trac

Create a directory for your repositories, and an initial repository for testing, and create your htpasswd file. Then create a trac environment and set it up.

# mkdir /home/svn/
# svnadmin create testrepo
# chown -R apache:apache /home/svn/*
# htpasswd -c  /home/svn/.htpasswd brandon

#mkdir /home/trac/
# trac-admin /home/trac/ initenv
    ... answer questions as appropriate ...
# chown apache:apache /home/trac/*
# htpasswd -c  /home/svn/.htpasswd brandon

Add this to your Apache configuration in the relevant place (I like to put it under an SSL VirtualHost)

    <Location /svn>
        DAV svn
        SVNParentPath /home/svn/
        #SVNListParentPath on
        # Authentication
        AuthType Basic
        AuthName "RoundSphere SVN Repository"
        AuthUserFile /home/svn/.htpasswd
        Order deny,allow
        Require valid-user
    </Location>
    <Location /trac>
        SetHandler mod_python
        PythonHandler trac.web.modpython_frontend
        PythonOption TracEnv /home/trac
        PythonOption TracUriRoot /trac
        # Authentication
        AuthType Basic
        AuthName “MyCompany Trac Environment"
        AuthUserFile /home/svn/.htpasswd
        Require valid-user
    </Location>

Now test to make sure that you can view your test repository in a browser and that it prompts for a username and password as desired:

https://your-hostname/svn/testrepo/

You should retrieve a plain looking page that mentions the name of your repository and that it is at Revision 0

You should also be able to access your trac installation at

https://your-hostname/trac/

Customize your logo, change the home page, start making some tickets, using the wiki and get to work.

PHP Wrapper Class for a Read-only database

Posted on January 4th, 2010 in General,Linux System Administration,MySQL,PHP,Programming by Brandon

This is a pretty special case of a database wrapper class where I wanted to discard any updates to the database, but want SELECT queries to run against an alternative read-only database. In this instance, I have a planned outage of a primary database server, but would like the public-facing websites and web services to remain as accessible as possible.

I wrote this quick database wrapper class that will pass all SELECT queries on to a local replica of the database, and silently discard any updates. On this site almost all of the functionality still works, but it obviously isn’t saving and new information while the primary database is unavailable.

Here is my class. This is intended as a wrapper to an ADOdb class, but it is generic enough that I think it would work for many other database abstraction functions as well.

class db_unavailable {
    var $readonly_db;

    function __construct($readonly_db)
    {
        $this->query_db = $readonly_db;
    }

    function query($sql)
    {
        $args = func_get_args();
        if (preg_match("#(INSERT INTO|REPLACE INTO|UPDATE|DELETE)#i", $args[0])) {
            // echo "Unable to do insert/replace/update/delete query: $sql\n";
            return true;
        } else {
            return call_user_func_array(array($this->readonly_db, 'query'), $args);
        }
    }

    function __call($function, $args)
    {
        return call_user_func_array(array($this->readonly_db, $function), $args);
    }
}

I simply create my $query_db object that points to the read-only database. Then create my main $db object as a new db_unavailable() object. Any select queries against $db will behave as they normally do, and data-modifying queries will be silently discarded.

LUG Presentation on SQL Basics

Posted on December 17th, 2009 in General,LUG,MySQL,PHP,Programming by Brandon

I gave a presentation tonight at my local Linux Users Group meeting on SQL Basics. I had a fun time preparing the presentation and made up a bunch of examples having to do with Santa’s database.

It started out with a simple table for kids who were either naughty or nice. We then added some reports to that. Then imported kids’ wish lists from CSV files. From there we were able to generate some manufacturing reports for the workshop.

When we joined the wish list table with the kids table, we were then able to generate a sleigh-loading report which included only gifts for kids who had been good. Then we got even more complicated and introduced several joins with some complicated mathematics to select gifts for kids within a certain radius from a given zip code.

The presentation is available for download here. And Brian recorded part of the presentation which is available to view on uStream.tv or here. (We’re still experimenting with getting the video recording set up correctly)

PHP Code to Sign any Amazon API Requests

Posted on June 30th, 2009 in General,Linux System Administration,PHP,Programming by Brandon

Starting next month, any requests to the Amazon Product Advertising API need to be cryptographically signed. Amazon has given about three months notice and the deadline is quickly approaching. I use the Amazon web services on several sites and came up a fairly generic way to convert an existing URL to a signed URL. I’ve tested with several sites and a variety of functions, and this is working well for me so far:

function signAmazonUrl($url, $secret_key)
{
    $original_url = $url;

    // Decode anything already encoded
    $url = urldecode($url);

    // Parse the URL into $urlparts
    $urlparts       = parse_url($url);

    // Build $params with each name/value pair
    foreach (split('&', $urlparts['query']) as $part) {
        if (strpos($part, '=')) {
            list($name, $value) = split('=', $part, 2);
        } else {
            $name = $part;
            $value = '';
        }
        $params[$name] = $value;
    }

    // Include a timestamp if none was provided
    if (empty($params['Timestamp'])) {
        $params['Timestamp'] = gmdate('Y-m-d\TH:i:s\Z');
    }

    // Sort the array by key
    ksort($params);

    // Build the canonical query string
    $canonical       = '';
    foreach ($params as $key => $val) {
        $canonical  .= "$key=".rawurlencode(utf8_encode($val))."&";
    }
    // Remove the trailing ampersand
    $canonical       = preg_replace("/&$/", '', $canonical);

    // Some common replacements and ones that Amazon specifically mentions
    $canonical       = str_replace(array(' ', '+', ',', ';'), array('%20', '%20', urlencode(','), urlencode(':')), $canonical);

    // Build the sign
    $string_to_sign             = "GET\n{$urlparts['host']}\n{$urlparts['path']}\n$canonical";
    // Calculate our actual signature and base64 encode it
    $signature            = base64_encode(hash_hmac('sha256', $string_to_sign, $secret_key, true));

    // Finally re-build the URL with the proper string and include the Signature
    $url = "{$urlparts['scheme']}://{$urlparts['host']}{$urlparts['path']}?$canonical&Signature=".rawurlencode($signature);
    return $url;
}

To use it, just wrap your Amazon URL with the signAmazonUrl() function and pass it your original string and secret key as arguments. As an example:

$xml = file_get_contents('http://webservices.amazon.com/onca/xml?some-parameters');

becomes

$xml = file_get_contents(signAmazonUrl('http://webservices.amazon.com/onca/xml?some-parameters', $secret_key));

Like most all of the variations of this, it does require the hash functions be installed to use the hash_hmac() function. That function is generally available in PHP 5.1+. Older versions will need to install it with Pecl. I tried using a couple of versions that try to create the Hash in pure PHP code, but none worked and installing it via Pecl was pretty simple.

(Note that I’ve slightly revised this code a couple of times to fix small issues that have been noticed)

Array versus String in CURLOPT_POSTFIELDS

Posted on May 29th, 2009 in General,PHP,Programming by Brandon

The PHP Curl Documentation for CURLOPT_POSTFIELDS makes this note:

This can either be passed as a urlencoded string like ‘para1=val1&para2=val2&…’ or as an array with the field name as key and field data as value. If value is an array, the Content-Type header will be set to multipart/form-data.

I’ve always discounted the importance of that, and in most cases it doesn’t generally matter. The destination server and application likely know how to deal with both multipart/form-data and application/x-www-form-urlencoded equally well. However, the data is passed in a much different way using these two different mechanisms.

application/x-www-form-urlencoded

application/x-www-form-urlencoded is what I generally think of when doing POST requests. It is the default when you submit most forms on the web. It works by appending a blank line and then your urlencoded data to the end of the POST request. It also sets the Content-Length header to the length of your data. A request submitted with application/x-www-form-urlencoded looks like this (somewhat simplified):

POST /some-form.php HTTP/1.1
Host: www.brandonchecketts.com
Content-Length: 23
Content-Type: application/x-www-form-urlencoded

name=value&name2=value2

multipart/form-data

multipart/form-data is much more complicated, but more flexible. Its flexibility is required when uploading files. It works in a manner similar to MIME types. The HTTP Request looks like this (simpified):

POST / HTTP/1.1
Host: www.brandonchecketts.com
Content-Length: 244
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------26bea3301273

And then subsequent packets are sent containing the actual data. In my simple case with two name/value pairs, it looks like this:

HTTP/1.1 100 Continue
------------------------------26bea3301273
Content-Disposition: form-data; name="name"

value
------------------------------26bea3301273
Content-Disposition: form-data; name="name2"

value2
------------------------------26bea3301273--

CURL usage

So, when sending POST requests in PHP/cURL, it is important to urlencode it as a string first.

This will generate the multipart/form-data version

$data = array('name' => 'value', 'name2' => 'value2');
curl_setopt($curl_object, CURLOPT_POSTFIELDS,  $data)

And this simple change will ensure that it uses the application/x-www-form-urlencoded version:

$data = array('name' => 'value', 'name2' => 'value2');
$encoded = '';
foreach($data as $name => $value){
    $encoded .= urlencode($name).'='.urlencode($value).'&';
}
// chop off the last ampersand
$encoded = substr($encoded, 0, strlen($encoded)-1);
curl_setopt($curl_object, CURLOPT_POSTFIELDS,  $encoded)
Next Page »