Synchronize Remote Memcached Clusters with memcache_sync

Posted on May 12th, 2009 in General,Linux System Administration,MySQL,Programming by Brandon

The problem: Servers in two separate geographic locations each have their own memcached cluster. However, there doesn’t currently exist (that I know of) a good way to copy data from one cluster to the other cluster.

One possible solution is to configure the application to perform all write operations in both places. However, each operation requires a round-trip response. If the servers are separated by 50ms or more, doing several write operations causes a noticable delay.

The solution that I’ve come up with is a perl program that I’m calling memcache_sync. It acts a bit like a proxy that asynchronously performs write operations on a remote cluster. Each geographic location runs an instance of memcache_sync that emulates a memcached server. You configure your application to write to the local memcache cluster, and also to the memcache_sync instance. memcache_sync queues the request and immediately returns a SUCCESS message so that your application can continue doing its thing. A separate thread then writes those queued operations to the remote cluster.

The result is two memcache clusters that are synchronized in near-real time, without any noticable delay in the application.

I’ve implemented ‘set’ and ‘delete’ operations thus far, since that is all that my application uses. I’ve just started using this on a production environment and am watching to see how it holds up. So far, it is behaving well.

The script is available here. I’m interested to see how much need there is for such a program. I’d be happy to have input from others and in developing this into a more robust solution that works outside of my somewhat limited environment.

3 Responses to 'Synchronize Remote Memcached Clusters with memcache_sync'

Subscribe to comments with RSS or TrackBack to 'Synchronize Remote Memcached Clusters with memcache_sync'.

  1. Prashant said,

    on August 31st, 2009 at 1:38 am

    Hi Brandon,
    Even our organization is having similar situation where we need to sync between 2 separate memcached servers remotely.
    Your script looks promising, will try and share my thoughts.
    Thanks for putting it up.

    Cheers!


  2. on February 1st, 2011 at 5:25 am

    Hi Brandon,

    This script is definitely helpful. We are planning to have memcache servers at two different geographical locations (to provide the fastest response time to users at both the locations). Do you have the newer version of the script which implements all the memcache operations?

    Thanks,
    Rajiv


  3. on March 8th, 2012 at 5:35 pm

    Also curious if you’ve updated this script. It is exactly what I was about to write but you’ve done a vastly better job than I would have done :)

Post a comment

Please copy the string oaKkDi to the field below: