Author: Brandon

After struggling with this problem in my mind for a while, I finally had the opportunity to experiment with Cloud Init and come up with a working solution for securely (I think) deploying code and credentials to a stock Ubuntu Instance on EC2.

My primary goals are:

• Must use an stock AMI with no customization
• Human readable user-data that contains appName, environment, and role.
• user-data must be easily modified by a developer for their own app or environment
  (No forcing them to base64 encode, gzip, or use special tools)
• Must be portable between providers.
  The example works with EC2, but initial ‘include’ file can be customized for each provider or OS.

The diagram below shows how this is to be accomplished

I’ve successfully deployed several instances using this method and it seems to work well. Getting the cloud init include file, and the script

I spent the last several hours troubleshooting any annoying error with the mongodb init script on Ubuntu. The script would start the daemon easy enough, but would report a failure. When subsequently trying to stop the daemon, it would then then say it was successful, but in-fact, still be running.

Fortunately, I found that somebody else had already reported the bug, but the comments pointing developers to the wrong place – looking inside the mongod code instead of in the init script.

After digging in to it for longer than I’d like to have spent, I found the cause was the –make-pidfile option being used in the init script.

My understanding of this process is that the start-stop-daemon command was creating the pidfile (as root), before spawning the actual mongod process (as the mongodb user). mongod in some cases (at least when not configsvr=true) must fork again before saving its own pidfile. Since the file created by start-stop-daemon is being run as root, the less-privileged mongodb user can not overwrite it (perhaps this should be logged, or logged at a less verbose level?), leaving the pidfile containing a pid that is no longer correct.

On my machine, the pidfile created with the –make-pidfile options was consistently exactly three less than the PID shown in the output of ‘ps’

After making that change to the init file, I can now reliably start/stop the mongod process using the expected commands.

Hopefully that bug will be closed soon and released so that I don’t have to customize the init script on every mongo server I have.

Instructions for Getting this working on Ubuntu 12.04 and more modern systems than my previous post

Install the required system and pecl packages:

  # apt-get install gnupg  libgpgme11 libgpgme11-dev
  # pecl install gnupg
  # echo extension=gnupg.so > /etc/php5/conf.d/gnupg.ini
  # apache2ctl restart

Generate a Private key

 # gpg --homedir /path/to/your/directory --gen-key

On a virtual machine, if that stalls for a while, you may have to generate some “randomness” somehow. Try one of these commands in a separate session, according to this bug report:

 # find / -type f | xargs grep blahblahblha
 # tcpdump -i any > /dev/null

At this point, you should have a working GPG key in the home directory you specified. You can list your secret keys with the command:

 
  # gpg --homedir /path/to/your/directory -K

You’ll then want to export the key with the command:

 # gpg --homedir /path/to/your/directory --export-secret-key --armour

You’ll want to copy that secret key to another machine. DON’T LOSE IT or you won’t be able to decrypt anything. Once you’ve got it safely stored somewhere, you want to delete it from your web server:

 #  gpg --homedir /path/to/your/directory --delete-secret-key your@address.com

You can then make sure that the public key is still there. It is what you’ll need to encrypt messages:

 # gpg --homedir /path/to/your/directory -k

Finally, you’ll need the fingerprint for the key to refer to it within your PHP code.

 # gpg --homedir /path/to/your/directory --fingerprint 
pubring.gpg
-------------
pub   2048R/5BB54E26 2013-04-14 [expires: 2023-04-12]
      Key fingerprint = AAAA BBBB CCCC DDDD EEEE  FFFF 0000 1111 2222 3333
uid                  Your Name <your@address.com>
sub   2048R/2EF4937A 2013-04-14 [expires: 2023-04-12]

You can then use the gnupg pecl functions to encrypt a messages:

<?php
$CONFIG['gnupg_home'] = '/var/www/.gnupg';
$CONFIG['gnupg_fingerprint'] = 'FA451EE9877270EF1CFA99CE048A613921CCC3D6';

$data = 'this is some confidential information';

$gpg = new gnupg();
putenv("GNUPGHOME={$CONFIG['gnupg_home']}");
$gpg->seterrormode(GNUPG_ERROR_SILENT);
$gpg->addencryptkey($CONFIG['gnupg_fingerprint']);
$encrypted =  $this->gpg->encrypt($data);
echo "Encrypted text: \n$encrypted\n";

// Now you can store $encrypted somewhere.. perhaps in a MySQL text or blob field.

// Then use something like this to decrypt the data.
$passphrase = 'Your_secret_passphrase';
$gpg->adddecryptkey($CONFIG['gnugp_fingerprint'], $passphrase);
$decrypted = $gpg->decrypt($encrypted);

echo "Decrypted text: $decrypted";
?>

I just recently fixed an issue I wanted my Monit monitoring process to restart a daemon who was segfaulting and causing 100% CPU usage according to top and most other system tools. I had seen configuration examples where Monit could detect that and restart the process, so I figured that adding a configuration like that below would fix it easily enough:

check process foo with pidfile /var/run/foo.pid
  start program = "/etc/init.d/foo start" timeout 10 seconds
  stop program  = "/etc/init.d/foo stop"
  if cpu usage > 90% for 8 cycles then restart

After letting that run for a bunch of cycles the process remained running, and monit didn’t do anything to acknowledge it even in log files. (FYI, a “cycle” is defined in the Monitrc config file in the “set daemon” line and defaults to 120 seconds).

After some research, I finally came upon this post on the Monit mailing list where somebody describes that the CPU usage that Monit bases its numbers off is a percentage of the CPU available for all processors. My machine had 4 processors, so what was seeing as 100% CPU usage in top, monit would only see that as 25%.

I quickly changed my Monit config to check for CPU Usage > 22% as ween in the following. That now works perfectly, even acknowledging in the log each of the 8 times that the CPU was over the limit before restarting it:

check process foo with pidfile /var/run/foo.pid
  start program = "/etc/init.d/foo start" timeout 10 seconds
  stop program  = "/etc/init.d/foo stop"
  if cpu usage > 22% for 8 cycles then restart

…. Now I need to solve the real problem and see why the latest Mongo PHP pecl module is segfaulting….