Category: Amazon APIs

Docker Syslog Container for Sending Logs to CloudWatch

AWS’s CloudWatch Logs was first available about a year ago, and to my estimation has gone largely unnoticed. The initial iteration was pretty rough, but some recent changes have made it more useful, including the ability to search logs, and generate events for monitoring in CloudWatch from log content.

Unfortunately, the Cloudwatch Logs agent just watches log files on disk and doesn’t act as a syslog server. An AWS blog post explained how to get the the Cloudwatch Logs Agent running inside a container and monitoring the log output from rsyslogd, but the instructions used Amazon’s ECS service, which still doesn’t quite offer the flexibility that CoreOS or Deis offer IMHO. ECS does some magic behind the scenes in passing credentials around that you have to do yourself when using CoreOS.

I’ve just provided a GitHub repository with the tools to make this work pretty easily, as well as a Docker Image with some reasonable defaults.

When trying to pull all of this together to work, I discovered a problem due to a bug in the overlayfs that is in current Deis Releases which causes the AWS Logs agent not to notice changes in the syslog files. A workaround is available that reformats the host OS back to btrfs to solve that particular problem

Note when running on Deis 561+ to revert to btrfs

Unattended install of Cloudwatch Logs Agent

So far, I’m pretty impressed with cloudwatch logs. The interface for it isn’t as fancy, and search capability isn’t as deep as other tools like PaperTrail or Loggly, but the cost is significantly less, and I like the fact that you can store different log groups for different lengths of time.

I’m trying to get the cloudwatch logs agent to install as part of an automated script, and couldn’t find any easy instructions to do that, so here is how I got it working with a shell script on an Ubuntu 14.04 host

echo Creating cloudwatch config file in /root/awslogs.conf
cat <<EOF >/root/awslogs.conf
[general]
state_file = /var/awslogs/state/agent-state
## Your config file would have a lot more with the logs that you want to monitor and send to Cloudwatch
EOF

echo Creating aws credentials in /root/.aws/credentials
cat <<EOF > /root/.aws/credentials
[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_HERE
aws_secret_access_key = YOUR_AWS_SECRET_KEY_HERE
EOF

echo Downloading cloudwatch logs setup agent
cd /root
wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
echo running non-interactive cloudwatch-logs setup script
python ./awslogs-agent-setup.py --region us-west-2 --non-interactive --configfile=/root/awslogs.conf

Fix For Amazon API Error: Your request is missing required parameters. Required parameters include AssociateTag.

Users of Amazon’s Product Advertising API will begin seeing the following error as of about November 1st:

Your request is missing required parameters. Required parameters include AssociateTag.

This is due to changing requirements to the Product Advertising API. The AssociateTag parameter is now a required and validated field. You can see in the list of changes that it is the first thing mentioned.

These changes are made to keep the Product Advertising API in line with its purpose of driving affiliate sales through the Amazon.com website. The AssociateTag is a tag generated from your Amazon Associates account.

So far, it looks like the tag is not entirely validated to your account. I’ve had luck using fictitious tags such as aztag-20.