Check your logs with CloudWatch

When I implemented a JEE Glassfish gas station price service for our Android / iOS SprIT app, I wanted to introduce some simple monitoring what is going on in my application.

When I implemented a JEE Glassfish gas station price service for our Android / iOS SprIT app, I wanted to introduce some simple monitoring what is going on in my application.

What would this mean to implement everything from scratch? On the one hand, another monitoring service checks periodically SprIT regarding its state. On the other hand, the SprIT service should transfer messages to my monitoring component as well, e.g. in case of an error.

Ever since this day and age of cloud computing, it’s a waste of money to implement basic monitoring yourself.

I wanted to introduce an out-of-the-box monitoring tool based on my application logs. Fortunately, my service was already hosted on an Amazon EC2 instance and thereby finding an appropriate tool seemed to be natural: CloudWatch 🙂

In this blog post, I gonna give you an introduction how to integrate log files in CloudWatch.

About CloudWatch

“Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources.“, aws.amazon.com.

CloudWatch can be used to apply a palette of tools to monitor applications and resources, for example to shut down unused EC2 instances. In the upcoming sections, I gonna explain how to wiretap log files.

Applying CloudWatch

Your AWS EC2 instance needs to be announced to CloudWatch. The following preconditions are required:

An IAM role must be linked to your EC2 instance.
The AmazonAPIGatewayPushToCloudWatchLogs policy must be added to the specified role.

First of all, ensure that an IAM role has been configured for your EC2 instance. Otherwise CloudWatch cannot be applied and another instance needs to be created.Beware: When you are creating a new instance, ensure that your wanted role is part of the EC2 service. If not, you cannot assign the role to an EC2 instance and the new instance is getting useless again 😉Last but not least, add the AmazonAPIGatewayPushToCloudWatchLogs policy!

awslogs

Establish an SSH connection to your EC2 instance to install the awslogs tools. In my particular case it’s an EC2 instance running on CentOS Linux.

$ ssh -i sn.pem ec2-user@my-cloud-watch-instance-ip-address
$ sudo yum install awslogs

1 2	$ ssh -i sn.pem ec2-user@my-cloud-watch-instance-ip-address $ sudo yum install awslogs

Configuring log groups

To look up into a favoured log file in the AWS CloudWatch web interface, it has to be configured via log groups and log streams in the CloudWatch logs agent configuration file located at /etc/awslogs/awslogs.conf on CentOS instances.

In my scenario, one log group symbolizes my application SprIT with several running application instances represented by log streams.

This is the basic AWS configuration for the /var/log/messages file.

[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/message

[/var/log/messages]

datetime_format = %b %d %H:%M:%S

file = /var/log/messages

buffer_duration = 5000

log_stream_name = {instance_id}

initial_position = start_of_file

log_group_name = /var/log/message

It’s up to you to create your required log groups pointing to your log files of interest, e.g. those of your JBoss, GlassFish, Tomcat etc.

After defining all your required log groups, the awslogs service has to be restarted.

$ sudo service awslogs restart
Starting awslogs:                                          [  OK  ]

1 2	$ sudo service awslogs restart Starting awslogs: [ OK ]

Now the log groups are listed in the AWS CloudWatch web interface.

Working with logs

By accessing log groups, we can enter log streams.

What does this mean?

My application has currently one log group where every application instance represents a log stream. They are named by there instance id (see log_stream_name = {instance_id} in the CloudWatch logs agent configuration file above).If there are several running instances of my application, I can easily distinguish them to check out logs independently or to apply a filter across all instances for issues.

It’s quite useful and straightforward to check out logs for salience.

What to do next?

Making your application logs visible in CloudWatch is just the tip of the iceberg. No DevOp is interested to check out log files manually 😉 Therefore, further use cases could be…