Automate AWS Infrastructure with Boto 3: AWS Health Checks

When I first joined a DevOps/SRE team, I realized there were a lot of simple AWS infrastructure changes that took up a large chunk of our engineering team’s time. I didn’t want to spend my valuable coding time on these manual, yet essential, tasks so I set out on a mission to automate them. Since I had wanted to build my Python scripting skills anyway, I discovered a way to solve two problems at once - using the software development kit Boto 3 to automate my simple, manual AWS tasks using Python.

For the second installment in this series I wanted to cover using Boto 3 and Python to automate AWS health checks for instances and their services and events. If you came from Part 1, then you already know how to import Boto 3 and create a client to use in your script. While we created an EC2 client in the last script, we want to create one for ECS here. Using ECS is going to give us the information we need about services and events, something that EC2 does not provide.

You can then use this client to perform any of the methods listed in the ECS service section of the Boto 3 documentation. The documentation is super helpful in listing all of the resources you can use, their various methods, and how exactly to configure these methods. Feel free to explore!

Picture this scenario. You have lots of ECS clusters in your environments. Each cluster has multiple services, each with their own events, tasks, and target groups. If a cluster is unhealthy, it will begin to drain. However, it can be hard to tell which clusters are draining. In the past, I’ve found the only way to see if the services are decreasing or not using is by constantly refreshing the AWS CLI. To avoid doing this, you can print out detailed event messages, which can be much more helpful in troubleshooting.

Let’s figure out how we can write a script that tells us which clusters are healthy, which are not, and what is going wrong.

You probably noticed that the clusters this gives us start with arn:aws:ecs followed by the region and a string of integers. Because we only want the name of the cluster, not this pattern the query returns, we use the split function to get the name after the /.

There are different ways to do this, but this is what I find easiest.

Look at the documentation. What method would we have to use in order to find the services for a cluster? If you look at the documentation, there is really only one method that would make sense - list_services. We can see that this method needs the name of the cluster in order to find the service, which is where we will use the clusterName variable we just found.

Good! We found the services using the list_services method and found the list of names by filtering that method even further. Now, what method can we use to help us find the events of each service?

Describe_services seems to be the only one that lists out events for each service. Typically, when scripting with Boto 3, if you use a list method, you will then want to use a describe method.

Good! We found the service description by using describe_services and specifying the cluster and services that we found earlier.

Oops! Looks like some of our services are empty. This means we need to add a statement so that if this is the case, the script still runs without any errors.

Problem fixed!

Now we want to use the service names we found to find important metrics like the desired count and the running count. We must loop through each service in the list of service names in order to find these metrics for each one.

This might be hard to read still… How can we make this so it’s clear to our user what is the desired count vs the running count?

That’s better! This way we are only displaying the service counts for instances that don’t have the correct amount of services running. We are also printing how many are running out of how many are SUPPOSED TO BE running.

Now we want to look at event messages. They give us helpful information to tell us the current state of the services and tasks in our environment. Event messages are something that may otherwise be hard to find and read. You’d have to go through each event one by one to see its state.

We want to easily be able to see if a service in our environment has failed, and what exactly the message is.

Earlier, we saw that the describe_service methods lists information about events. Now it’s time to save that information to an events variable! Let’s look at the json structure of the response and see how we can filter it so we only get the events.

This is going to find all the events and event information for every service.

We only really care about the event message, so let’s find that and print that for the user.

If we look at all of these event messages we see that every healthy service has a message of “has reached a steady state.” Since we only care about the services that are failing and DO NOT have a steady state, let’s filter these messages out.

I am using regex to search for a certain string in the event messages. If you use this, make sure you import it at the top of your script.

If the regex returns None then that means ‘has reached a steady state’ is probably in the message which, in that case, we don’t need to see. We also want to make it easier to read event messages that indicate something may have gone wrong.

There we go! That is a lot easier to read.

Now we can easily see what may be going wrong in our environments without searching through every cluster for service and event information.

***

Stay tuned for the next installment in our series where we’ll cover automating AWS Snapshots.