Boto3 Best Practices: Assert to Stop Silent Failures

Good morning!

Today’s post covers a pattern I use to increase my confidence that my infrastructure code is working. It turns silent errors into loud ones. I’ve handled plenty of code that runs without errors but still ends up doing the wrong thing, so I’m never really sure if it’s safe to go to sleep at night. I don’t like that. I want silence to be a real indicator that everything is fine. Like The Zen of Python says, “Errors should never pass silently.”

It’s easy to write assumptions that’ll create silent errors into boto code. Imagine you have an EBS volume called awesome-stuff and you need to snapshot it for backups. You might write something like this:

import datetime

import boto3

ec2 = boto3.resource('ec2')
volume_filters = [{'Name': 'tag:Name', 'Values': ['awesome-stuff']}]
volumes = list(ec2.volumes.filter(Filters=volume_filters))
volume = volumes[0]
now = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H-%M-%S%Z")
volume.create_snapshot(Description=f'awesome-stuff-backup-{now}')

Simple enough. We know our volume is named awesome-stuff, so we look up volumes with that name. There should only be one, so we snapshot the first item in that list. I’ve seen this pattern all over the boto code I’ve read.

What if there are two volumes called “awesome-stuff”? That could easily happen. Another admin makes a copy and tags it the same way. An unrelated project in the same account creates a volume with the same name because awesome-stuff isn’t super unique. It’s very possible to have two volumes with the same name, and you should assume it’ll happen. When it does, this script will run without errors. It will create a snapshot, too, but only of one volume. There is no luck in operations, so you can be 100% certain it will snapshot the wrong one. You will have zero backups but you won’t know it. 🔥

There’s an easy pattern to avoid this. First, let me show you Python’s assert statement:

awesome_list = ['a', 'b']
assert len(awesome_list) == 1

We’re telling Python we expect awesome_list to contain one item. If we run this, it errors:

Traceback (most recent call last):
  File "error.py", line 2, in <module>
    assert len(awesome_list) == 1
AssertionError

This is a sane message. Anyone reading it can see we expected there to be exactly one object in awesome_list but there wasn’t.

Back to boto. Let’s add an assert to our backup script:

import datetime

import boto3

ec2 = boto3.resource('ec2')
volume_filters = [{'Name': 'tag:Name', 'Values': ['awesome-stuff']}]
volumes = list(ec2.volumes.filter(Filters=volume_filters))
assert len(volumes) == 1
volume = volumes[0]
now = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H-%M-%S%Z")
volume.create_snapshot(Description=f'awesome-stuff-backup-{now}')

Now, if there are two awesome-stuff volumes, our script will error:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    assert len(volumes) == 1
AssertionError

Boom. That’s all you have to do. Now the script either does what we expect (backs up our awesome stuff) or it fails with a clear message. We know we don’t have any backups yet and we need to take action. Because we assert that there should be exactly one volume, this even covers us for the cases where that volume has been renamed or there’s a typo in our filters.

Here’s a good practice to follow in all of your code: If your code assumes something, assert that the assumption is true so you’ll get a clear, early failure message if it isn’t.

In general, these are called logic errors. Problems with the way the code thinks (its “logic”). Often they won’t even cause errors, they’ll just create behavior you didn’t expect and that might be harmful. Writing code that’s resilient to these kinds of flaws will take your infrastructure to the next level. It won’t just seem like it’s working, you’ll have confidence that it’s working.

Happy automating!

Adam

If this was helpful and you want to save time by getting “copy and paste” patterns for Python Cloud DevOps in your inbox, subscribe here. If you don’t want to wait for the next one, check out these: