Credit Card Debt

Technical debt is like a new credit card, it often comes with a 0% introductory interest rate. In the short term tech debt can look like a win; you get the new feature on time, you automate a manual process, you patch the bug. Maybe the implementation wasn’t perfect, but dealing with a bit of funky code or living with a few bugs is better than missing the deadline.

That loan comes due right away, you have to live with what you wrote, but the interest comes later. In a month (or three or six) something will happen that magnifies the impact of that funkiness or those bugs. You’ll need an unrelated feature but because you monkey-patched in the config for the first feature you’ll be forced to rewrite the config system before you can start, adding days to your timeline. You’ll need to install a zero-day security patch but your runtime hack will force you to shut down before you can patch, causing an outage.

Like a credit card, tech debt is manageable. If you pay back the new card on the right schedule it can get you the new TV without making you miss a rent payment. If you clean up your runtime hack in the next couple weeks it’s unlikely that a zero-day patch will be released before you’re done. If you don’t pay it back or you take out too many new cards, you can end up like the guy who makes six figures but rents a basement because his credit cards cost him $3,000 every month. You’ll fall behind on new feature development because you can’t build anything without fixing three old hacks first.

Unlike a credit card, the introductory rates of tech debt are hard to predict. You don’t know how many months of freedom from interest you get, and they may expire at the worst times. That zero-day patch might come out the week after you push your funky code to prod and you’ll be stuck with an outage. You might gamble if you know you’ll still be within the month’s SLA, but if you’ve gambled on twenty things like that you’ve got great odds that the bill on several debts will blow up at a bad time.

Every win has to come with hard questions about the debt it sits on. How much of this implementation will we be forced to rewrite? Does this new feature really work or does it just mostly work but we haven’t looked deep enough to see the problems? Do the funky parts of this code overlap with upcoming work?

Loans can get you ahead, and are manageable if you’re careful, but if you win by taking out too many it won’t matter how far ahead they got you. You’ll fall behind when they get too heavy. You’ll be a six figure team living in a basement.

Coverage, Syntax, And Chef

In Why I Don’t Track Test Coverage I explained why I don’t think coverage measures the quality of my tests. There’s a counter argument, though: 100% coverage means every line of code runs when the tests run, so it’s impossible to break prod with a misplaced comma. I think this counter argument is wrong when you’re working in Chef (I think it’s wrong in Python too but I’ll cover that in a separate post).

When Chef runs it processes each recipe into a list of actions (the ‘compile’ phase) and then it does those actions on the system (the ‘converge’ phase). The compile phase will (usually) execute every line of every recipe that’s in the run_list or is included with include_recipe. Both ChefSpec and kitchen/Serverspec take Chef through its compile phase, so in simple cases a syntax error will make both fail before the system is touched.

There are three (anti-)patterns in Chef that I know of that can sneak changes to system state past the compiler even when there are syntax errors:

#1 Raw Ruby

Chef recipes are Ruby files. You can put any valid Ruby code in them. You could do this:

File.delete('/etc/my_app/config.ini')

Ruby deletes config.ini as soon as it hits this line, before the rest of the compile finishes. If there’s a syntax problem later in the code you’ll still get an error but you’ll already have edited the system. The fallout of incomplete Chef client runs can get ugly (more on that another time).

Imagine if the tests for a Jenkins cookbook deleted the Jenkins config file. Then, a side-effect like this could take down a build server that does the innocent task of running ChefSpecs (which are only supposed to simulate Chef’s actions). It’s also surprisingly easy to accidentally hide this from the tests using #2 or #3 from below, which can cause incomplete Chef runs in production.

If you have side-effects like this in your code, replace them with a Chef resource (file with the :delete action in this case), write a custom resource, extract them into a gem that’s run before Chef runs, etc. Chef shouldn’t touch the state of the system before its converge phase.

#2 Ruby Conditions

Foodcritic, the linter for Chef, warns you not to do this:

if node['foo'] == 'bar'
  service 'apache' do
    action :enable
  end
end

Their argument is that you should use Guards, the library’s built-in feature:

service 'apache' do
  action :enable
  only_if { node['foo'] == 'bar' }
end

That’s a great argument, but there’s one more: with a Ruby condition, the resource won’t be compiled unless node[‘foo’] == ‘bar’. That means that unless you have a test where this is set, the compiler will never touch this resource and a syntax error won’t make the tests fail.

If you follow foodcritic’s recommendation, conditional resources will always be compiled (but may not be converged) and syntax errors will fail early without you doing any work.

#3 Conditional Includes

These technically belong with the other Ruby conditions, but they’re extra-nasty so I’m dedicating a section to them.

If you do this:

if node['foo'] == 'bar'
  include_recipe 'my_cookbook::my_recipe'
end

The resources in my_recipe will only be compiled if foo is set to bar in the node object. This is like putting a Ruby condition around every resource in my_recipe.

It gets worse if your condition is processed in the converge phase. For example, you could do an include in a ruby_block:

block 'run_my_recipe' do
  if File.directory?('/etc/my_app')
    run_context.include_recipe 'my_cookbook::my_recipe'
  end
end

Even if /etc/my_app exists, my_recipe won’t be compiled until Chef enters the converge phase and reaches the run_my_recipe resource. I bet you that nobody reading your cookbook will realize that it changes Chef’s “compile then converge” order into “compile some stuff then converge some stuff then compile the rest then converge the rest”. This is likely to bite you. Plus, now you have to start looking at mocks to make sure the tests exercise all your recipes. My advice is to avoid this pattern. Maybe there’s some special situation I haven’t found, but the few cases of converge-time conditional includes that I’ve seen have been hacks.

Conditional includes are usually a symptom of using Chef for something it’s not designed for. Chef is designed to converge the host where it’s running to a specific state. Its resources do a great job of detecting if that state is already present and skipping their actions if it is. If you have lots of resources that aren’t safe to run multiple times and that Chef isn’t automatically skipping then you should take a step back and make sure Chef is the right tool. Your standard approach should be to include all the recipes that may need to run and write each recipe to guard its resources from running when they shouldn’t.

 

If you write your Chef cookbooks well then you get 100% syntax coverage for free if you’ve written even one test. You can focus on exercising the logic of your recipes. Leave it to the robots to catch those misplaced commas.

Thanks for reading!

Adam

Why I Don’t Track Test Coverage

Last year I went through my programming habits looking for things to improve. All my projects had coverage reports, and my coverage was 95-100%. Looking deeper, I found that developing that coverage had actually hurt my projects. I decided to turn off coverage reports, and a year later I have no reason to turn them back on. Here’s why:

#1 Coverage is distracting.

I littered my code with markers telling the coverage engine to skip hard-to-test lines like Python’s protection against import side effects:

if __name__ == "__main__": # pragma: no cover
    main()

In some projects I had a test_coverage.py test module just for the tests that tricked the argument handling code into running. Most of those tests barely did more than assert that core libraries worked.

I also went down rabbit trails trying to find a way to mock enough of boilerplate code like module loaders to get a few more lines to run. Those were often fiddly areas of the language and their rabbit trails were surprisingly long.

#2 Coverage earns undeserved confidence

While cleaning up old code written by somebody else I wrote a test suite to protect me from regressions. Its had 98% coverage. It didn’t protect me from anything. The code was full of stuff like this:

main.py

import dbs

def helper_function():
    tables = dbs.get_tables()
    # ... other db-related stuff.

dbs.py

DBS = ['db1.local', 'db2.local']
TABLES = list()
for db in DBS:
    # Bunch of code that generates table names.

This is terrible code, but I was stuck with it. One of its problems is that dbs.py is a side-effect; ‘import dbs’ causes the code in that module to execute. To write a simple test of helper_function I had to import from main.py, which caused an import of the dbs module, which ran all the lines in that module. A test of a five-line function took me from 0% coverage to over 50%.

When I hit 98% coverage I stopped writing tests, but I was still plagued by regressions during my refactors. The McCabe complexity of the code was over 12 and asserting the behavior buried in those lines needed two or three times the number of tests I’d written. Most tests would run the same lines over and over because of the import side-effects, but each test would work the code in a different way.

 

I considered revising my test coverage habits. Excluding broader sections of code from coverage reports so I didn’t have to mock out things like argument handling. Reducing my threshold of coverage from 95% to 75%. Treating legacy code as a special case and just turning off coverage there. But if I did all those things, the tests that were left were the tests I’d have written whether or not I was thinking about coverage.

Today, I don’t think about covering the code, I think about exercising it. I ask myself questions like these:

  • Is there anything I check by hand after running the tests? Write a test for that.
  • Will it work if it gets junk input? Use factories or mocks to create that input.
  • If I pass it the flag to simulate, does it touch the database? Write a test with a mocked DB connector to make sure it doesn’t.

Your tests shouldn’t be there to run lines of code. They should be there to assert that the code does what it’s supposed to do.

Happy Thanksgiving!

Adam

Python on Lambda: Better Packaging Practices

Update 2018-10-22: This is out of date! Since I wrote this, lambda released support for Python 3 and in the new version I don’t have to do the handler import described below (although I don’t know if that’s because of a difference in Python 3 or because of a change in how lambda imports modules). In a future post I’ll cover Python 3 lambda functions in more detail.

Lambda is an AWS service that runs your code for you, without you managing servers. It’s my new favorite tool, but the official docs encourage a code structure that I think is an anti-pattern in Python. Fortunately, after some fiddling I found what I think is a better way. Originally I presented it to the San Diego Python meetup (if you’re in Southern California you should come to the next one!), this post is a recap.

The Lambda Getting Started guide starts you off with simple code, like this from the Hello World “blueprint” (you can find the blueprints in the AWS web console):

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event['key1'])
    print("value2 = " + event['key2'])
    print("value3 = " + event['key3'])
    return event['key1']  # Echo back the first key value
    #raise Exception('Something went wrong')

This gets uploaded to Lambda as a flat file, then you set a “handler” (the function to run). They tell you the handler has this format:

python-file-name.handler-function

So for the Hello World blueprint we set the handler to this:

lambda_function.lambda_handler

Then Lambda knows to look for the lambda_handler function in the lambda_function file. For something as small as a hello world app this is fine, but it doesn’t scale.

For example, here’s a piece of the Alexa skill blueprint (I’ve taken out most of it to keep this short):

from __future__ import print_function

def build_speechlet_response(title, output, reprompt_text, should_end_session):
    ...human interface stuff

def build_response(session_attributes, speechlet_response):
    ...more human interface stuff

def get_welcome_response():
    ...more human interface stuff

def handle_session_end_request():
    ...session stuff

def create_favorite_color_attributes(favorite_color):
    ...helper function

def set_color_in_session(intent, session):
    ...the real logic of the tool part 1

def get_color_from_session(intent, session):
    ...the real logic of the tool part 2

etc...

This doesn’t belong in one long module, it belongs in a Python package. The session stuff goes in its own module, the human interface stuff in a different module, and probably a bunch of modules or packages for the logic of the skill itself. Something like the pypa sample project:

sampleproject/
├── LICENSE.txt
├── MANIFEST.in
├── README.rst
├── data
│   └── data_file
├── sample
│   ├── __init__.py  <-- main() ('handler') here
│   ├── package_data.dat
├── setup.cfg
├── setup.py
├── tests
│   ├── __init__.py
│   └── test_simple.py
└── tox.ini

The project is super simple (all its main() function does is print "Call your main application code here"), but the code is organized, the setup.py tracks dependencies and excludes tests from packaging, there's a clear place to put docs and data, etc. All the awesome things you get from packages. This is how I write my projects, so I want to just pip install and then set the handler to sample.main and let Lambda find main() because it's part of the package namespace.

It turns out this is possible, and the Deployment Package doc sort of tells you how to do it when they talk about dependencies.

Their example is the requests package. They tell you to create a virtualenv, pip install requests, then zip the contents of the site-packages directory along with your module (that’s the directory inside the virtualenv where installed packages end up). Then you can import requests in your code. If you do the same thing with your package, Lambda can run the handler function and you don’t have to upload a module outside the package.

I’m going to use the pypa sample project as an example because it follows a common Python package structure, but we need a small tweak because Lambda calls the handler with args and the main() function of the sample project doesn’t take any.

Change this:

def main():

To this:

def main(*args):

Then do what you usually do with packages (here I’m going to create and install a wheel because I think that’s the current best practice, but there are other ways):

  1. Make a wheel.
    python setup.py -q bdist_wheel --universal
    
  2. Create and activate a virtualenv.
  3. Install the wheel.
    pip install sample-1.2.0-py2.py3-none-any.whl
    
  4. Zip up the contents of this directory:
    $VIRTUAL_ENV/lib/python2.7/site-packages
  5. Upload the zip.
  6. Set the handler to sample.main.

Then it works!

START RequestId: bba5ce1b-9b16-11e6-9773-7b181414ea96 Version: $LATEST
Call your main application code here
END RequestId: bba5ce1b-9b16-11e6-9773-7b181414ea96
REPORT RequestId: bba5ce1b-9b16-11e6-9773-7b181414ea96	Duration: 0.37 ms...

… except when it doesn’t.

The pypa sample package defines its main() in __init__.py, so the handler path is sample.main.

sampleproject/
├── sample
│   ├── __init__.py  <-- main() here

But what if your entry function is in some other module?

sampleproject/
├── sample
│ ├── __init__.py
│ ├── cool.py <– main() here

We can just set the handler to sample.cool.main, right? Nope! Doesn't work.

{
"errorMessage": "Unable to import module 'sample.cool'"
}

I'm working on figuring out why this happens, but the workaround is to import the function I need in __init__.py so my hanlder path only needs one dot. That's annoying but not too bad; lots of folks do that to simplify their package namespace anyway so most Python readers know to look for it. I met a chap at the SD Python group who has some great ideas on why this might be happening, if we figure it out I'll post the details.

To sum up:

  • You don’t have to cram all your code into one big file.
  • If you pip install your package, all you need to upload is the site-packages directory.
  • Remember to import your handler in the root __init__.py if you get import errors.

Thanks for reading!

Adam