CodePipeline: Python AWS Lambda Functions Without Timeouts

Hello!

Today we’re going to cover how to add Python AWS lambda functions to CodePipeline, and specifically how to do that without getting stuck in timeout loops you can’t cancel. Copy/pastable code first, details below.

import logging
import boto3

def lambda_handler(event, context):
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    logger.debug(event)

    codepipeline = boto3.client('codepipeline')
    job_id = event['CodePipeline.job']['id']

    try:
        # raise ValueError('This message will appear in the CodePipeline UI.')
        logger.info('Doing cool stuff!')
        response = codepipeline.put_job_success_result(jobId=job_id)
        logger.debug(response)
    except Exception as error:
        logger.exception(error)
        response = codepipeline.put_job_failure_result(
            jobId=job_id,
            failureDetails={'type': 'JobFailed', 'message': str(error)}
        )
        logger.debug(response)

Replace the two highlighted lines with the code you actually need to run in the pipeline. The commented raise is there to show you how the error handling works. More on that in a minute.

This article assumes you’re familiar with CodePipeline and lambda and that you’ve granted the right IAM permissions to both.

This is Python 3 code, of course. If you’re still using Python 2, it’s time to upgrade.

Logging

See my other article.

Success/Failure Results

CodePipeline uses a callback pattern for running lambda functions: it invokes the function and then waits for that function to call back with a result. Your function code has to send that signal. The simplest thing to do is send success when you’re done, like we did above:

        response = codepipeline.put_job_success_result(jobId=job_id)

If something goes wrong you can put_job_failure_result, instead. We have to send that even if exceptions stop our code’s execution because if no result is returned the pipeline will spin and spin until it times out (like with CloudFormation Custom Resources).

Timeouts

First, more on what happens when CodePipeline doesn’t get a result:

  1. The action’s details link will give you an error specifically saying it didn’t receive a result (so if you’re not seeing that message then this article probably isn’t the solution you need):NoResultReturnedErrorMinimal.png
  2. The CodePipeline limits doc says it takes 1 hour for lambda functions to time out. That used to apply to functions that didn’t send results, I tested it several times. Sadly, I didn’t think to keep screenshots back then. Today, it takes consistently 20 minutes for the action to time out:ConsistentTwentyMinuteTimeout
  3. It doesn’t matter what the lambda function’s timeout is. Mine was set to 3 seconds in the above screenshot. The limits doc linked above confirms that’s expected. Wouldn’t it be cooler, though, if CP could check the status of the lambda function’s execution? If the execution stopped, there’s no reason to wait.

20 minutes is better than an hour but it’s still a pain. It’s better to fail right away and not wait. You can do that with the except of a try/except block:

    except Exception as error:
        logger.exception(error)
        response = codepipeline.put_job_failure_result(
            jobId=job_id,
            failureDetails={'type': 'JobFailed', 'message': str(error)}
        )

Catch all exceptions and return a failure result.

  1. Normally, catching all exceptions is an anti-pattern. You should only catch exceptions your code knows how to handle. The rest should cause failures. CodePipeline lambda actions are an exception to catching exceptions because of their required callbacks. It’s ok to write except Exception in this case, but usually it’s not.
  2.  logger.exception(error) logs the exception and its stack trace. Even though we’re catching all errors, we shouldn’t let them pass silently.
  3. The failureDetails message will appear in the CodePipeline UI. We fill that will the exception message so it’s visible to operators:ExceptionMessageInUIMinimal.png In this screenshot, it came from a pipeline run where I uncommented the example ValueError from above:
            raise ValueError('This message will appear in the CodePipeline UI.')
    

Conclusion

For Python lambda functions running as actions in CodePipeline, the best practice is to catch all exceptions and ensure that the function sends a failure result back to the pipeline before it exits. Make sure you log the errors so you have enough information to diagnose them later. I recommend copy/pasting the snippet at the top of this article and filling it in with your code. That’s what I do. 😎

Happy automating!

Adam

If this was helpful and you want to save time by getting “copy and paste” patterns for Python Cloud DevOps in your inbox, subscribe here. If you don’t want to wait for the next one, check out these: