Python: JSON Structured Logging


If you’re setting up JSON logging in AWS lambda, check out this instead. You need some extra code to prevent duplicate log messages.

Recently, I’ve been switching to logs structured as JSON. Using the sample command in my pattern for production-ready Python scripts, that means we replace delimited-strings like these:

2019-09-29 19:54:44,243 | INFO | sample_scripts.good | Acting.
2019-09-29 19:54:49,244 | INFO | sample_scripts.good | Action complete.

With JSON objects like these:

{"asctime": "2019-09-29 19:53:28,654", "levelname": "INFO", "name": "sample_scripts.good", "message": "Acting."}
{"asctime": "2019-09-29 19:53:33,654", "levelname": "INFO", "name": "sample_scripts.good", "message": "Action complete."}

Or, pretty-printed for human-readability:

  "asctime": "2019-09-29 19:53:28,654",
  "levelname": "INFO",
  "name": "sample_scripts.good",
  "message": "Acting."
  "asctime": "2019-09-29 19:53:33,654",
  "levelname": "INFO",
  "name": "sample_scripts.good",
  "message": "Action complete."

This way, your log processor can reference keys in a JSON object instead of splitting strings and hoping the split works right. Modern log processors are pretty good with delimited strings, but I’ve still seen them break because of things like unexpected whitespace or messages that happen to contain delimiter characters.

The Python logging library doesn’t have native JSON support, so I use the python-json-logger library. Here’s how I set it up:

from pythonjsonlogger import jsonlogger

def setup_logging(log_level):
    logger = logging.getLogger(__name__)
    json_handler = logging.StreamHandler()
    formatter = jsonlogger.JsonFormatter(
        fmt='%(asctime)s %(levelname)s %(name)s %(message)s'

That’s it! Just call setup_logging and then you can get loggers with logging.getLogger(__name__) as usual. If you’re not sure how to get and call loggers, check out the sample script in the production-readiness pattern I mentioned above. It has some code you can copy/paste.

Technically you could do this in raw Python if you set up your loggers right, but you’d basically be re-implementing what the python-json-logger library already does so I don’t recommend that approach.

I recommend switching to logs structured as JSON. It can reduce failures and complexity in log processing, and the output is cleaner overall.

Happy automating!


Need more than just this article? I’m available to consult.

You might also want to check out these related articles: