When you need to publish logs to CloudWatch (e.g. from a lambda function), you need an IAM role with access to CloudWatch. It’s tempting to use a simple policy like the one in the AWS docs. You might write a CloudFormation template like this:
# Don't use this! AWSTemplateFormatVersion: '2010-09-09' Resources: DemoRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Path: '/' Policies: - PolicyName: lambda-logs PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:DescribeLogStreams - logs:PutLogEvents Resource: arn:aws:logs:*:*:* DemoFunction: Type: AWS::Lambda::Function Properties: Code: ZipFile: | def handler(event, context): print('Demo!') FunctionName: demo-function Handler: index.handler Role: !GetAtt DemoRole.Arn Runtime: python3.7
Obviously, the role is too permissive:
But, there’s another problem: it grants
Here’s what happens:
- Launch a stack from this template
- Because we granted it permission,
/aws/lambda/demo-functionlog group in CloudWatch Logs
- Delete the stack
- CloudFormation doesn’t delete the
CloudFormation doesn’t know about the function’s log group because it didn’t create that group, so it doesn’t know anything needs to be deleted. Unless an operator deletes it manually, it’ll live in the account forever.
It seems like we can fix that by having CloudFormation create the log group:
DemoLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: /aws/lambda/demo-function RetentionInDays: 30
But, if the function still has
logs:CreateLogGroup I’ve seen race conditions where the stack deletes the group before the lambda function and the function recreates that group before it gets deleted.
Plus, there aren’t any errors if you forget to define the group in CF. The stack launches. The lambda function runs. We even get logs, they’ll just be orphaned if we ever delete the stack.
That’s why it’s a problem to grant
logs:CreateLogGroup. It allows lambda (or EC2 or whatever else is logging) to log into unmanaged groups.
All resources in AWS should be managed by CloudFormation (or terraform or whatever resource manager you use). Including log groups. So, you should never grant
logs:CreateLogGroup except to your resource manager. Nothing else should need that permission.
And that’s the other reason: lambda doesn’t need
logs:CreateLogGroup because it should be logging to groups that already exist. You shouldn’t grant permissions that aren’t needed.
Here’s the best practice: always manage your CloudWatch Logs groups and never grant permission to create those groups except to your resource manager.
If this was helpful and you want to save time by getting “copy and paste” patterns for Cloud DevOps in your inbox, subscribe here. If you don’t want to wait for the next one, check out these: