Audit logging with Kinesis Firehose

Audit logs

We had a requirement to maintain an audit log of user actions that management would be able to report on one day. Our systems currently forward logs to Datadog, but we have fairly short retention periods. Extending the retention period for a limited number of logs would be too costly at this stage.

The other option that seemed to make sense was storing these logs in S3 - they could be easily search at some stage later using AWS Athena. I'd heard of Kinesis Firehose before, so this seemed to be an ideal case to try it out.

Kinesis Firehose allows you to put events into the stream and configure an output destination, of which S3 is one. It takes care of bundling events into files, transforming and outputting the data at very little cost, all without standing up any servers.

The code

To setup Kinesis Firehose through cloudformation (Serverless specifically) you can use the following configuration and code.

Creating the infrastructure is really fast, so if you create a new stack for each branch of code, this will work well with your pattern. If you put your branch into the s3 prefix, it'll make life easier to determine which logs belong to which branch, and you can run Athena based on a S3 prefix, not just the whole bucket

Stream Configuration

Create an IAM role for the stream that allows it to write to S3 in a given path, configure the stream

Push events in our app

Now that we have a stream, it's simple to create a module for sending events:

Spinning down your cloud costs

One of the best things about the cloud is the ability to spin up new infrastructure for your dev environment. You can have a completely isolated environment for a new branch that you're working on, allowing you to test your changes in isolation.

This is fantastic as you can speed ahead on changes unencumbered - the only problem is that you'll leave a trail of cash sitting in your cloud providers pockets, which often doesn't work out so well for your business.

To attack our cost blowouts, we started a side project in our lunchtime to figure out how we can stop the hurt, and came up with Batman built using Serverless. Most of our infrastructure is managed in a single place (cloudformation) with a set of tags associated with each stack including the slice (or branch name), and the cloudformation stack keeps a created date & last update date. Utilizing this information Batman runs every night to cleanup stacks that are on non-master branches where the stack hasn't been updated for 7 days. This is usually enough time for us to have tested the branch and integrated it into the master stack.

We've also recently added a webhook trigger that deletes a stack when the branch is deleted in git which saves us up to 7 days of running costs.

We've stopped over 300 stacks in a month, and at a guesstimate of $1 per day running costs for a stack, the dollars quickly add up.

There are many ways to save costs, and we've found that for our journey into AWS, Batman has come about at the right time. Not only has it been interesting to explore serverless, our team got to do a lot of the work with some mob programming which was really fun (more to come).