Audit logging with Kinesis Firehose

Audit logs

We had a requirement to maintain an audit log of user actions that management would be able to report on one day. Our systems currently forward logs to Datadog, but we have fairly short retention periods. Extending the retention period for a limited number of logs would be too costly at this stage.

The other option that seemed to make sense was storing these logs in S3 - they could be easily search at some stage later using AWS Athena. I'd heard of Kinesis Firehose before, so this seemed to be an ideal case to try it out.

Kinesis Firehose allows you to put events into the stream and configure an output destination, of which S3 is one. It takes care of bundling events into files, transforming and outputting the data at very little cost, all without standing up any servers.

The code

To setup Kinesis Firehose through cloudformation (Serverless specifically) you can use the following configuration and code.

Creating the infrastructure is really fast, so if you create a new stack for each branch of code, this will work well with your pattern. If you put your branch into the s3 prefix, it'll make life easier to determine which logs belong to which branch, and you can run Athena based on a S3 prefix, not just the whole bucket

Stream Configuration

Create an IAM role for the stream that allows it to write to S3 in a given path, configure the stream

Push events in our app

Now that we have a stream, it's simple to create a module for sending events: